Audit of the Advanced Exploration Technology Development Program Management Framework (1.2.2.3) - Audit Report

Project # 12/13 01–06

Prepared by the
Audit and Evaluation Directorate

November 2013

1.0 Summary

1.1 Audit objective

The objective of this audit was to determine whether the management framework in place enables the program to achieve its objectives and to comply with relevant policies, regulations and guidelines issued by the Canadian Space Agency (CSA) and the central agencies.

1.2 Audit opinion

In our opinion, the current management framework allows the program to achieve its objectives and comply with the policies, regulations and guidelines of the CSA and the central agencies.

1.3 Statement of Assurance

As Chief Audit Executive, I am of the opinion that sufficient and appropriate audit procedures were conducted and evidence gathered to support the accuracy of the information contained in this report. This opinion is based on a comparison of the conditions, as they existed at the time of the audit, against pre-established audit criteria that were discussed with management. The opinion is applicable only to the particular entity examined. The evidence was gathered in compliance with Treasury Board policy, directives and standards for internal audits. The procedures followed comply with the professional standards of the Institute of Internal Auditors. The evidence gathered is sufficient to persuade senior management of the validity of the opinion derived from the internal audit.

1.4 Summary of recommendations

Our audit demonstrated that the Advanced Exploration Technology Development (AETD) Program has put in place good practices to ensure that operations are planned effectively and that resources used are properly monitored. In addition to the integration by the Program of various CSA corporate procedures, several practices and tools were developed by the Program in order to plan and monitor the utilization of financial and human resources.

We also found that the eligibility criteria for selecting contractors to whom Economic Action Plan (EAP) contracts were awarded were properly implemented and that the selection process was properly documented; the Program identifies and manages risks that may hamper the achievement of expected outcomes; financial transactions are recorded in accordance with acts, regulations and guidelines in effect; and all Program activities are subject to accountability reporting.

We also found some deficiencies for which we have drawn up the following two recommendations:

  1. Review the procedure for monitoring and approval of expenditures for interdepartmental payments; and
  2. Make adjustments to the Performance Measurement (PM) Strategy based on future activities and implement the strategy.

The original version signed by
Dominique Breden, Chief Audit Executive

Audit team members

  • Jimmy Cheung
  • Dany Fortin

2.0 Audit Report

2.1 Background and risks

Through Canada's 2009 Economic Action Plan (EAP), the Government of Canada (GoC), through the Canadian Space Agency (CSA), invested $110 million over three years in the Advanced Exploration Technology Development Program to support the development of prototypes of systems that might become contributions to future Moon or Mars missions. This sub-sub-program (SSP) is part of the Exploration Missions and Technology (1.2.2) sub-program (SP) of the Agency's Program Activity Architecture (PAA).

The Program encompasses the development of Canadian signature technologies to be used in potential astronomy and planetary missions whose destinations may be the Moon, Mars, asteroids or other celestial bodies. It also includes terrestrial deployments on analogue sites that offer geological similarities with Martian or Lunar surfaces, where this technology and its operational aspects are being tested and where exploration-related science is conducted for proof-of-concept purposes.

The Program activities are carried out in collaboration with foreign space agencies and GoC agencies and through the CSA's participation in the activities of the International Space Exploration Co-ordination Group. The Program activities are usually implemented through the awarding of contracts to Canadian aerospace firms or under agreements with international partners.

The following is a summary of Program activities:

  • Management of prototype concept and development projects;
  • Plan and manage advanced exploration technology development procurements and contracts;
  • Integrate and utilize advanced exploration prototypes;
  • Carry out terrestrial deployments on analogue sites;
  • Maintain operational infrastructure;
  • Attend international meetings concerning space exploration;
  • Identify opportunities for Canada's possible contribution to international space exploration missions.

The EAP included significant measures for providing short-term assistance for key sectors of the economy, such as Canada's space industry, to help them deal with the economic downturn. Timely, targeted and temporary stimulus had to be provided to achieve the following objectives:

  • Help Canadian firms struggling with the greatest problems; and
  • Generate maximum spinoffs in terms of job creation in Canada.

Through this investment, the CSA implemented the Stimulus initiative by carrying out the Exploration Surface Mobility (ESM) and Next Generation Canadarm (NGC) projects. The two main objectives of this initiative were to maintain and increase Canada's expertise and leadership in space robotics and to help Canada better prepare itself and ensure its credibility as a partner in future space robotics and space exploration projects. This initiative contributed to the development of terrestrial prototypes of space robotic vehicles ("rovers") as well as to progress in the development of advanced space robotics and other technologies.

  • The ESM Project involved the development of several rovers and associated rover systems for the purposes of future Moon or Mars missions. The purposes of the two types of rovers developed are
    • Scientific exploration of resources; and
    • Scientific activities on Mars.
  • The NGC Project involved the following four advanced-robotics prototypes as well as a mission control station:
    • Next-Generation Large Canadarm;
    • Next-Generation Small Canadarm;
    • Proximity Operations System Testbed; and
    • Semi-Autonomous Docking System.

Program activities other than those related to the EAP are grouped under the name ExCore.

The following is an overview of the Program between 2009–2010 and 2014–2015:

Budget for the CSA's Overall Advanced Exploration Technology Development Program

  2009-2010
$M
2010-2011
$M
2011-2012
$M
2012-2013
$M
2013-2014
$M
2014-2015
$M
ESM 8.6 22.2 39.0 8.3 - -
NGC 6.5 27.3 19.7 - - -
ExCore 11.2 8.8 9.2 3.1 7.7 6.2
Total 26.3 58.3 67.9 11.4 7.7 6.2

*Sources: Program statistics (2009–2013) and statistics based on annual reference level update (2013–2015)

This audit project is part of the risk-based audit plan for the period from 2012–2013 to 2014–2015, which was approved by the Audit Committee. This project was warranted by the $110 million in expenditures incurred for EAP purposes for the period from April 1, 2009 to March 31, 2013.

In addition, the Office of the Auditor General of Canada (OAG) issued special instructions concerning the management framework to be adopted for the EAP.

2.2 Audit objective, scope and approach

Objective

The audit objective is to determine whether the management framework in place enables the program to achieve its objectives and to comply with relevant policies, regulations and guidelines issued by the Canadian Space Agency (CSA) and the central agencies.

Scope

This internal audit project covered activities implemented for EAP purposes between April 1, 2009, and the current date, as well as activities between April 1, 2011, and the current date for the overall Advanced Exploration Technology Development Program. More specifically, we examined the following:

  • Planning of activities;
  • Monitoring of activities; and
  • Monitoring of outcomes and accountability reporting.

Approach

The audit criteria were drawn up on the basis of the best management practices and guidelines issued by the Treasury Board Secretariat (TSB). The audit included various audit procedures, such as the holding of interviews and a review of documents, contract records and financial transactions.

2.3 Findings, recommendations and management response

To determine whether the management framework in place enables the program to achieve its objectives and to comply with relevant policies, regulations and guidelines issued by the Canadian Space Agency (CSA) and the central agencies, we expected to find out the following:

  • Operations that are planned effectively;
  • Operations and resources used that are monitored; and
  • Organization's operations that are subject to accountability reporting and results that are measured.

It should be noted that the audit objective and the audit criteria were discussed with management.

2.3.1 Planning of activities

Audit objective: To determine whether the management framework in place enables the program to achieve its objectives and to comply with relevant policies, regulations and guidelines issued by the Canadian Space Agency (CSA) and the central agencies.

Findings
Criterion 1

Operations are planned effectively.

Condition

Financial resources planning

We found that the Program's financial planning process provided support for the implementation of activities. In addition to the integration of various CSA's corporate planning procedures, several practices and tools were developed by the Program in order to plan and monitor the utilization of available financial resources.

Program planning is part of the Agency's integrated planning cycle that requires the integration of work and operational plans with financial estimates. Accountability reporting is then carried out based on the planned activities. For example, the 2012 operational plan establishes the link between the operational objectives and the financial resources, as required by the corporate procedures. The planning process is based on a set of standard documents and corporate procedures.

At the corporate level, the Agency's 2012-2014 integrated management cycle schedule is used to co-ordinate and facilitate the participation of managers and their employees in various departmental management planning and monitoring activities. The schedule is used to provide advance notice of the deadlines for each requirement, identifies the resource persons responsible, and provides timely instructions and associated documents to help the programs in terms of the planning of activities and resources.

For the financial monitoring of operations, a manager has been assigned to each ESM and NGC project and to each of the five main Program activities, including concept and development projects, prototypes, analogue deployments, infrastructure and operational space medicine. Each manager is responsible for keeping the financial information under his/her responsibility up-to-date. A main folder is used to keep a record of actual Program expenditures. It is updated regularly and aligned with the Agency's SAP financial system. Each manager also has access to the SAP financial system to monitor expenditures. Every month, financial forecasts are drawn up and validated by the senior manager and the director responsible for the program.

Moreover, a project manager is assigned to each contract to monitor and manage expenditures. The contract managers are in frequent communication with the program manager. The senior manager responsible for the entire program uses the main folder to oversee the status of expenditures. Every two weeks, meetings are held with the managers to discuss progress in the activities.

Human resources planning

In the area of human resources planning, an Integrated Human Resources Plan (IHRP) for the Space Exploration (formerly Space Technology) sector, which includes the DTEA program, was drafted for the purpose of implementing Program activities. The IHRP includes an analysis of current and future human resource and skill requirements for meeting operational and organizational requirements.

The IHRP was used to draw up a Human Resources Business Plan (HRBP) identifying all of the staffing requirements for the sector. In the case of the DTEA program, in order to carry out ESM and NGC projects, we found that the human resources requirements had been met through internal assignments within the CSA. In addition, we found nothing that lead us to believe the Program employee turnover rate posed a risk in relation to the CSA as a whole. For example,

  • For the period ending on March 31, 2012, the Program employee turnover rate was 2.5%, compared to the Agency's employee turnover rate of 9.2%; and
  • For the period ending on March 31, 2013, the Program employee turnover rate was 0%, compared to the Agency's employee turnover rate of 5.4%.

Risk identification and management

In the area of operational risks, our audit found that the risks that might hamper achievement of the Program's expected outcomes were identified and managed. The risk identification procedures take internal and external sources of risk, such as costs, deadlines and technical and program problems, into account.

In the case of the EAP-related ESM and NGC projects, a qualitative assessment of the impact and probability of identified risks was carried out. Mitigation strategies were then developed and implemented. The risks were classified in low, medium and high categories, based on the results of the assessment. The risk levels were subsequently updated, as needed. The risks posed by these two projects were also assessed in accordance with the Agency's organizational risk profile (ie, strategic vision, the space craft concerned, support provided by participants, trust in Agency governance, integration and implementation, and the labour force) in order to better determine the overall corporate risk. For the other Program activities, ie, ExCore, the risks are identified and managed at the level of each contract.

Eligibility criteria

In our opinion, the eligibility criteria for selecting contractors to carry out EAP-related contracts were properly implemented and the selection processes were properly documented. The criteria were the following:

  • Firms had to be Canadian;
  • Firms had to have and develop operations in Canada; and
  • Firms had to be incorporated, registered and recognized in accordance with federal and provincial legislation.

The CSA also set up a competitive process for selecting contractors' technical proposals to ensure that they met technical, project management and impact criteria.

The following is an overview of the CSA's bid assessment process:

  • Meetings were held to ensure that the following appropriate documents were provided:
    • Bid selection assessment plan:
    • Requests for Proposals (RFPs);
    • Technical proposals;
    • Assessment grids (one sheet per evaluator per proposal).
  • Each evaluator evaluated the proposals individually and the results were entered in the system designated for that purpose;
  • Evaluation Committee review meetings were held for each bid assessed by the Agency. During these meetings, the evaluators discussed each bid in order to resolve any differences in their evaluations.
  • Bids were rated according to their overall scores;
  • The evaluation reports were prepared and approved by the Program Director and sent to Public Works and Government Services Canada (PWGSC);
  • The Agency reviewed the financial proposals and sent its comments (if any) to PWGSC.

To ensure that the eligibility criteria were properly defined and implemented, we selected a sample of 5 contracts out of the 35 for the ESM Project. The sample was compiled based on the monetary value of the contracts. The value of the five selected contracts amounted to $29 million out of the $78 million in awarded contracts. The sample represented 37% of the total value of contracts for this project. We found in our review that the eligibility criteria and the selection procedures had been implemented appropriately and consistently. We also found that appropriate documents had been provided in support of the recommendations for the choice of eligible recipients.

In the case of the NGC Project, a written rationale for the analysis had been provided as supporting documentation for the non-competitive contract worth a total value of about $54 million that was awarded by PWGSC, with the required authorizations from the Treasury Board of Canada (TB).

Cause

N/A

Effect

N/A

Recommendation

N/A

Identified responsibility
Organization

N/A

Function

N/A

Management response

N/A

Management action plan
Action plan details

N/A

Timetable: N/A

2.3.2 Current monitoring of operations and resources

Audit objective: To determine whether the management framework in place enables the program to achieve its objectives and to comply with relevant policies, regulations and guidelines issued by the Canadian Space Agency (CSA) and the central agencies.

Findings
Criterion 2

Operations and resources used are monitored.

Condition

Organization of work and distribution of responsibilities

We found that the work organization and the distribution of responsibilities contributed to effective delivery of DTEA program activities. Some adjustments had been made in terms of the distribution of responsibilities in order to meet the requirements arising from implementation of the ESM and NGC projects. Moreover, in the Program operational plans and work plans, the duties and responsibilities of each team member were properly defined in order to co-ordinate and implement all of the operational activities.

All of the Program operations are under the responsibility of the Director General, Space Exploration. The work is organized in three main activities: ESM, NGC, and the other small projects in the ExCore program.

The ESM project includes three sectors in addition to five sub-projects. Each sub-project was overseen by a manager. All of the sectors and the managers of sub-projects were under the supervision of the ESM Project Manager. In the case of the NGC, the project was divided into three sectors, whose chiefs all reported to the Project Manager.

In the case of the five other activities that are part of the general program (ExCore), a manager is responsible for each activity, ie, Concept and Development, Prototypes, Analog Deployments, Infrastructure and Operational Space Medicine.

Expenditure approval and compliance

In our opinion, the spending authorizations are properly monitored and expenditures are approved and incurred in accordance with acts, regulations and guidelines.

To meet this criterion, we put together and reviewed a sample of 30 financial transactions. The transactions were selected on the basis of their monetary value. The value of this sample was about $23 million, or 14% of recorded program expenditures during the 2009–2013 audit period. The sample had the following characteristics:

  • 5 travel expenses transactions;
  • 18 transactions for the purchase of professional services, including 10 transactions of more than $1 million;
  • 2 transactions for office building repairs (construction associated with the storage centre and garage for rovers);
  • 3 transactions for expenditures on machinery and materials;
  • 1 transaction for an expenditure on miscellaneous items;
  • 1 transaction for an expenditure on computer equipment.

After reviewing this sample, we found that 29 expenditure transactions had been approved in accordance with delegated financial authorities in effect and with acts, regulations and guidelines.

However, in one case, we found that a $75,000 payment had been made without certification from the delegated manager that the services had been provided in accordance with the contract (section 34 of the Financial Administration Act [FAA]). This payment was an interdepartmental settlement that had been initiated and committed in the SAP system at the time of signing of the interdepartmental Memorandum of Understanding. The payment had been made automatically at the request of the department concerned after the service had been provided for the CSA. In this case, the CSA never received the invoice related to this service.

Recording of expenditures

In our opinion, the financial transactions are recorded in accordance with the accounting rules in effect. In order to respond to this sub-criterion, we reviewed the descriptions of all the transactions recorded between April 1, 2009 and March 31, 2013, in order to identify any exceptions and we also thoroughly reviewed the coding of the 30 selected expenditure transactions for the preceding sub-criterion. During this review, we found no situation demonstrating that the transactions were recorded improperly.

Monitoring of activities

In our opinion, there are controls in place to monitor activities. As previously mentioned, project managers are assigned to do monitoring according to an established contract management procedure. Program managers meet with project managers on a regular basis to review the advancement of various contracts. The project managers are the CSA's key stakeholders with contractors.

We had an initial sample of three contracts that had been selected to determine whether the documentation was appropriate and if the existing controls made it possible to periodically measure the degree of completion of the contracts. Following our review, we found that, in the case of a 95,000$ value contract, two monthly progress reports required from the supplier had not been obtained, as stipulated in the contract.

Following our review and initial finding, we decided to add seven additional contracts to our review sample, for a total of 10, in order to determine if this was an exceptional case or if this practice was widespread.

Following our review of the seven additional contracts, we found that the documentation was appropriate and that the existing controls made it possible to periodically measure the degree of completion of the contracts. The progress reports were obtained and kept on file, as stipulated in the contracts.

Following the review of the seven additional contracts and our discussions with management, it was concluded that this case of non-compliance was an isolated event and therefore no recommendation in that regard was made.

Cause

Expenditure approval and compliance

No formal procedures regarding the monitoring of interdepartmental payments after the expenditure commitment was made in the SAP financial system upon signing of the Memorandum of Understanding.

Effect

Expenditure approval and compliance

Unsatisfactory monitoring may result in interdepartmental payments not being in compliance with acts, regulations and guidelines. For example, a payment may be made without the service having been provided.

Recommendation

Expenditure approval and compliance

1. Review the procedure for monitoring and approval of expenditures for interdepartmental payments.

Identified responsibility
Organization

Expenditure approval and compliance

Finance

Function

Expenditure approval and compliance

Chief Financial Officer

Management response

Expenditure approval and compliance

Management agrees with the recommendation.

Management Action plan
Action plan details

Expenditure approval and compliance

Following this audit, we have already set up a centralized monthly procedure whereby a departmental accounting officer ensures that each departmental settlement transaction entered in the system has an invoice associated with the expenditure. This invoice must be verified and approved by a person with delegated authority, who will ensure that the service has been provided or the good received.

Timetable: Ongoing. March 31, 2014

2.3.3 Accountability reporting and performance measurement

Audit objective: To determine whether the management framework in place enables the program to achieve its objectives and to comply with relevant policies, regulations and guidelines issued by the Canadian Space Agency (CSA) and the central agencies.

Findings
Criterion 3

The entity's operations are subject to accountability reporting and the results are measured.

Condition

Accountability reporting

We found that all of the Program activities were subject to accountability reporting.

For internal accountability reporting purposes, the activities are identified at the start of the year as part of the CSA's integrated management cycle. Detailed operational plans for the Program are submitted, including descriptions of activities to be carried out during the following year, resources to be assigned to the activities, desired outcomes and performance targets. Follow-up is done at mid-year and at the end of the fiscal year. We found that these procedures have been followed for the Program every year since 2009-2010 and that entries were made at the start of the 2013-2014 fiscal year.

In terms of external accountability reporting, the Program is subject to accountability reporting to the central agencies, as is the case with all CSA activities; it is documented in the Departmental Performance Report (DPR), which is drafted from information obtained from the Performance Measurement Framework (PMF).

In regard to the EAP-related programs to develop the new Canadarm (NGC) and exploration prototypes (ESM), the accountability reporting has been very frequent because, since 2009, the CSA has had to produce for the central agencies more than 70 reports providing financial data and information on the advancement of projects. However, the benefits resulting from these technological developments are still to be evaluated.

Performance Measurement Strategy

A Performance Measurement (PM) Strategy must be implemented in order to oversee and measure the results of the Program systematically; produce useful, relevant reports for decision-making purposes; and gather credible, reliable information to help assess the Program effectively. According to the Directive on the Evaluation Function, in effect since April 1, 2009 (Section 6.1.4), "Developing, implementing and monitoring ongoing performance measurement strategies for programs is the responsibility of programs managers."

We found that a PM Strategy had been developed for the Program in 2013 in accordance with the requirement in that regard. However, according to the 2012-2013 Annual Report on the State of Performance Measurement, it had been determined that this PM Strategy required some adjustments in order to (1) reflect recent changes to the Program resulting from the termination of the EAP and (2) be useful in future program evaluations. Specifically, the logic model and the indicators are to be reviewed to ensure that they reflect the changes to the Program. The PM Strategy must also include targets to be used to assess the achievement of objectives.

Corrective measures and lessons learned

Flexible procedures were set up for the Program to make it possible to respond on the basis of risks and opportunities that arise in the management of budgets and execution of contracts and in relationships with international partners. The Program provided us with a few examples of corrective measures taken in order to take advantage of opportunities or achieve results.

In terms of lessons learned, we found very detailed documentation concerning the ESM and NGC projects.

Cause

Performance Measurement Strategy

In recent years, the CSA has developed a set of performance management strategies.

The recently developed PM Strategy for the Program resembles more closely to an evaluation strategy in the sense that it was developed to compile data from the past and use it for the current Program evaluation.

Effect

Performance Measurement Strategy

The possible risk in not having a performance measurement strategy is that the information required for decision-making or to carry out an evaluation of the Program may not be available at an opportune time.

Recommendation

Performance Measurement Strategy

2. Make adjustments to the Performance Measurement (PM) Strategy based on future activities and implement the strategy.

Identified responsibility
Organization

Space Exploration Branch

Function

Strategic planning

Management response

Performance Measurement Strategy

There is already a recognized need to update the performance management strategy for the Advanced Exploration Technology Development sub-sub-program (1.2.2.3). The drafting of the new PM Strategy, including a new logic model, indicators, targets and data-gathering framework, has already begun.

Management action plan
Action plan details

Performance Measurement Strategy

The PM Strategy for sub-sub-program 1.2.2.3 (DTEA) will be updated.

Timetable : March 31, 2014

Data-gathering and analysis tools will be developed.

Timetable : March 31, 2015

The data will be gathered in compliance with the approved Performance Measurement Strategy.

Timetable : March 31, 2016