Agence spatiale canadienne
Symbol of the Government of Canada

Table of Contents

Audit report 2004

Audit of the Management Framework of the Canada/ESA Cooperation Agreement
PROJECT #04/05-01-01

Prepared by
Audit, Evaluation and Review Directorate

November 2004

Top of page

EXECUTIVE SUMMARY

The objective of this audit project is to evaluate the extent to which the systems and procedures used to manage the Canada/ESA Cooperation Agreement in all its aspects are effective, efficient and economical.

The current Canada/ESA Cooperation Agreement was signed on March 28, 2000 and will be in force until 2008-2009, subject to an agreement to renew its terms and conditions effective April 1, 2005. Up to $30 million in contributions will be paid out to ESA each fiscal year throughout the term of the Agreement.

Implementation of the recommendations made in this audit report will allow the manager of the Canada/ESA Cooperation Program to carry out the duties of this position more effectively, especially with respect to compliance with program policies and conditions, decision making and accountability.

More specifically, the audit report makes the following recommendations:

  • Complete administrative records for each optional program should be retained to ensure an adequate audit trail and support decision making.
  • Program approval documents (PADs), submitted to the Program Review Advisory Board (PRAB) with a view to obtaining approval for optional programs, should contain all the necessary information, as described in the program terms and conditions approved by the Treasury Board (TB).
  • A process for holding consultations with industry stakeholders should be implemented with a view to supporting decisions respecting optional programs selected by Canada.
  • The rationale for considering the contribution to the General Budget as a generic expenditure related to various activities according to the new Program Activities Architecture (PAA) should be reviewed in line with the actual contribution to optional programs so that more accurate financial information can be obtained.
  • Monitoring of accumulated appropriations in ESA accounts should be carried out in early 2005 to ensure that they have actually been used when the next payment call occurs.
  • A mechanism for minimizing the risks of fluctuations in exchange rates should be implemented in consultation with the Treasury Board Secretariat (TBS).
  • Monitoring of expenditures related to optional programs should be carried out to ensure that cumulative payments match the amounts in foreign currency that the Canadian Space Agency (CSA) committed itself to, thereby allowing the detection of cost overruns.
  • The Risk-Based Audit Framework (RBAF) and the Results-Based Management and Accountability Framework (RMAF) should be implemented.

This internal audit was carried out in accordance with the Treasury Board Policy on Internal Audit and the IIA (Institute of Internal Auditors) Standards for the Professional Practice of Internal Auditing. In our professional opinion, the audit procedures followed and evidence gathered were sufficient and appropriate and support the accuracy of the conclusions in this report. The conclusions are based on a review of the situations in question using established audit criteria. The conclusions only apply to the entity examined.

Top of page

1.0 INTRODUCTION

1.1 RATIONALE FOR THE AUDIT PROJECT

The audit of the Canada/ESA Cooperation Agreement is part of the approved 2004-2005 audit plan. This audit was included in the annual plan further to item 4.2 of the program's Risk-Based Audit Framework (RBAF), which states that an audit of the Canada/ESA Cooperation Agreement shall be carried out in 2004-2005. This review will also be included in the evaluation report required by the Treasury Board (TB) in anticipation of the renewal of the terms and conditions of the program, pursuant to the Policy on Transfer Payments.

This audit project was carried out from July to October 2004.

1.2 OBJECTIVE AND SCOPE OF THE AUDIT

The objective of this audit project consists of obtaining assurances that all systems and procedures currently used to manage the Canada/ESA Cooperation Agreement in all its aspects are effective, efficient and economical.

The audit covered elements of the Canada/ESA Cooperation Agreement's management framework that were in force during the 2003-2004 fiscal year.

1.3 BACKGROUND

The current Canada/ESA Cooperation Agreement was signed on March 28, 2000, and has been in force since January 1, 2000. It will remain in force for 10 years, subject to an agreement to renew its terms and conditions.

Under the Agreement, Canada makes annual contributions to ESA expenditures under the General Budget, with the exception of costs associated with the Scientific Program and Technological Research Programs. Canada contributes to the ESA General Budget at 50% of the GNP-based rate normally used to calculate the contribution of ESA member states. The General Budget relates to all expenditures involved in the overall management of ESA. The contribution to the General Budget is mandatory and provides certain rights and privileges, the most important one being the right to participate in optional programs. These programs relate to all space hardware and technology development initiatives open to member states, which can decide whether or not to participate in a program. The Agreement also provides for a clear obligation to ensure Canada a fair industrial return and guarantees Canada the same rights as ESA member states in optional programs.

The ESA is to receive up to $30 million in contributions each fiscal year over the period covered by the Cooperation Agreement (1999-2000 to 2008-2009).

1.4 METHODOLOGY

This audit was carried out in accordance with audit standards set forth in the Treasury Board Secretariat (TBS) Policy on Internal Audit, which requires that audit objectives be set on the basis of audit criteria. The audit objectives and criteria used are outlined in Appendix A.

Audit standards also require that the audit be conducted in a structured manner, according to a process that includes

  • a planning and preliminary review phase,
  • an execution phase, and
  • the reporting and disclosure of results.

Various audit procedures were used, including interviewing employees and reviewing and analysing documents, records and reports.

The auditor provided periodic verbal reports to keep the entity being audited informed about how the audit project was progressing.

Top of page

RESULTS OF THE AUDIT

2.0 ADMINISTRATIVE RECORDS

2.1 AUDIT TRAIL

Pursuant to the Treasury Board Secretariat's Policy on Transfer Payments, departments must establish policies and procedures to ensure that proper program and accounting records and other relevant documents are maintained to provide documentary evidence of decisions made and results achieved and to allow for disclosure of the amounts paid to recipients of such payments.

Under the Canada/ESA Cooperation Agreement, contributions are made to ESA's General Budget and to optional programs in which Canada has chosen to participate following an analysis of each individual optional program. The selection and approval process for optional programs is one of the elements covered by the Terms and Conditions of Contributions Under the 2000-09 Canada/ESA Cooperation Agreement (Appendix 1 of the TB decision). The selection and approval process for optional programs typically generates the following types of documents:

  • Program Approval Documents (PADs),
  • Arrangements,
  • Declarations,
  • Results of consultations, and
  • Correspondence.

To ensure proper monitoring of the entire selection and approval process for optional programs, maintain an adequate audit trail and account for decisions made in connection with these programs, it is essential that complete administrative records be kept.

Our review of the situation showed that records relating to optional programs were not up to date and did not contain all the documents required to ensure a complete audit trail.

For the purposes of our audit, we had to consult several different sources to obtain the required documents:

  • Copies of signed Arrangements were obtained from the CSA's Legal Services Directorate;
  • Copies of unsigned Declarations were obtained from ESA's electronic records management system (DODIS); and
  • Copies of submissions to the PRAB (which takes the place of a PAD) were obtained from the sector's financial analyst.

DODIS-The ESA's Official Documents Distribution and Archiving System

The manager of the Canada/ESA Cooperation Program told us that he relied to a large extent on DODIS to obtain information, such as copies of Arrangements, Declarations or other documents used in decision making, in a timely manner.

DODIS is ESA's electronic archiving system, and not the CSA's system. If, at any time, for reasons beyond the CSA's control, this system were no longer available to the CSA, it would be very difficult to track down certain documents required to maintain an adequate audit trail as described above. This makes it important for the CSA to retain complete administrative records for all optional programs to which the CSA contributes.

RECOMMENDATION

Ensure that complete administrative records are kept for every optional program in order to maintain an adequate audit trail and support decision making.

Top of page

3.0 SELECTION AND APPROVAL PROCESS FOR OPTIONAL PROGRAMS

3.1 PROGRAM APPROVAL DOCUMENT (PAD)

Item 3.1, Selection Process, of Appendix 1 of the TB decision stipulates that a PAD must be prepared for each optional program under consideration for submission to the Program Review and Advisory Board (PRAB). This provision also describes what a PAD must contain:

  • Content and objectives;
  • Estimated policy, program and industrial development benefits;
  • Results of consultations with concerned federal partners and industry;
  • ESA position on Canadian participation in the program;
  • Eligible costs;
  • Proposed subscription rate;
  • Potential contributions and required budgets by fiscal year; and
  • Performance measurement indicators.

Our review showed that the submission to the PRAB takes the place of a PAD for optional programs. However, the format of submissions to the PRAB is preset and does not cover all the information required to prepare a PAD, as detailed in the following table:

TABLE 1 - REQUIRED INFORMATION FOR PADS

PAD Element Comments
Objectives The linkage between the optional program and Canada's objectives for participating in the ESA program was satisfactory in two of the four optional programs reviewed.
Consultations with industry, departments or other stakeholders The "Consultations" section referred to in submissions to the PRAB generally refers to the CSA's internal consultations with managers in the sectors concerned. In some cases, other departments had to be consulted, so this was mentioned in the submission. The only information regarding consultations with industry partners that appeared in the submissions referred to a company designated to receive one or more contracts further to a contribution to the optional program under consideration.
ESA position on Canadian participation in an optional program Submissions to the PRAB do not mention the ESA position on Canadian participation. It is assumed that ESA agreed. Otherwise, there would be no reason to make a submission to the PRAB, which raises the following question: is this information relevant to preparing a PAD?
Total eligible costs There was no mention of the total eligible costs of optional programs in submissions. Only the CSA's contribution was indicated. For this reason, there was no way to determine the extent of Canadian participation in relation to the program as a whole.
Performance measurement indicators Objectives for optional programs were identified in the text, but there were no performance measurement indicators linked to the objectives.

RECOMMENDATIONS

  1. Review the relevance of including an item respecting "ESA position on Canadian participation in an optional program" in the TB submission regarding the renewal of the program's terms and conditions.
  2. Ensure that all required information is included in PADs in accordance with the program terms and conditions approved by TB.

3.2 CONSULTATION PROCESS

The TB decision on the Canada/ESA Cooperation Agreement stipulates in item 4 that the CSA shall consult with DFAIT, DND and other departments and agencies concerned before signing any Arrangement respecting an ESA optional program. Appendix 1 of the TB decision states that the results of consultations with concerned federal partners and industry shall form part of the PAD's content.

However, very little documentation detailing the consultations held was available.

We obtained a list of main events used to promote the ESA program, or one of its components, and/or used as a mechanism for holding consultations on the relevance of maintaining the ESA program. The events on that list were the following:

TABLE 2 - PROMOTIONAL AND CONSULTATION EVENTS

Date Event Purpose
September 10, 2001 Consultative conference with Canadian industry, federal departments, universities and other interested organizations Establish Canada's priorities for participating in new ESA optional programs, in preparation for the ministerial-level ESA Council meeting in November 2001.
March 1, 2002 Envisat launch Promote ESA program and its successes, in this case, Envisat, a major earth observation project.
November 6 and 7, 2002 ESA EO Business Day and Canada Day Promote the ESA program (networking). Give representatives of Canadian earth observation (EO) firms access on an individual basis to EO technical representatives from ESA (ESTEC) and representatives of European EO industries.
October 29, 2002 Space Tech 2002 Promote the ESA program - a presentation on ESA was given.
February 3, 2003 EOADP Day Promote the ESA program.
February 4, 2003 STDP Day Promote the ESA program.
December 2 and 3, 2003 Technology Days Promote the ESA program - ESA presentation as part of broader series of presentations to promote achievements under the STDP program.
May 3 and 4, 2004 EO INFO Days Promote the ESA program.
June 16, 2004 Reception at the Canadian Embassy in Paris Celebrate 25 years of Canada/ESA cooperation with leaders of the national delegations of ESA member states.
June 18, 2004 Colloquium on EO Promote Canada/CNES (French Space Agency) cooperation, with ESA presence.
June 29, 2004 Galileo Concession Day Promote Galileosat program and introduce government representatives (from federal departments and the Government of Quebec) to potential Galileo concessionaires.

Of the events on the list, the only one to result in an actual consultation process was the consultative conference held at the CSA on September 10, 2001.

After that meeting, the stakeholders who attended were asked to complete a survey of their needs. In all, 41 responses were collected. The responses were examined by the ESA program manager. However, the manager told us that the survey did not provide a better idea of the industry's needs.

The ESA program manager also told us about some of the weaknesses of this process. No true consultations had actually taken place. Comments, opinions and needs were one-way, in that there was no direct feedback from managers of ESA or CSA programs.

According to the ESA program manager, it would be appropriate at this time to implement a consultation process with a view to preparing for the ministerial-level ESA Council meeting scheduled for December 2005. The objective of this process will be to make known Canada's position on optional programs to which the CSA would like to commit in coming years.

Given that these consultations only take place every four years or so, a new means of holding consultations with industry should be considered immediately in order to gain a better understanding of industry needs.

We also noted that, during the consultative meeting held on September 10, 2001, 10 Canadian firms had the opportunity to meet with an ESA representative. These bipartisan meetings allowed these Canadian firms to forge ties with ESA. However, the CSA was not present at the meetings and therefore, could not take advantage of this opportunity to learn about the needs of Canadian industry.

With regard to consultations with departments such as DFAIT and DND, these departments were generally consulted when appropriate, and such consultations were mentioned in submissions to the PRAB as required.

RECOMMENDATION

Implement a structured and transparent process for consultations with industry with a view to supporting decisions respecting optional programs selected by Canada.

Top of page

4.0 CONTRIBUTION ACCOUNTING

4.1 COMMITMENT OF FUNDS

Under the Canada/ESA Cooperation Agreement, Canada makes an annual contribution to expenditures entered under the General Budget and to optional programs.

Funds for the General Budget and optional programs are committed by service line in the following manner:

  • Optional Programs - EO
  • Optional Programs - Satellite Communications
  • GSTP 3 (General Support Technology Program) - Space Technology
  • Aurora - Space Exploration (under EO)
  • General Budget - EO
  • General Budget - Satellite Communications
  • General Budget - Space Technology

According to information obtained from the sector's financial analyst, the General Budget is broken down among three CSA service lines: EO, Satellite Communications and Space Technology. The financial analyst also indicated that this breakdown had been arrived at several years ago and had been prorated according to the optional programs existing at that time. This breakdown had not, however, been revised since then and, therefore, no long reflects reality in terms of the current proportions of optional programs. Consequently, the General Budget is no longer broken down in a representative manner with respect to the optional programs to which the CSA contributes. This has an impact on the CSA's financial results by service line. Table 3 illustrates the scope of this problem.

For example, if we were to take the $5,451,172 allocated to ESA's General Budget for the 2003-2004 fiscal year and break it down on a pro rata basis among optional programs, a variance in the CSA's financial results would be produced, as follows:

TABLE 3 - BREAKDOWN OF THE 2003-2004 GENERAL BUDGET

Service line Current breakdown Prorated breakdown Variance
Earth Observation $1,165,000 $2,301,634 $1,136,634
Satellite Communications $2,500,000 $2,858,319 $358,319
Space Technology $1,786,172 $291,219 ($1,494,953)
Total $5,451,172 $5,451,172 0

Breaking down the General Budget on a pro rata basis with respect to optional programs would have provided more accurate financial data that reflects the true situation.

Furthermore, under the new Program Activity Architecture (PAA), programs will be categorized differently, effective April 1, 2005. According to information from the Assistant Manager, Planning and Financial Analysis, generic expenditures incurred under certain programs including the General Budget will be broken down according to a predetermined percentage set by senior management in accordance with the CSA's long-term objectives and vision for each of these activities.

Once again, according to the Assistant Manager, Planning and Financial Analysis, the following percentages will be used for the breakdown of ESA General Budget expenses, effective April 1, 2005:

  • Earth Observation (EO): 46.39%
  • Space Sciences and Exploration (SSE): 41.24%
  • Satellite Communications (SC): 12.37%

Since participation in the General Budget is a prerequisite to being able to contribute to optional programs, it would be reasonable to consider breaking down the ESA General Budget in accordance with the optional programs to which the CSA contributes. Like the current breakdown method, the proposed approach that is to take effect April 1, 2005, does not allow the contribution to the General Budget to be broken down according to the relative size of optional programs to which the CSA contributes.

To illustrate the consequences of using one method versus another, we broke down the $7,144,000 contribution to the 2004-2005 General Budget among the three policy sectors according to the pre-established rates that will be in force for 2005-2006, then recalculated the breakdown, this time prorating it according to the projected contribution to optional programs for 2004-2005. The magnitude of the variances in Table 4 clearly shows the impact on financial data.

TABLE 4 - BREAKDOWN OF THE 2004-2005 GENERAL BUDGET

Policy sectors Pre-established %, effective April 1, 2005 Prorated Variance
Earth Observation $3,314,101 $2,525,783 ($788,318)
Space Sciences and Exploration $2,946,186 $291,692 ($2,654,494)
Satellite Communications $883,713 $4,326,525 $3,442,812
Total $7,144,000 $7,144,000 0

RECOMMENDATION

Review the rationale for treating the contribution to the General Budget as a generic expenditure related to various activities and instead consider breaking down these expenditures according to the actual contributions to optional programs, thereby obtaining more accurate financial data.

4.2 ADVANCE PAYMENT OF TRANSFER PAYMENTS

The TBS Policy on Transfer Payments states that any overpayments in the hands of a recipient at year-end or program-end must be accounted for and refunded if necessary.

Our review of payments to ESA under the Canada/ESA Cooperation Agreement showed that such payments-two payments of 40% of the annual total for the General Budget and optional programs paid in January and June respectively, plus the balance of 20% in October-were not fully used in the financial year in which they were made. Because cash flow does not always conform to the budgets initially set for optional programs, surpluses occur every year. These surpluses or portions thereof are used up in the following fiscal year. This obliges the CSA to account for these accumulated appropriations in its financial statements under an "Advance payments of transfer payments" account.

The item "Advance payments of transfer payments" in the CSA's financial statements to March 31 comes from two sources:

  • Accumulated appropriations to ESA to December 31;
  • Unused portion of the final payment (prorated for the remaining months in the period covered by the payment)

Table 5 below shows the amounts of advance payments of transfer payments recorded in the CSA's financial statements since 2002 and the small year-over-year variance.

TABLE 5 - STATUS OF "ADVANCE PAYMENTS OF TRANSFER PAYMENTS" ACCOUNT

Date Appropriations to ESA to December 31 Prorated payment calls to March 31 Total
March 31, 2002 $8 006 668 $3 677 287 $11 683 955
March 31, 2003 $10 171 467 $4 658 051 $14 829 518
March 31, 2004 $10 419 051 $1 194 365 $11 613 416

The current practice of using the accumulated amount of appropriations to December 31 as reported by ESA has the effect of overvaluing the "Advance payment of transfer payments" account. The accumulated amount of appropriations as of March 31 would better reflect the actual situation, but ESA policy is to report this information to December 31.

The financial analyst responsible for the ESA program also told us that ESA's accounting system will be changed effective January 1, 2005. The new accounting system will allow ESA to make payment calls that will be very close to actual expenditures to be incurred by optional programs in the period covered by the payment call. The analyst also told us that accumulated appropriations to December 31, 2004 would be completely used up by the first payment call in January 2005.

Starting in January 2005, the amount of transfer payments to be accounted for as advance payments will be the portion of the latest payment call not yet used, prorated for the remaining months in that period. The amount entered in the CSA's financial statements would thus be more representative of reality.

RECOMMENDATIONS

Monitor accumulated appropriations to ESA to ensure that all appropriations accumulated to date are actually used before the next payment call is made, thereby bringing the amount of accumulated appropriations to zero, as planned.

4.3 EXCHANGE RATES

The TBS's Policy on Transfer Payments states that, in order to deal with possible foreign exchange fluctuations, a department should consult with the Treasury Board Secretariat where it is proposing a payment based on an assessment made by an international organization for Canada's contribution to the organization or its projects.

This provision is part of a section of the Policy concerning Treasury Board submissions for approval of program terms and conditions. Since the CSA is currently preparing a submission to the TB concerning the renewal of the terms and conditions of the Canada/ESA Cooperation Agreement, it would be an opportune moment to consult with the TBS to solicit their input on how the issue of exchange rates should be approached in the submission and eventually handled in terms of the day-to-day management of the program.

One of the recommendations of the May 1997 evaluation report prepared by the Hickling Corporation addressed the exchange rate aspect of the contribution to ESA. The following statement appeared in the report:

A contingency fund should be provided in the CSA budget for ESA to allow for exchange rate fluctuations which affect the cost of Canadian participation in ESA programs. The size of this fund should be assessed in light of the adoption of the new financial system based on the ECU.

Our discussions with the manager of the Canada/ESA Cooperation Program revealed that, to date, no mechanism had been set up to protect the CSA against fluctuations in exchange rates.

The recommendation of the May 1997 Hickling report is still valid, and the need to implement it is even more urgent, since exchange rates have been climbing steadily for several years, as the following table shows:

About three years ago, the program manager used an estimated exchange rate of approximately 1.60 for the budget estimates, whereas the actual rate was roughly 1.40. This allowed for a reasonable tolerance level to cover the risks of fluctuations in exchange rates, inflation and possible cost overruns in certain programs. Today, budget estimates are calculated using an exchange rate of 1.65, but actual exchange rates are currently ranging between approximately 1.60 and 1.65. The current situation thus leaves little or no room for covering the previously mentioned risks. This makes it even more urgent to set up a mechanism for covering the risks of exchange rate fluctuations and protecting the CSA against program cost overruns.

There are a number of different mechanisms that could be used to protect against exchange rate fluctuations, such as using foreign exchange contracts or adding the Canada/ESA Cooperation Agreement to the CSA's risk management framework. Furthermore, before an exchange rate fluctuation management mechanism is implemented, it is essential that the TBS be contacted, as stated in the TBS Policy on Transfer Payments, to obtain as much information as possible so that an informed decision can be made.

RECOMMENDATION

In consultation with the Treasury Board Secretariat, set up a mechanism to mitigate the risks of exchange rate fluctuations.

4.4 FINANCIAL CONTROL OF CONTRIBUTIONS TO OPTIONAL PROGRAMS

The TBS Policy on Transfer Payments states that departments must establish policies and procedures to ensure that effective financial and program controls are designed and implemented within departmental transfer payment programs.

Appendix 1 of the TB decision stipulates that Canada's financial contribution to an optional program may be increased up to 120% of the original approved envelope. It also states that Canada may decide to withdraw from the optional program once the approved budget exceeds 120% of the original envelope.

Our discussions with the manager of the Canada/ESA Cooperation Program and the sector's financial analyst revealed that there were no mechanisms in place to ensure compliance with these provisions.

When an Arrangement is signed for a particular optional program, Canada is aware of the amount it has committed, in euros (or AUs and ECUs, for older programs). The amount of Canada's contribution is identified in the optional program's Declaration. Payment calls are then made in euros and converted into Canadian dollars for accounting entry purposes. Contributions to which Canada commits are also generally spread out over a long period, until 2008-2009 in most cases.

Currently, the sector's financial analyst follows up on each payment, including annual payments, made in Canadian dollars for each optional program. However, there is no reconciliation of the total amount called for in euros and the amount Canada committed. It is therefore impossible to tell if optional programs have incurred cost overruns.

In order to ensure compliance with the clause that states that Canada's financial contribution to an optional program may be increased up to 120% of the original approved envelope and that Canada may withdraw if that limit is exceeded, the program manager must monitor the amounts paid out for each optional program. The manager may, however, call upon the sector's financial analyst to submit the necessary reports to facilitate this monitoring.

Our discussions with the sector's financial analyst revealed that the current analyst's predecessor had already complied such information. However, the reports were not completely up to date, and the information had not been forwarded to the program manager. The financial analyst also told us that he had recently received two documents from ESA's finance directorate that allow actual expenses in euros to be compared against amounts committed in euros. One report is a compilation of total expenditures in euros by optional program and by member state since the very beginning of optional programs in progress; the other is a compilation of actual expenditures and estimates to the optional programs' end, once again by member state. This information will allow the ESA program manager to monitor changes in optional programs and any cost overruns that might occur.

RECOMMENDATION

Monitor optional program expenditures to ensure that payments match the amounts in foreign currency to which the CSA has committed under its Arrangements for optional programs and make it possible to detect cost overruns.

Top of page

5.0 RISK-BASED AUDIT FRAMEWORK (RBAF)

5.1 IMPLEMENTATION OF THE RBAF

The TBS Policy on Transfer Payments stipulates the following:

  • Treasury Board submissions for program approval of terms and conditions for... contributions should include... a risk-based framework for audit of recipients of contributions;
  • Departments must develop a risk-based audit framework for the audit of contributions;
  • It is government policy... to manage transfer payments in a manner sensitive to risks, complexity, accountability for results and economical use of resources.

According to the TBS Risk-Based Audit Framework Guide, these requirements are met by developing a risk-based audit framework (RBAF).

In April 2003, an RBAF was drawn up for the Canada/ESA Cooperation Agreement. The RBAF was not, however, submitted to the TB at that time. The manager of the Canada/ESA Cooperation Program planned to submit the RBAF to the TB with the renewal of the program's terms and conditions in April 2005.

The RBAF primarily addressed two elements: the risk-management plan and the audit framework. Actions to implement the audit framework were carried out through this audit of the program. This audit activity was also provided for in Appendix 1 of the TB decision, "Program Terms and Conditions", under item 8, "Evaluation and Audit". However, the risk management plan has not been implemented. No action has been taken to mitigate identified risks.

The program manager told us that the RBAF has not been implemented because it has not yet been submitted to the TB for approval. Once it has been approved by the TB, the RBAF will become an integral part of the terms and conditions of the program and will have to be implemented.

Since the RBAF is a management tool that helps managers carry out their duties and honour their obligation to be accountable, there is no need to wait for the RBAF to be approved by the TB before implementing it.

RECOMMENDATIONS

  1. Implement the RBAF.
  2. Monitor the risk management plan on a regular basis to keep track of changes in identified risks and ensure identified risk management actions are implemented.

5.2 PROGRAM RISK MANAGEMENT PLAN

During our audit, we reviewed the program risk management plan included in the RBAF. We looked at the following aspects of each identified risk:

  • The probability of the risk occurring;
  • The consequences, should the risk occur;
  • The measures identified to mitigate the risks;
  • The tangible actions to be taken to reduce the risk to a level the CSA deems acceptable.

Our review led to the following observations:

  • The measures to be implemented, as well as the tangible actions to manage the risk of exchange rate fluctuations, should be
    • reviewed, given that they refer to a TB program about which we have been unable to obtain any information, and
    • revised, in light of the recommendation made in section 4.3 of this audit report.
  • One of the identified measures for mitigating the risks arising from staff turnover is the retention of appropriate administrative records. This makes the recommendation in section 2.1 of this audit report even more important.
    • The actions regarding records management identified in the RBAF should be revised.
  • In the "Program Governance" section, the first identified risk refers to Canada's level of participation in an optional program that is below the critical mass required to achieve industrial returns and sustainable socio-economic benefits.
    • In order to justify the level of risk identified in the RBAF, the critical mass must be defined. If it is not, any evaluation of that risk will be arbitrary.
    • The action identified for mitigating that risk, ie, developing an evaluation methodology and procedures, has not been implemented.
  • The action for mitigating the risk of rising program costs refers to ESA signing firm-price contracts. However, the CSA has no influence over the awarding of such contracts.
    • This item should instead refer to CSA's option to withdraw from any optional program when cost overruns exceed 20%.
  • The "Information and Management Systems" section refers to a 2003 pilot project to evaluate whether performance measurement indicators identified in the results-based management and accountability framework (RMAF) are adequate.
    • Our audit showed that no pilot project was carried out, and no data collection system was set up.
  • Having completed our audit, we find that the deemed level of risk of a paper and electronic records management system should be raised from low to medium (see section 2.1 of this audit report).

We gave the program manager an annotated version, with comments, of the risk management plan included in the RBAF.

RECOMMENDATION

Revise the RBAF's risk management plan to take the above mentioned comments into account.

Top of page

6.0 RESULTS-BASED MANAGEMENT AND ACCOUNTABILITY FRAMEWORK (RMAF)

6.1 IMPLEMENTATION OF THE RMAF

The TBS Policy on Transfer Payments stipulates that "Treasury Board submissions for program approval of terms and conditions... for contributions should include... a results-based accountability framework including: performance indicators, expected results and outcomes, methods for the reporting on performance, and evaluation criteria to be used in the assessment of the effectiveness of the transfer payments. Furthermore, the TBS Evaluation Policy states that "[m]anagers have a responsibility to monitor the performance of policies, programs and initiatives - to make sound decisions and to report on performance."

In March 2003, a results-based management and accountability framework (RMAF) was drafted for the Canada/ESA Cooperation Agreement. As was the case with the RBAF, the RMAF was not submitted to TB at that time. Once again, the manager of the Canada/ESA Cooperation Program planned to submit the RMAF to the TB upon the expected renewal of the program's terms and conditions in April 2005.

As required by the PMAF and Appendix 1 of the TB decision, a program evaluation was carried out. The evaluation coincided with our audit.

The program manager told us that, since the RMAF had not yet been submitted to the TB for approval, it had not been implemented. Consequently, no data collection system had been implemented to date. Such a system is required to be able to measure the expected results of programs in terms of the performance measurement indicators set out in the RMAF.

Like the RBAF, the RMAF is a management tool that helps managers to carry out their duties and honour their obligation to be accountable. There is no need to wait for the RMAF to be approved by the TB before implementing it, especially since a pilot project to help evaluate the adequacy of the RMAF's performance measurement indicators had been planned for 2003.

RECOMMENDATIONS

  1. Implement the RMAF.
  2. Set up a data collection system to measure the expected results of programs in terms of the performance measurement indicators set out in the RMAF.

6.2 CONTENT OF THE RMAF

A formative evaluation of the Canada/ESA Cooperation Agreement took place at the same time as our audit.

Given that the CSA is in the process of preparing a TB submission regarding the renewal of the terms and conditions of the Canada/ESA Cooperation Agreement, the RMAF should be revised and amended, if need be, taking into account the results of the formative evaluation to be submitted at the same time as our audit report.

RECOMMENDATION

Revise the RMAF, taking into account the results of the formative evaluation of the Canada/ESA Cooperation Agreement.

Top of page

APPENDIX A - AUDIT OBJECTIVES AND CRITERIA

Objective 1 Ensure that adequate policies, procedures and control mechanisms for verifying the eligibility and admissibility of payment requests are in place.
Criterion 1.1 Current procedures ensure that the contribution to the General Budget complies with the program's terms and conditions.
Criterion 1.2 Payment calls are admissible and are actually paid out (General Budget and optional programs).
Criterion 1.3 Payments relating to optional programs are in line with the approved funding level.
Objective 2 Ensure that the Canada/ESA Cooperation Agreement is administered in compliance with the terms and conditions of the Cooperation Agreement.
Criterion 2.1 Current policies and methods allow managers and financial officers to meet the terms and conditions of the Cooperation Agreement.
Criterion 2.2 Current procedures ensure that the optional programs selected are those that are most likely to contribute to the achievement of the objectives of the Cooperation Agreement.
Criterion 2.3 The different phases of Canada's participation in ESA optional programs (selection process, program approval document (PAD)/program approval submission (PAS), approval process, eligibility of costs, etc) are carried out in compliance with the terms and conditions of the relevant components of the Cooperation Agreement.
Criterion 2.4 Based on expected industrial spin-offs, the selection process for optional programs includes consultations with the following:
  • relevant federal departments and organizations (such as DFAIT and DND);
  • Canadian firms in the space industry;
  • concerned stakeholders.
Objective 3 Ensure that contributions are managed with due diligence and in compliance with the relevant legislation and policies.
Criterion 3.1 Responsibilities are divided up adequately among officers in charge of managing the program.
Criterion 3.2 The program is delivered in compliance with TBS policies regarding financial management and transfer payments.
Criterion 3.3 Approval authority is only delegated in conjunction with appropriate control mechanisms and a proper division of responsibilities.
Criterion 3.4 The program is managed efficiently.
Objective 4 Obtain assurances that the current risk management framework is efficient.
Criterion 4.1 The risk management framework allows the following to be carried out adequately:
  • identifying internal and external events or factors that could compromise the program's execution;
  • measuring the probability that an event will occur;
  • evaluating the potential impact of unfavourable event on a program;
  • setting out actions to be implemented to mitigate identified risks.
Criterion 4.2 The process for monitoring risks allows program managers to evaluate the effectiveness of evaluation and response plans, recognize unexpected changes and deal with new risks using appropriate intervention measures.
Criterion 4.3 Staff assigned to the management and administration of program components have the knowledge and skills required to deal with risks.
Objectif no. 5 Obtain assurances that the results-based management and accountability framework (RMAF) is efficient.
Criterion 5.1 An RMAF is in place.
Criterion 5.2 Performance measurement indicators are relevant.
Criterion 5.3 Data collection systems are efficient.

Top of page

APPENDIX B – MANAGEMENT ACTION PLAN

Ref.
Recommendation
Designated Authority Action Plan Details Timetable
Organization Individual
2.0 Administrative records
2.1 Audit trail

Ensure that complete administrative records are kept for every optional program in order to maintain an adequate audit trail and support decision making.

Space Technology

Manager

The ESA already gives all delegations access to a Web-based electronic records keeping system (DODIS) that contains all official documents issued by ESA in the last 15 years and is updated on a daily basis. Although it is highly unlikely that DODIS would become unavailable (since it serves not only Canada but all member states and remains the main mechanism for consulting ESA records), the ESA Program Office is committed to maintaining complete administrative records for each optional program (more specifically, a copy of the program's Declaration, Arrangement and any other record relevant to decision making) in order to maintain an adequate audit trail and support decision making.

March 31, 2005

3.0 Selection and approval process for optional programs
3.1 Program approval document (PAD)
  1. Review the relevance of including an item respecting "ESA position on Canadian participation in an optional program" in the TB submission for renewal of the program's terms and conditions.







  2. Ensure that all required information is included in PADs in accordance with the program terms and conditions approved by the TB.

Space Technology

ESA Program Office

Management agrees that there is little point in including an item respecting "ESA position on Canadian participation in an optional program" in the TB submission for renewal of the program's terms and conditions, since it is assumed that ESA agrees. Otherwise, there would be no reason for making a submission to the PRAB.

 

Agreed.

To be put in practice when the next submission to the PRAB is made.

 

 

 

 

 

 

To be put in practice when the next submission to the PRAB is made.

3.2 Consultation process

Implement a structured and transparent process for consultations with industry with a view to supporting decisions respecting optional programs selected by Canada.

Space Technology

ESA Program Office

A structured and transparent process for consultations with industry to reach a consensus with a view to supporting decisions respecting optional programs selected by Canada will be implemented with the coming round of consultations required before selecting the optional programs in which Canada will want to participate.

The ESA Program Office will apply the recommendation.

The process will be implemented in time for the ministerial-level ESA Council meeting in December 2005.

4.0 Contribution accounting
4.1 Commitment of funds

Review the rationale for treating the contribution to the General Budget as a generic expenditure related to various activities and instead consider breaking down these expenditures according to the actual contributions to optional programs, thereby obtaining more accurate financial data.

Space Technology

Integrated Management

Strategic Development

ESA Program Office

The rationale for considering the contribution to the General Budget as a generic expenditure relating to various activities will be reviewed, and consideration will be given to a method that breaks down these expenditures according to the actual contribution to optional programs, thereby giving us more accurate financial data.

The ESA Program Office will apply the recommendation in cooperation with the responsible officials within the CSA.

This will be done when the next ARLU is prepared.

The timetable for implementation is not wholly dependent on the ESA program manager. This item will be implemented in cooperation with Integrated Management (Financial Planning), who are ultimately responsible for the breakdown.

4.2 Advance payment of transfert payments

Monitor accumulated appropriations for ESA to ensure that all appropriations accumulated to date are actually used before the next payment call is made, thereby bringing the amount of accumulated appropriations to zero, as planned.

Space Technology

Integrated Management

ESA Program Office

The ESA Program Office and Finance have already made every effort to make payments owed to ESA. The ESA Program Office will use up all appropriations accumulated to date with ESA when the next payment call occurs, thereby bringing the amount of accumulated appropriations to zero.

The next payment call will be in January 2005.

4.3 Exchange rates

In consultation with the Treasury Board Secretariat, set up a mechanism to mitigate the risks of exchange rate fluctuations.

Space Technology

ESA Program Office

The ESA Program Office (in cooperation with other CSA entities with jurisdiction in this field) will take care of setting up a mechanism for mitigating the risks of exchange rate fluctuations, in consultation with the TBS. The CSA's current risk management mechanism could be the mechanism of choice for managing this type of risk.

March 31, 2005

4.4 Financial control of contributions to optional programs

Monitor optional program expenditures to ensure that payments match the amounts in foreign currency to which the CSA has committed under its Arrangements for optional programs and make it possible to detect cost overruns.

Space Technology

ESA Program Office

A mechanism for monitoring expenditures will be introduced.

March 31, 2005

5.0 Risk-based audit framework (RBAF)
5.1 Implementation of the RBAF
  1. Implement the RBAF.








  2. Monitor the risk management plan on a regular basis to keep track of changes in identified risks and ensure identified risk management actions are implemented.

Space Technology

ESA Program Office

The RBAF will be implemented once it has been approved by the TB. Approval should come sometime in the first quarter of 2005 (Jan-Mar 2005).


The ESA Program Office will apply the recommendation. The requested monitoring will take place with one or two control points during the year (to be determined).

Implementation on an ongoing basis.
The RBAF will be implemented immediately once it has been approved (before April 1, 2005).

Implementation on an ongoing basis.

5.2 Program risk management plan

Revise the RBAF's risk management plan to take the comments in section 5.2 of this report into account.

Space Technology

ESA Program Office

The plan will be reviewed before the TB submission currently being prepared is finalized.

By December 8, 2004.

6.0 Results-based management and accountability framework (RMAF)
6.1 Implementation of the RMAF
  1. Implement the RMAF.



















  2. Set up a data collection system to measure the expected results of programs in terms of the performance measurement indicators set out in the RMAF.

Space Technology

ESA Program Office

The ESA Program Office will apply the recommendation. The RMAF will be implemented once it has been approved by the TB. Approval should come sometime in the first quarter of 2005 (Jan-Mar 2005).

Mechanisms and resources for implementing the RMAF will be put in place gradually throughout the coming year.

 

The ESA Program Office will apply the recommendation. A data collection system will be set up (details to be announced later) so that we can measure the expected results of the program in terms of the performance measurement indicators identified in the RMAF.

Implementation on an ongoing basis.
The RMAF will be implemented immediately once it has been approved (before April 1, 2005).

 

 

 

 

October 1, 2005.

6.2 Content of the RMAF

Revise the RMAF, taking into account the results of the formative evaluation of the Canada/ESA Cooperation Agreement.

Space Technology

ESA Program Office

The ESA Program Office will apply the recommendation. The plan will be reviewed before the TB submission currently being prepared is finalized.

By December 8, 2004.