Language selection

Search

Top of page

Audit of the Management Framework for the Enabling Technology Development Program

Audit Report

Project No. 18/19 01–01

Prepared by the
Audit and Evaluation Directorate

Table of contents

Acronyms

AIAC

Aerospace Industries Association Canada

AO

Announcement of Opportunity

CSA

Canadian Space Agency

eWP

Electronic Work Plan

FAA

Financial Administration Act

G&C COE

Grants and Contributions Centre of Expertise

IIRB

Integrated Investment Review Board

IM/IT

Information Management and Information Technology

IPCG

Innovation Planning and Commercialization Group

IPMTT

Intellectual Property Management and Technology Transfer

PCA

Procurement and Contract Administration

PSPC

Public Services and Procurement Canada

RBAP

Risk-Based Audit Plan

RFP

Request for Proposals

SCDP

Space Capacity Development Program

SE

Space Exploration

SST

Space Science and Technology

STDP

Space Technology Development Program

SU

Space Utilization

TB

Treasury Board

TDR

Formal Technology Development Request

TPWG

Technology Planning Working Group

1.0 Summary

1.1 Audit objective

The purpose of the audit project was to determine if a management framework is in place to enable the Enabling Technology Development Program to achieve its objectives and comply with the relevant legislation, policies, regulations and guidelines issued by the Canadian Space Agency (CSA) and central agencies.

1.2 Audit opinion

In our opinion, the management framework in place enables the program to meet its objectives and comply with the relevant legislation, policies, guidelines and regulations issued by the CSA and central agencies. Opportunities for improvement were nevertheless identified with respect to activity planning and accountability.

1.3 Statement of assurance

As Chief Audit Executive, I am of the opinion that sufficient and appropriate audit procedures were followed, and that audit evidence was collected to support the accuracy of the opinion provided in this report. This opinion is based on a comparison of the situations that existed at the time of the audit with the pre-established audit criteria determined in consultation with management. This opinion applies only to the specific entity examined. Evidence was gathered in accordance with the internal audit policy, directives and standards issued by the Treasury Board (TB). The procedures that were followed comply with the professional standards of the Institute of Internal Auditors. The evidence gathered is sufficient to convince senior management of the validity of the opinion derived from the internal audit.

1.4 Summary of findings

The audit showed that the Enabling Technology Development Program has established best practices in planning its activities. Many consultations are carried out, and a process for approving decisions is in place. Contribution agreements and contracts are awarded according to a structured, controlled process. Once awarded, the agreements and contracts are monitored adequately and results are tracked. We also found that the program complies with the relevant legislation, policies, regulations and guidelines issued by the CSA and central agencies. We reviewed a sample of contribution agreements, contracts, amendments and payments, and found they were consistent with existing rules.

We identified opportunities for improvement with respect to program activity planning and accountability. We found that there was no clearly defined link between the program's research and development (R&D) activities and organizational planning. Furthermore, no information was available regarding the long-term outcomes of R&D activities funded through contracts. The proposed opportunities for improvement are as follows:

  1. Determine when would be the best time to consult the CSA Executive Committee (EC) with respect to organizational priorities for R&D activities in order to ensure earlier EC involvement in the planning process, thereby establishing a clearer link between program activities and organizational priorities;
  2. Continue to put in place elements that will improve the accountability of Space Technology Development Program (STDP) R&D activities in order to support planning and decision making for future investments.

Original signed by the Chief Audit Executive

Signature of the Chief Audit Executive

Audit team members

  • Dany Fortin
  • Louis Martel
  • Johanna Gailer
  • Fatima Raveen

2.0 Audit report

2.1 Background

The program covered by this audit is the Enabling Technology Development Program, which includes the activities of the Space Technology Development Program (STDP) and the services offered by the Intellectual Property Management and Technology Transfer (IPMTT) group. The STDP is a CSA R&D program that provides financial assistance through contracts and contribution agreements for organizations and stakeholders that are selected according to their capacity to develop specific space technologies. The two fundamental and interrelated goals of the STDP are to develop, implement and manage R&D contracts related to enabling technology development in order to support the future needs of the Canadian Space Program; and to support industrial capability-building through the development of new products, services, processes and know-how. Some 60 contracts and 50 contribution agreements were active at the time of the audit.

The program's financial data and number of employees appear in the tables below.

Expenses -
Actual
M $
-
Actual
M $
-
Actual
M $
-
Actual
M $
-
Actual
M $
-
Forecast
M $
-
Forecast
M $
STDP 10.8 11.1 19.7 27.4 24.4 22.6 22.6
IPMTT 0.0 0.8 0.5 0.6 0.6 0.8 0.8
Total 10.8 11.9 20.2 28.0 25.0 23.4 23.4
Employees -
Actual
-
Actual
-
Actual
-
Actual
-
Actual
-
Forecast
-
Forecast
STDP - Dedicated 4.4 6.3 6.7 6.8 7.8 7.9 8.0
STDP - Matrix 6.3 7.1 9.3 3.7 3.2 2.8 3.2
STDP - Total 10.7 13.4 16.0 10.5 11.0 10.7 11.2

STDP Expenses -

Active Contracts in - (Professional Services)
Year of Issue of Contracts Total RFP 5
-
RFP 6
-
RFP 9
-
RFP 10
-
RFP 12
-
RFP 13
-
RFP 14
-
RFP 15
-
Others
Expenses 13.7 M$ 0.1 M$ 0.4 M$ 6.3 M$ 1.7 M$ 0.3 M$ 2.0 M$ 2.4 M$ 0.3 M$ 0.2 M$
Number of contracts with expenses in / 59 2 4 21 11 1 2 11 1 6
Number of contracts issued in /               13    
Active Contribution Agreements in -
Year of Issue of Contribution Agreements Total AO 2
-
AO 3
-
AO 4.1
-
AO 4.2
-
AO 4.3
-
Unsolicited
Expenses 8.2 M$ 0.0 M$ 5.3 M$ 1.6 M$ 0.5 M$ 0.1 M$ 0.7 M$
Number of agreements with expenses in / 53 2 18 12 16 4 1
Number of agreements issued en /       12 18 5 1
Salary 2.1 M$            
Travel, Office Expenses, Students, etc. 0.4 M$            
Total 24.4 M$            

2.2 Audit objective, scope and approach

Objective

This audit project is part of the - to - Risk-Based Audit Plan (RBAP) approved by the CSA Audit Committee. The objective of this audit was to determine if a management framework is in place to enable the program to achieve its objectives and comply with the relevant legislation, policies, regulations and guidelines issued by the CSA and central agencies.

Scope

The audit focused on all program expenditures, as well as the existing systems and processes for activity planning, monitoring and accountability. The audit examined activities carried out from to .

Services delivered by the IPMTT group were excluded from this audit. An audit of these services is currently planned for -, as indicated in the RBAP for - to -.

Approach

The audit criteria were established in accordance with best management practices and the requirements set out in the legislation, policies, regulations and guidelines issued by the CSA and central agencies. The audit used a number of audit processes, including interviews and the review of documents, operating expenses and contract files.

We reviewed the cycle, which included the following:

  • The preparation and publication of an Announcement of Opportunity (AO) and a Request for Proposals (RFP);
  • The screening, assessment and selection of proposals;
  • The awarding and monitoring of contribution agreements and contracts.

We also reviewed

  • Ten payments related to five contribution agreements and five contracts;
  • Five amended contracts;
  • Five travel expenditures.

The samples were selected according to the auditors' judgment.

It should be noted that the audit objective and criteria were discussed with management. The audit criteria and sub-criteria are outlined in Appendix A.

2.3 Findings, recommendations and management response

Expected results

To determine whether the existing management framework enables the program to meet its objectives and comply with the legislation, policies, regulations and guidelines issued by the CSA and central agencies, we expected to find the following elements:

  • Activities that are planned effectively;
  • Activities and resources used that are monitored;
  • Activities that are subject to accountability.

2.3.1 Activity planning

Audit objective: The purpose of the audit project was to determine if a management framework is in place to enable the program to achieve its objectives and comply with the relevant legislation, policies, regulations and guidelines issued by the CSA and central agencies.

Findings
Criterion 1

Program activities are planned appropriately.

Condition

Conclusion about the criterion

The audit showed that program activities are planned appropriately. We nevertheless identified one opportunity for improvement that involved integrating the planning of the program's R&D activities with CSA organizational planning.

Investment priorities

We found that stakeholder needs are identified on a regular basis, and the selected areas for investment reflect the identified needs and priorities. Investment plans are prepared, approved and reviewed on a regular basis.

Strategic planning roles and responsibilities

The STDP is under the authority of the CSA Space Science and Technology (SST) Directorate. Strategic planning for this directorate is carried out by the Innovation Planning and Commercialization Group (IPCG). This group uses an integrated stakeholder approach. The key STDP stakeholders are Canadian space-industry players (businesses and universities) and CSA program sectors, namely Space Exploration (SE), Space Utilization (SU) and SST. Planning begins by identifying stakeholder needs in order to determine the program's investment priorities.

The CSA's internal Technology Planning Working Group (TPWG) is made up of representatives from the three CSA program sectors and the Policy and Programs and Integrated Planning Directorates. The TPWG is the main CSA committee for space technology development. Investments are endorsed by the SST Steering Committee and approved by the Integrated Investment Review Board (IIRB).

Needs identification

Consultations are occasionally held with Canadian space-industry representatives to discuss their needs. These consultations are carried out according to different streams. One consultation stream was with the Aerospace Industries Association Canada (AIAC). A joint CSA/AIAC working group has been created. The working group has been tasked with developing and suggesting changes to the CSA STDP in order to make it more responsive to the current commercial environment in the space sector. Another stream was with the Canadian space industry and universities. Its goal was to present a draft of the new STDP framework for feedback. In , 90 of the 200 invited organizations and universities took part in webinars to endorse the joint recommendations of the CSA/AIAC working group. A third stream consisted of meeting with program sectors at the CSA and other key departments to identify challenges and discuss potential solutions. The recommendations emerging from these consultations were implemented, to the extent possible, and led to a number of changes in the STDP. For example, specific funds now target small and medium-sized enterprises. Moreover, in , Euroconsult submitted a report to the CSA entitled Technology Trend Assessment of the Canadian Space Sector. Euroconsult assessed technological trends in order to identify technologies most likely to have a major impact on the space sector over the next 5 to 10 years, and to determine their relevance for the Canadian space sector. These trends will help to inform CSA investment priorities.

The investment priorities of CSA program sectors, in line with their respective objectives, are established following their own consultations with the industry. These needs are submitted to the STDP through the internal consultation exercise, along with an estimated value of the work. Before an RFP is officially launched, the industry is informed of the CSA's intention to launch this AO, the details of the expected technological development (as well as the allocated budget). Industry is also asked to validate industry interest, the content of the work and the budget allocation. Since the RFP 16 (/), business needs must be validated by the IIRB before being submitted to the TPWG and the RFPs be posted.

Prioritization of areas for exploration and approval of investment plans

Announcement of Opportunity (Contributions)

In order to meet the priorities of Canadian space stakeholders, the CSA helps to position the sector through contributions to technology development for potential commercialization opportunities. A broad range of projects is funded through this type of financing vehicle. Specifically funded projects are based on industry needs.

The contributions awarded through AO 4 in targeted three project categories: basic space R&D for projects that are expected to produce economic benefits in the short and medium term (2 to 5 years); basic space R&D for projects from small businesses (50 or fewer employees) that are expected to produce economic benefits in the medium and long term (5 to 10 years); and, basic space R&D for feasibility studies for projects that are expected to produce economic benefits in the medium and long term (5 to 10 years).

A selection process is used to review the proposals received in response to an advertisement. Proposals that meet the mandatory criteria listed in the AO are screened in for assessment. Once this assessment is completed, the assessed proposals are listed in order of their scores and submitted to the SST Steering Committee for review and endorsement. Funding is then allocated to the best-rated proposals until the budget is fully allocated. The final list is approved by the IIRB. The approvals are recorded in logs to this effect.

Request for proposals (contracts)

This funding vehicle addresses the technological uncertainties related to future CSA missions. The projects funded by the program are based on needs identified by CSA program sectors for the development of potential projects. The CSA is the client for R&D activities carried out through the contracts awarded.

A process for prioritizing the technologies to be developed is in place and consists of a structured method for identification, prioritization and selection by the TPWG. Technology identification begins with the presentation, by each CSA program sector, of their needs in relation to their mission plans, roadmaps and according to the budget amounts allocated to them. Every technological need is associated with a formal Technology Development Request (TDR).

The TDRs are sent to the TPWG, which includes representatives from the three CSA program sectors, the Policy Directorate and the Programs and Integrated Planning Directorate. At this stage, the TPWG's work consists of validating the process used by the sectors and aligning the selection criteria. A list of priority technologies prepared by the TPWG is recommended to the SST Steering Committee and sent to the STDP for presentation to and approval by the IIRB (see process at Appendix B). In recent years, the STDP's budget for the RFP has been equally divided between the sectors SU and SE.

Following the presentation to the IIRB, the STDP manager defines the actions that will be taken to fund the development of prioritized technologies. A statement of work is created for each technology, and a development value is assigned to each statement of work. A letter of interest is then posted by Public Services and Procurement Canada (PSPC) to obtain feedback from interested contractors on the direction taken by the CSA and the estimated amounts. This feedback is used to refine the statement of work, among other things. The IIRB approves all contracts over $75K. The approvals are recorded in logs to this effect.

Although many consultations are held with stakeholders and a rigorous approval process is used, it is unclear how R&D investments through the awarding of contracts by the STDP align with organizational planning. The initial prioritization is currently carried out by the sectors, and the IIRB is consulted fairly late in the process. We feel that prior consultation with the EC would better integrate organizational priorities and improve the effectiveness of the planning process.

Financial planning

We found that an appropriate planning of financial resources is carried out. The roles and responsibilities for planning financial resources are well defined and applied. The program has no capital expenditures. Specific budgets are established for contribution agreements and for contracts.

The budgets are continuously updated. Management monitors financial planning on a monthly basis. In -, 82% of the original budget for contributions was used. The available amounts were reallocated to other CSA activities throughout the year.

With respect to contracts, a net amount of $4.3M was added to the initial program budget for the - fiscal year. This amount was mainly used to fund additions to existing projects. These funds were allocated to the program during the year.

Recommendation
  1. Determine when would be the best time to consult the CSA Executive Committee with respect to organizational priorities for R&D activities in order to ensure earlier EC involvement in the planning process, thereby establishing a clearer link between program activities and organizational priorities.
Identified responsibility
Organization

Programs and integrated Planning Directorate

Function

Executive Director, Programs and Integrated Planning Directorate

Management response

Obtaining the list of strategic priorities (including the list of priority missions) related to research and development activities and aligning with science priorities prior to the commencement of the STDP investment planning process would help make the process more focused and strategic.

Management action plan
Action plan details

The Programs and Integrated Planning Directorate, in collaboration with the Finance, Space Utilization and Space Exploration sectors and the Science Advisor, will be responsible for defining and updating strategic priorities (including the list of priority missions), while taking into account various science and mission roadmaps and submitting them to EC for approval. Once the priorities are approved, the STDP investment planning process can begin. The Program and Integrated Planning Directorate and Finance will update the processes to maintain the list of priorities on an ongoing and timely basis.

Deadline
  • Priority list:
  • Process update:

2.3.2 Program activity monitoring

Audit objective: The purpose of the audit project was to determine if a management framework is in place to enable the program to achieve its objectives and comply with the relevant legislation, policies, regulations and guidelines issued by the CSA and central agencies.

Findings
Criterion 2

Program activities are monitored appropriately.

Condition

Conclusion about the criterion

The audit showed that program activities are monitored appropriately. Roles and responsibilities are clearly defined. Day-to-day activities are implemented, supervised, measured and compared with forecasts. Activities are reported and approved in accordance with internal CSA policies and directives. Standard, structured processes and procedures are applied. Furthermore, the preparation and publication of AOs and RFPs, and the intake, analysis, assessment and selection of proposals follow established processes and procedures. These are applied in accordance with program terms and conditions, and in compliance with the applicable legislation, policies, regulations and guidelines.

Roles and responsibilities are clearly defined

We found that roles and responsibilities are clearly defined and communicated. The senior STDP administrator and two managers are responsible for day-to-day activities. These three people lead activities related to the planning, drafting and posting of AOs/RFPs, as well as the process for evaluating proposals and awarding funding. The contributions process is supported by the CSA Grants and Contributions Centre of Expertise (G&C COE). As for contracts, only PSPC has the contracting delegation to award R&D contracts for the federal government. The program has signed a memorandum of understanding with PSPC to provide procurement and support services for the STDP. A single operational point of contact at PSPC has been assigned for the entire STDP.

During the assessment process, evaluators in the subject areas of each proposal are recruited inside and outside the CSA, according to the expertise sought. One person is put in charge of each proposal. This person is responsible for leading the group of evaluators and drafting the final comments for each proposal assessed by the group.

After the agreements and contracts are awarded, a project manager is assigned to each file. This person is responsible for following up with organizations and reviewing claims for payment, among other things. The level of follow-up varies according to the nature of the agreement and the organization's risk level.

Day-to-day activities are supervised, measured, compared with forecasts, reported and approved in accordance with internal CSA policies and directives. Furthermore, the preparation and publication of AOs and RFPs, and the intake, analysis, assessment and selection of proposals follow established processes and procedures that comply with program terms and conditions, as well as relevant legislation, policies, regulations and guidelines

We found that adequate control measures are in place with respect to the implementation, oversight, approval and measurement of activities. The preparation and publication of AOs and RFPs, and the intake, analysis, assessment and selection of proposals follow rigorous standard processes and procedures. Their application complies with program terms and conditions, as well as the relevant legislation, policies, regulations and guidelines.

Assessment of proposals

We reviewed the processes used to assess AOs and RFPs. We found that the expected procedures and controls are applied. The evaluation work was well structured and supervised. The necessary approvals were obtained, and the final assessment reports were adequate.

The STDP group is responsible for the process used to assess bids. The actual assessment work is carried out with the support of appropriate technical or scientific authorities. Assessment plans have been prepared and applied. These plans include a description of the assessment process and schedule, as well as a list of the evaluation committees involved and the methodology used. Proposals are assessed using the UNITAS application. This application is well structured, easy to use and includes links to documents that assessors need.

Evaluators with expertise in the subject areas of the proposals were recruited inside and/or outside the CSA. There are at least two "technical" evaluators and two "commercial" evaluators for each proposal. Preliminary meetings were held to inform assessors of the assessment process and schedule.

Once the proposals were assessed, review meetings were held to compare each evaluator's results and reach a consensus. The final comments and scores are sent to all successful and unsuccessful bidders. The substantive manager reviewed the results of each proposal, and initial assessment reports were produced, containing the ranking of the provisional winning proposals.

The appropriate CSA authorities were consulted, and the necessary recommendations, endorsements and/or approvals were obtained. The final list of proposals recommended for contribution funding were submitted to the IIRB for final approval.

When necessary, risk assessments were completed as stipulated in the Recipient Risk Assessment Framework for CSA Grants and Contributions Program. When required, final assessment reports were submitted to the G&C COE for endorsement and, lastly, to the DG, SST sector, for approval.

Awarding of agreements/contracts

The STDP worked with the G&C COE to establish and manage contribution agreements. The G&C COE ensured compliance with CSA contribution-agreement rules and provided organizational support during the various phases of the contribution process. For contracts, a dedicated PSPC officer is assigned exclusively to the STDP to act as the contracting authority.

Oversight

We found that contribution agreements and contracts receive appropriate oversight. Every contribution agreement and contract is assigned to a project manager in order to ensure monitoring. This person is responsible for day-to-day management until completion.

The claims are verified. A due-diligence process is used, supporting documents are requested, activity reports are sought according to the recipient's risk level and project advancement, and budgets are monitored. For contracts, the participation of technical authorities is also required for monitoring.

No recipient audits have been conducted since the program was introduced. The Treasury Board Policy on Transfer Payments proposes a risk-based approach for recipient audits. At the present time, the audit framework for CSA contribution recipients does not provide for audits.

Spending authorizations are controlled, expenditures are approved and consistent with legislation and regulations, and transactions are recorded appropriately

To assess this criterion, we selected five contribution agreements and five contracts. Our review determined that spending authority is exercised in accordance with delegated authorities (FAA section 32); that expenditures are made in accordance with contractual agreements, legislation, policies and regulations; that payments are based on statements certifying that the goods have been delivered, the services rendered and the work performed (FAA section 34); that the payments are made in accordance with the agreement and approved (FAA section 33); and that expenditures are reported appropriately.

Initial agreements

Each of the five contribution agreements and five contracts examined were certified pursuant to section 32 of the FAA, and approved and signed by authorized incumbents. Needs, including deliverables, were clearly defined in a quote or statement of work. No work was authorized/performed before the agreement was officially approved. A copy of the agreement signed by both parties was on file.

Amendments

One element noted with respect to contracts issued under the program is that many amendments were made during the year. From the time of their original grant date until the date of our review, 16 amendments had been made to the five contribution agreements and five contracts in our initial sample. Four amendments included an increase in the funding value of the contract following the addition of deliverables.

After reviewing the initial sample, we examined all financial amendments for -, and identified 30. Of these, we selected five for more in-depth review.

The following is a summary of the financial amendments for -:

Amendments awarded in - affecting fiscal - < $75K > $75K Total
Number 10 20 30
Increase in value $219,230 $5,572,081 $5,791,311

Almost all of the amendments were funded from the budget increase allocated to STDP during the year.

We found that the amendments are made in accordance with the relevant legislation, policies, regulations and guidelines issued by the CSA and central agencies. We also noted that many controls are used when implementing amendments. The proposals received from contractors are reviewed by the sectors, project authorities and technical authorities. The process for approving selected investments requires the signatures of the Director General, SST, the Chief Financial Officer and the Vice-President. Investments exceeding $75K must be approved by the IIRB. Regardless of size, proposed STDP R&D amendments are reviewed, questioned and approved by the PSPC contracting authority. A process to this effect is well defined. The information requested by PSPC includes a description of the additional work, the reason why this work was not included in the original contract, the reason why an amendment to the existing contract was being requested instead of creating a new posting and a cost estimate.

As soon as requests are received, the PSPC contracting authority analyses them, assesses the relevant risks, compares the advantages and disadvantages of an amendment or new posting and determines the level of approval required on their part. PSPC confirmed that no contract had been increased in order to allow a contractor to complete the work in an existing contract. Amendments always involve new deliverables.

With respect to non-financial contract amendments, the initial sample indicated that only three of the seven contract amendments that did not increase the value of the agreement were supported by an amendment request certified by an authorized incumbent, pursuant to section 32 of the FAA. The other four amendments were supported by correspondence from the project authority or technical authority. The Requisition Guide for the CSA Procurement and Contract Administration (PCA) division stipulates that an amendment request is required when the cost, delivery date, description or any other change must be made to a contract. The Guide also indicates that the amendment should always be made on the same requisition used for the original contract. Although the four contract amendments supported only by correspondence did not include an increase in the value of the agreement, we feel that an amended requisition should be prepared in order to comply with the PCA Requisition Guide and maintain a synchronized number of amendments between the requisitions and contracts prepared by PSPC. If the CSA is of the opinion that an exchange of correspondence between the program and PCA/PSPC is sufficient, and the preparation of an amended requisition is not necessary, we feel that the Requisition Guide should be updated accordingly.

Payments

With respect to payments, we selected the last invoice submitted for each of the five contribution agreements and five contracts in our sample. We found that the amounts invoiced matched the basis of payment in the agreement, that any gaps between the amounts claimed and the terms and conditions of the agreements/contracts were justified, that the goods and services were received within the timeframes specified in the agreements/contracts, and that the necessary supporting documents have been provided. Furthermore, the invoices were certified by an authorized incumbent pursuant to section 34 of the FAA, the claims made against contracts awarded by PSPC were authorized by these people before payment, and the payments were certified by an authorized incumbent pursuant to section 33 of the FAA. Lastly, the expenses matched the amounts entered in the CSA financial systems (SAP) and the expenditures were recorded appropriately (expenditures, assets, general ledger, etc.) in CSA logs.

Recommendations

No recommendation

Identified responsibility
Organization

N/A

Function

N/A

Management response

N/A

Management action plan
Action plan details

N/A

Deadline

N/A

2.3.3 Accountability

Audit objective: The purpose of the audit project was to determine if a management framework is in place to enable the program to achieve its objectives and comply with the relevant legislation, policies, regulations and guidelines issued by the CSA and central agencies.

Findings
Criterion 3

Program activities are subject to accountability.

Condition

Conclusion about the criterion

The audit found that there is accountability for the program's activities. However, we identified one opportunity for improvement with respect to the information available on long-term outcomes for technology funded through contracts.

Program activities are subject to accountability

Accountability at the CSA is achieved through a number of processes. First, there is the SST business plan, which articulates sector objectives for the fiscal year. The program is required to submit quarterly reports on the progress of main tasks that must be completed during the year. This document is used to prepare the overall CSA business plan.

Second, the program reports on its activities while preparing the annual SST performance report. The annual performance report provides the SST directors and director general with data and analyses on the results achieved during the previous fiscal year. This report is also used to complete Electronic Work Plans (eWPs).

Contribution agreements

Short-term contribution results

Contribution agreements provide for the disclosure of results achieved following project implementation. Recipients must demonstrate the R&D work that was accomplished. The CSA requires recipients to produce a report demonstrating how the contribution was used. Results and data on indicators must be shared with the CSA.

The CSA compiles these results at the end of the agreements. We reviewed a performance-data compilation report for the AO 2. The program has used data obtained from recipients to produce this report, which was dated . The data are provided according to the following five indicators set out in the contribution agreement:

  • Technology readiness levels (TRLs) and know-how;
  • Intellectual properties generated;
  • Basic industrial skills and products developed;
  • Nature of the infrastructure developed;
  • Highly qualified personnel involved.

The report provides an overview of the impact of the AO 2 on the Canadian space industry. Some of the data collected for this AO were not sufficiently well defined to draw accurate conclusions on some issues, and enhanced directives were expected to be issued for subsequent AOs.

Long-term outcomes and feedback on technological developments for contributions

For the first time, in , STDP contribution recipients were asked to complete a survey on long-term outcomes for the AO 2. The agreements had ended a few years before. The key elements assessed during this survey focussed on the development of industrial capacity and the commercial benefits generated with the help of the developed technology. Since this exercise had yet to be completed at the time of this report, the results were not reviewed by the audit team.

This monitoring of outcomes over the long-term is a best practice, as it makes it possible to determine the program's effectiveness with respect to industrial capacity building. The program plans to continue this activity.

Contracts

Short-term outcomes

With respect to contracts, the deliverables are set out in the contracts. There are specific performance requirements. The contractors must therefore achieve the expected outcomes. If these outcomes are not met, the contractor's remuneration is withheld until the CSA is fully satisfied. The last phase of the projects includes a final review by the CSA with the supplier. The sectors affected by the work are invited to take part in order to express their level of satisfaction.

Long-term outcomes and feedback on funded technology

Program management has started conducting internal surveys of CSA sectors to obtain feedback on technological developments funded through contracts (RFP) in order to identify the impact and benefits observed in the longer term.

We reviewed some of the completed surveys. They indicate, for instance, that some contracts had helped to eliminate many technological uncertainties, finalize estimates for mission aspects (e.g., cost, level of effort, schedule) and reduce major technological risks for a potential mission. They also indicate that some technological developments sought for specific missions have been approved while others were pending. Some technological developments should help to inform decision making regarding the approval of Canada's participation. It is expected that new internal surveys will be carried out shortly, that the results will be analysed and that a report will be submitted to management. For example, it would be relevant to know how many technological developments funded have been used in a mission when assessing the program's effectiveness.

We feel that more detailed accountability and statistics could support planning and decision making for future investments.

External accountability

External accountability requirements are met. The program's main external accountability requirements come from the following directives: proactive disclosure of contracts; disclosure of grant and contribution awards; disclosure of results; and disclosure of travel, hospitality, conference and event expenditures.

All contracts awarded under the STDP are issued by PSPC. PSPC is responsible for disclosing them in accordance with applicable regulations. Moreover, STDP agreements are posted on the CSA website in the weeks prior to the contracts being awarded. The same is true for contribution agreements awarded by the CSA.

The disclosure of travel, hospitality, conference and event expenditures is done by the CSA Finance Directorate.

Recommendation
  1. Continue to put in place elements that will improve the accountability of Space Technology Development Program (STDP) R&D activities in order to support planning and decision making for future investments.
Identified responsibility
Organization

Space Science and Technology Directorate

Function

Director General of the Space Science and Technology Directorate

Management response

Obtaining more detailed results, data and information on the impact of technology development achieved would support planning and decision-making for future investments.

Management action plan
Action plan details

The Space Science and Technology branch will continue developing the mechanism established in under the STDP's contribution element. We will therefore survey, on a annual basis, organizations that have conducted research and development projects to determine the impact of the research results on those organizations.

A similar mechanism will be developed and implemented under the STDP's procurement element. The internal sectors that initiated technology development requests will be surveyed at the end of each contract to determine the extent to which the research results are relevant and useful. Companies that have developed technology will also be surveyed to determine the benefits of their contracts.

  • The Space Science and Technology branch will be responsible for periodically following up on and disseminating the results as an internal report to support planning and decision-making for future investments; and
  • The internal sectors in question will be responsible for providing the information requested through a survey.
Deadline

Appendix A — Terms of reference

Audit objective: The purpose of the audit project was to determine if a management framework is in place to enable the program to achieve its objectives and comply with the relevant legislation, policies, regulations and guidelines issued by the CSA and central agencies.

Audit criteria Audit sub-criteria
  • Sub-criterion met 
  • Sub-criterion partially met 
  • Sub-criterion not met 
Criterion 1
Program activities are planned appropriately.
Sub-criterion 1.1: Investment priorities are determined based on a complete opportunity analysis. Sub-criterion partially met
Sub-criterion 1.2: Financial resource planning is based on planned activities. Sub-criterion met
Criterion 2
Program activities are monitored appropriately.
Sub-criterion 2.1: Roles and responsibilities are clearly defined. Sub-criterion met
Sub-criterion 2.2: Day-to-day activities are monitored, measured, compared with forecasts, reported and approved in accordance with internal CSA policies and directives. Sub-criterion met
Sub-criterion 2.3: The preparation and publication of AOs and RFPs, and the analysis, assessment and selection of proposals received are in accordance with established processes and procedures and comply with program terms and conditions, as well as the applicable legislation, policies, regulations and guidelines. Sub-criterion met
Sub-criterion 2.4: Spending authorizations are controlled, expenditures are approved and consistent with legislation and regulations, and transactions are recorded appropriately. Sub-criterion met
Criterion 3
Program activities are subject to accountability.
Sub-criterion 3.1: Program activities are subject to accountability. Sub-criterion partially met

Appendix B — Decision-making process – Awarding contributions and contracts

Text version

Contributions:
Canadian industry (Business and University):

  • STDP:
    • Consultations and Needs Identification
    • Posting
    • Intake, Screen and Assess Proprosals
    • List Proposals
  • Space Science and technology Steering Committee:
    • Review and Endorse the List
  • STDP:
    • Allocate Funding
  • IIRB:
    • Approval
  • STDP:
    • Actions to Award Agreements

Contracts:
Canadian industry (Business and University) / Other Government Department (OGD)/International Partners:

  • Sectors:
    • Consultations and Needs Identification
  • IIRB:
    • Approve Decision point 1
  • Sectors:
    • Submit Technological Development Request to the TPWG
  • TPWG:
    • Validate Process Used by the Sectors
    • Group Applications According to Technology Readiness Level (TRL)
    • Consolidate Applications into a Single List of Priorities
  • Space Science and technology Steering Committee:
    • Endorse the List of Priorities
    • Send the List to STDP
  • IIRB:
    • List Submitted by STDP
    • Approval
  • STDP:
    • Actions to Award Contrats
  • IIRB:
    • Approve Contracts ›75K$

Legend:

  • STDP: Space Technology Development Program
  • SST-SC: Space Science and technology Steering Committee
  • TPWG: Technology Planning Working Group
  • IIRB: Integrated Investment Review Board
  • Sectors: CSA Scientifics Sectors:
    • Space Exploration
    • Space Utilization
    • Space Science and Technology
Date modified: