Management Action Plans Follow-up - Evaluation - Annual Report as of March 31st, 2016

Audit and Evaluation Directorate

June 2016

Table of contents

Implementation summary

This follow-up report on the implementation of management action plans outlines the measures taken by the various entities concerned in response to findings and recommendations. As part of the follow-up process, management action plans are reviewed annually until they are fully implemented, and the extent of implementation is assessed and reported to the CSA's Departmental Evaluation Committee (DEC).

This annual report contains the follow-up findings, as of March 31st, 2016, of eight evaluation projects for which reports and management action plans have been submitted to and approved by the DEC, or by the Executive Committee, which, until 2015–2016, was acting as the DEC. The following charts give an overview of the implementation status of the management action plan elements.

To do ≤ 10% In progress 11% - 50% In progress > 50% Completed 100%
Evaluation of the ISS Assembly and Maint. Oper. Prog. (Feb. 2016) 1 1    
Evaluation of Evaluation of the IMAP (Canada-ESA) (July 2015)   3 1 1
Evaluation of the Qualifying and Testing Services Prog. (Dec. 2014)   1 3 2
Evaluation of the NEOSSAT Project (Feb. 2014)   1   2
Evaluation of the CASSIOPE Contribution Program (Feb. 2014)   3    
Evaluation of the AETDP (Feb. 2014)     1 4
Evaluation of the EODIUP (Jan. 2012)     4 4
Evaluation of the Canada/ESA Cooperation Agreement (Sept. 2010)     1 4
To do ≤ 10% (1 management action plan element) In progress 11% - 50% (9 management action plan elements) In progress > 50% (10 management action plan elements) Completed 100% (17 management action plan elements)
Management action plan elements 1 9 10 17

The following pages set out in detail the progress of the action plans for each of the evaluation projects.

Evaluation project: Goss Gilroy

Summative evaluation of the 2000-2009 Canada/ESA cooperation agreement

Evaluation objectives

This summative evaluation was conducted by Goss Gilroy Inc (GGI) Management Consultants in 2008 and was designed to assess relevance, success and cost-effectiveness of the Canada/European Space Agency (ESA) cooperation agreement. The evaluation report was presented to the Executive Committee in June 2010 and subsequently approved in September 2010.

Project support and communication Policy monitoring Program renewal Priority alignment
40% 20% 20% 20%

Nature of recommendations

The evaluation report provided recommendations relating to program renewal, strategic priority alignment, program support and communication to Canadian industry and, finally, policy monitoring. Specifically, the five recommendations for program improvement consisted of renewing the agreement, clarifying ESA's role within the CSA's strategic priorities, developing policies and procedures to support and formally communicate to industry ESA industrial opportunities, and monitoring impacts of space trends in the European Union (EU) on Canada.

Implementation status

As of March 31st, 2015, two of the five recommendations from the evaluation were ongoing.

This year, one recommendation was completed: drafting an action plan to communicate to Canadian businesses industrial opportunities identified by the ESA. The program team worked on updating the portion of the CSA's website that pertains to the ESA in the aim of adding information and Web links that could be useful to possible bidders or new Canadian participants. The part of the CSA's website that pertains to the ESA is now operational, which completes the implementation of this recommendation.

The recommendation to support possible Canadian bidders included drafting a guide for Canadian delegates at the ESA. A guide was drafted so that Canadian delegates could easily obtain information on the ESA's procurement process and understand what their role is within the ESA's program boards. This guide should be approved during the first quarter of the 2016–2017 fiscal year.

To do ≤ 10% 0
In progress 11% - 50% 0
In progress > 50% 1
Completed 100% 4

Evaluation project: # 10/11 02 – 01

Evaluation of the Earth Observation Data and Imagery Utilization Program (EODIUP) for the period from April 1st, 2005 to March 31st, 2010

Evaluation objectives

The evaluation of the Earth Observation Data and Imagery Utilization Program (EODIUP) was completed internally by the CSA's Evaluation function. The report was approved by the Executive Committee in January 2012. As per the Treasury Board Secretariat's Directive on the Evaluation Function (2009), the evaluation responded to the five core issues related to program relevance and performance.

Project support and communication Partner contributions and objectives Program performance and monitoring Program objectives
12% 13% 13% 62%

Nature of recommendations

The evaluation report provided recommendations relating to program objectives, project support and communication to stakeholders, coordination of partner contributions, and monitoring of program performance. Specifically, the eight recommendations for program improvement were to:

  1. Increase industry's role in the development of applications for federal departments' needs;
  2. Support academia's participation in the development of applications;
  3. Determine proportion of resources that should be allocated to applications developed from foreign data;
  4. Ensure data access and sharing is facilitated by Radarsat Constellation Mission's (RCM) Data Policy;
  5. Allocate a greater share of program resources to awareness and knowledge transfer activities;
  6. Increase funding for the development of products and services that will improve market competiveness;
  7. Introduce mechanisms to identify development clusters and coordinate partners' contributions; and
  8. Identify program objectives and monitor the performance with the development of a PM strategy.

Implementation status

As of March 31st, 2015, four of the eight recommendations from the evaluation report (1, 2, 5 and 6) had been entirely implemented. This year, the program team continued its work to implement the other recommendations (3, 4, 7 and 8). Each one is over 50% complete.

The RADARSAT Constellation Mission's Data Policy is in development. Work was done this year, in particular in collaboration with the departments involved, in the aim of ensuring that this policy meets RADARSAT Constellation Mission users' expectations with regard to access to and the sharing of data.

The eighth recommendation included two components: the development of a performance measurement strategy and its implementation. The performance measurement strategy was finalized during 2014–2015. A report on the program's performance including recommendations on the methodology used is expected soon, which will end the implementation of this recommendation.

Recommendations 3 and 7 also included two components. The first component, the implementation of governance mechanisms, had been completed during 2014–2015. A strategic plan concerning Earth observation data application and use programs was developed during the year in response to the second component of recommendations 3 and 7, and is currently awaiting a decision.

To do ≤ 10% 0
In progress 11% - 50% 0
In progress > 50% 4
Completed 100% 4

Evaluation project # 12/13 02-01

Evaluation of the Advanced Exploration Technology Development Program (AETDP) for the period from April 1st, 2008 to March 31st, 2013

Evaluation objectives

The evaluation of the Advanced Exploration Technology Development Program (AETDP) was completed internally by the CSA's Evaluation function. The report was approved by the Executive Committee in February 2014. As per the Treasury Board Secretariat's Directive on the Evaluation Function (2009), the evaluation responded to the five core issues related to program relevance and performance.

Project Management Project support and communication Program Performance and Monitoring Priority Alignment
20% 20% 20% 40%

Nature of recommendations

The evaluation report brought to light that, as a whole, the AETDP demonstrates continued relevance and its performance has been generally successful, effective and economical. However, the report's findings point to opportunities for program improvement. Specifically, the recommendations for program improvement were as follows:

  1. Conduct an analysis of the optimal level of resources that should be dedicated to scientific development by the program;
  2. Either create a clearer distinction between the Enabling Technology Development Program (ETDP) and the AETDP or merge the two programs;
  3. Clarify contracting processes regarding AETDP R&D activities;
  4. Implement a systematic method of communicating plans and priorities to all AETDP staff; and
  5. Include baseline data and targets in the AETD's program performance measurement and make all performance data available in an accessible format.

Implementation status

As of March 31st, 2015, two of the five recommendations of the AETDP evaluation had been entirely implemented (1 and 4). This year, the program team completed the implementation of two other recommendations (2 and 3). Work was carried out in order to review the program design and the contracting process concerning program R & D was clarified.

The last recommendation is over 50% complete. It involves including data and benchmark objectives from the AETDP performance measurement strategy and making all the performance data available in an accessible format. The program performance measurement strategy was approved in 2014, and efforts have been made throughout the year to develop a data collection tool.

To do ≤ 10% 0
In progress 11% - 50% 0
In progress > 50% 1
Completed 100% 4

Evaluation project # 13/14 02-01

Evaluation of the CASSIOPE Contribution Program for the period from December 2003 to December 2013

Evaluation objectives

A consortium headed up by Kelly Sears Consulting Group conducted the evaluation of the CASSIOPE Contribution Program. The purpose of this evaluation was to establish the relevance and performance of the CSA CASSIOPE Contribution Program. The evaluation report was approved by the CSA Executive Committee in February 2014.

Project support and communication Resources Management Program performance and monitoring
33% 33% 34%

Nature of recommendations

The evaluation report indicated that missions such as CASSIOPE serve important commercial, scientific and societal needs of Canadians. However, the report contained recommendations pertaining to program performance monitoring, communication of strategic directions and management of funding mechanisms.

More specifically, the three recommendations aimed at improving the program advised that the CSA:

  1. Make efforts to track the impacts of its projects on industrial capacity;
  2. Explore ways to better ensure the Canadian space industry has the technological capacity to undertake space projects sponsored by the agency; and
  3. Ensure that it has the internal capacity to both assess the merits of the business case presented by the recipient and to manage the repayment process, if the CSA intends to use repayable contributions in future space projects.

Implementation status

During 2015-2016, the CSA continued its discussions in the aim of establishing a strategic direction for the Canadian space sector.

Discussions also took place within the CSA intended to refine programs' performance measurement and mechanisms for socio-economic data collection on the Canadian space sector. All of these measures should eventually allow for better documentation of the repercussions of the projects.

The revision of the mandate, roles and responsibilities of the CSA's Grants and Contributions Centre of Expertise was initiated during the year. A comparative study on the roles and responsibilities of the expertise centres of other federal departments was conducted.

To do ≤ 10% 0
In progress 11% - 50% 3
In progress > 50% 0
Completed 100% 0

Evaluation project # 13/14 02-02

Evaluation of the NEOSSAT project for the period from February 2005 to December 2013

Evaluation objectives

A consortium headed up by Kelly Sears Consulting Group conducted this evaluation in 2014. The purpose of the evaluation was to assess the relevance and performance of the CSA NEOSSAT project. The evaluation report was approved by the CSA Executive Committee in February 2014.

Resources management Project support and communication Project management
33% 33% 34%

Nature of recommendations

The evaluation showed that a sound rationale for NEOSSAT-type satellite missions exists and that the project was implemented in accordance with activities in the initial plan. However, the evaluation did point to lessons learned from the experience acquired throughout NEOSSAT.

More specifically, the three recommendations aimed at improving the project advised that the CSA:

  1. Establish a process to address risk factors in cases where deficiencies are observed regarding the financial, technical and/or project management of contractor performance during contract execution;
  2. Continue to maintain technical capacity, either internally or available on an as needed basis, in order to mitigate potential risks related to contractor capacity; and
  3. Define, in collaboration with other federal partners and industry, a strategic direction to establish "niche" capacities and capabilities within the Canadian space industry.

Implementation status

As of March 31st, 2015, a new investment monitoring and governance framework as well as the related follow-up processes and tools had been approved by the CSA Executive Committee, enabling the implementation of the first recommendation from the evaluation report.

This year, the program team completed the implementation of the second recommendation. An analysis of key technical expertise and the extension of standing offers for engineering services were completed and guidelines for project monitoring are in place in the aim of mitigating risks.

Finally, the CSA continued its discussions intended to establish a strategic direction for the Canadian space sector.

To do ≤ 10% 0
In progress 11% - 50% 1
In progress > 50% 0
Completed 100% 2

Evaluation project # 14/15 02 – 03

Evaluation of the Qualifying and Testing Services Program (David Florida Laboratory) for the period from April 2009 to September 2014

Evaluation objectives

A consortium led by Kelly Sears Consulting Group conducted the evaluation of the CSA's Qualifying and Testing Services Program (DFL) in 2014. The evaluation report was approved by the Executive Committee in December 2014. As per the Treasury Board Secretariat's Directive on the Evaluation Function (2009), the evaluation responded to the five core issues related to program relevance and performance.

Program performance and monitoring Resources management Program objectives Project support and communication
33% 33% 17% 17%

Nature of recommendations

Overall, the evaluation report highlighted the relevance of the program and the achievement of expected results. It was nonetheless recommended that the CSA:

  1. Set up a process for consulting with stakeholders to provide guidance on current and future testing needs;
  2. Revise the existing performance measurement strategy;
  3. Explore a program of sharing staff, particularly engineers, between CSA HQ and DFL;
  4. Develop and put in place a consistent and robust approach to measuring utilization at the DFL;
  5. Undertake a study to identify realistic potential markets for the DFL beyond the space sector; and
  6. Review and revise the costing model for the DFL.

Implementation status

Two of the six recommendations of the DFL evaluation report were completed this year (2 and 3). The performance measurement strategy was revised and approved, as was the junior engineer development program that now covers a wider array of disciplines and could possibly allow the DFL to host an intern. Opportunity posting mechanisms were used as needed to staff positions at the DFL.

Recommendations 1, 4 and 6 are more than 50% complete. The program team began discussions with various stakeholders in the space sector in order to learn more about current and future needs in terms of testing capacities. The implementation of a thorough and consistent method for measuring utilization at the DFL should be finalized soon. The Finance Directorate, with the support of the DFL, began the review and revision of the costing model.

Reflections on the new management models at the DFL continued this year in response to the fifth recommendation, which should eventually allow for optimal utilization of the laboratory.

To do ≤ 10% 0
In progress 11% - 50% 1
In progress > 50% 3
Completed 100% 2

Evaluation project # 14/15 02-02

Evaluation of the International Market Access Program of the Canadian Space Agency (IMAP) for the period from April 2009 to December 2014

Evaluation objectives

The evaluation from the International Market Access Program was carried out by a consortium led by Kelly Sears Consulting Group. The evaluation report was approved by the Executive Committee in July 2015. The evaluation discussed the relevance, performance, design and delivery of the program.

Priority alignment Program objectives Project support and communication
20% 40% 40%

Nature of recommendations

In general, the evaluation report highlighted the relevance of the program and the achievement of the expected results. The recommendations had to do with communication with the Canadian space industry, priority alignment, and program objectives. In particular, the five recommendations emphasized the need to:

  1. communicate the implications of the Canada-EU Free Trade Agreement to the Canadian space industry;
  2. establish a strategic direction for the Canadian space sector;
  3. examine its funding allocation to ESA programs;
  4. undertake broader, formalized consultations with Canadian stakeholders; and
  5. assess the positioning of the Canada/ESA Cooperation Agreement within the CSA's overall suite of programs.

Implementation status

In response to the second recommendation, the CSA committed to continuing to participate in discussions on the strategic direction of the Canadian space sector during meetings with the Deputy Ministers' Governance Committee on Space and with national and international stakeholders. The CSA continued its discussions intended to establish a strategic direction for the Canadian space sector, which will end the implementation of this recommendation.

Implementation of the first recommendation is more than 50% complete. The CSA maintained its interactions with Global Affairs Canada and the Canadian space industry on the topic of the Comprehensive Economic Trade Agreement (CETA) between Canada and the EU.

The implementation of recommendations 3, 4 and 5 began during the year. A consultation was held with representatives of the Canadian space industry on opportunities to be taken advantage of as part of the ARTES program. The program team will continue over the coming months its reflections on the allocation of funds to the ESA's various programs with an eye to the next planning meetings with the member states.

To do ≤ 10% 0
In progress 11% - 50% 3
In progress > 50% 1
Completed 100% 1

Evaluation project # 14/15- 02-01

Evaluation of the CSA's ISS Assembly and Maintenance Operations Program for the period from March 2008 to September 2015.

Evaluation objectives

This evaluation was carried out by a consortium led by Kelly Sears Consulting Group. Its purpose was to determine the relevance and performance of CSA's ISS Assembly and Maintenance Operations Program. The evaluation report was approved by the DEC in February 2016.

Program objectives Priority alignment
50% 50%

Nature of recommendations

In general, the evaluation report highlighted the relevance of the program and the achievement of the expected results. It recommended that the CSA:

  1. assess the importance of maintaining Canada's leadership role in space robotics and the measures to be taken if Canada is to maintain a leadership position in space robotics;
  2. assess options for increasing the commercialization and transfer of technologies from the ISS Program to other areas.

Implementation status

Following approval of the evaluation report, the Space Exploration Branch implemented a working group tasked with producing a report on the status of space robotics in Canada, including an analysis of strengths, weaknesses, opportunities and threats. Mission roadmaps and technology will be updated following this exercise and the Branch will identify and prioritize opportunities for technological development and for missions using space robotics and related technology.

The Branch also committed to suggesting precise measures to increase economic impacts, by-products and daily advantages from space robotics. Reflections were begun within the CSA with the intention of producing a report on the commercialization and transfer of space robotics technology.

These two reports, on the status of space robotics and on the commercialization of technology, should be finalized during the next year.

To do ≤ 10% 1
In progress 11% - 50% 1
In progress > 50% 0
Completed 100% 0