Management Action Plans Follow-up - Evaluation - Annual Report as of March 31st, 2015

Download the PDF version (819 KB)

Audit and Evaluation Directorate

June 2015

Table of Contents

Implementation summary

This follow-up report on the implementation of management action plans outlines the measures taken by the various entities concerned in response to findings and recommendations. As part of the follow-up process, management action plans are reviewed annually until they are fully implemented, and the extent of implementation is assessed and reported to the Executive Committee, which serves as the Canadian Space Agency's (CSA) Departmental Evaluation Committee.

This annual report contains the follow-up findings, as of March 31st, 2015, of seven evaluation projects for which reports and management action plans have been submitted to and approved by the Executive Committee. The following charts give an overview of the implementation status of the management action plan elements.

To do ≤ 10% (5 management action plan elements) In progress 11% - 50% (1 management action plan element) In progress > 50% (14 management action plan elements) Completed 100% (16 management action plan elements)
Management action plan elements 5 1 14 16

The following pages set out in detail the progress of the action plans for each of the evaluation projects.

Evaluation project: Goss Gilroy

Summative evaluation of the 2000-2009 Canada/ESA cooperation agreement

Evaluation objective

This summative evaluation was conducted by Goss Gilroy Inc (GGI) Management Consultants in 2008 and was designed to assess relevance, success and cost-effectiveness of the Canada/ESA cooperation agreement. The evaluation report was presented to the Executive Committee in June 2010 and subsequently approved in September 2010.

Project support and communication Policy monitoring Program renewal Priority alignment
40% 20% 20% 20%

Nature of recommendations

The evaluation report provided recommendations relating to program renewal, strategic priority alignment, program support and communication to Canadian industry and, finally, policy monitoring. Specifically, the five recommendations for program improvement consisted of renewing the agreement, clarifying ESA's role within the CSA's strategic priorities, developing policies and procedures to support and formally communicate to industry ESA industrial opportunities, and monitoring impacts of space trends in the European Union (EU) on Canada.

Implementation status

By 2013-2014, the program team had implemented all but two of the recommendations arising from the evaluation. The team has continued its efforts this year, and the two outstanding recommendations are almost complete.

The recommendation to support potential Canadian bidders included an action item that pertains to the development of a Guide for the Canadian ESA delegates. A guide was developed to allow all Canadian delegates ready access to information on the ESA procurement process and to help them understand their role on ESA program councils. This guide was submitted to a consultation process and distributed to delegates for review. Changes have been made and the guide should be approved and circulated to delegates in the near future.

The other remaining recommendation calls for the development of a coherent plan of action for communicating targeted ESA industrial opportunities to Canadian companies. The ESA management team has been working on the update of the ESA section of the CSA website to include relevant information and web links that could be useful to potential Canadian bidders and new entrants. Although there has been considerable progress in this regard, this recommendation had not yet been fully implemented as of March 31, 2015, but it is expected to be in the very near future.

To do ≤ 10% 0
In progress 11% - 50% 0
In progress > 50% 2
Completed 100% 3

Evaluation project: 570-2800

Evaluation of the Space Technology Development Program (STDP) for the period from 2002/03 to 2007/08

Evaluation objective

Government Consulting Services (GCS) conducted the evaluation of the Space Technology Development Program (STDP) (now referred to as Enabling Technology Development) in 2009-10 and the report was subsequently approved by the Executive Committee in June 2011. The evaluation focused on four main areas: program relevance, program design and delivery, program success, and cost-effectiveness.

Project support and communication Program objectives Project management
33% 34% 33%

Nature of recommendations

The evaluation report shed light on the need to employ the appropriate funding mechanism and identify R&D technology requirements in order to attain the program's objectives, to improve project management through more efficient procurement processes and service standards, and to provide project support and establish communication strategies with companies and CSA clients. The evaluation report contained six recommendations for program improvement.

Implementation status

The follow-up report on the implementation of management action plans published last year described the implementation of five of the six recommendations stemming from the STDP evaluation report.

This year, the team responsible for the program focused its efforts on the only remaining recommendation, namely, to implement an official communication strategy to ensure that project results are shared with CSA clients.

In 2013-2014, the program had formalized the way in which project results are communicated to CSA clients by consulting them and inviting them to launch meetings, and by sending them copies of final review packages. Clients were also surveyed and survey findings were compiled into client satisfaction reports by the 2015 program team. All the recommendations from the STDP evaluation report have been fully implemented.

To do ≤ 10% 0
In progress 11% - 50% 0
In progress > 50% 0
Completed 100% 6

Evaluation project: # 10/11 02 – 01

Evaluation of the Earth Observation Data and Imagery Utilization Program (EODIUP) for the period from April 1st, 2005 to March 31st, 2010

Evaluation objective

The evaluation of the Earth Observation Data and Imagery Utilization Program (EODIUP) was completed internally by the CSA's Evaluation function. The report was approved by the Executive Committee in January 2012. As per the Treasury Board Secretariat's Directive on the Evaluation Function (2009), the evaluation responded to the five core issues related to program relevance and performance.

Program objectives Project support and communication Partner contributions and objectives Program performance and monitoring
62% 12% 13% 13%

Nature of recommendations

The evaluation report provided recommendations relating to program objectives, project support and communication to stakeholders, coordination of partner contributions, and monitoring of program performance. Specifically, the eight recommendations for program improvement were to:

  1. Increase industry's role in the development of applications for federal departments' needs;
  2. Support academia's participation in the development of applications;
  3. Determine proportion of resources that should be allocated to applications developed from foreign data;
  4. Ensure data access and sharing is facilitated by Radarsat Constellation Mission's (RCM) Data Policy;
  5. Allocate a greater share of program resources to awareness and knowledge transfer activities;
  6. Increase funding for the development of products and services that will improve market competiveness;
  7. Introduce mechanisms to identify development clusters and coordinate partners' contributions; and
  8. Identify program objectives and monitor the performance with the development of a PM strategy.

Implementation status

Two of the recommendations (2 and 6) were fully implemented in 2013-2014, in accordance with their target dates. As a result, the mechanisms to increase the role of both industry and academia in the development of applications were implemented through the CSA Class Grants and Contributions Program.

This year, the program team completed the implementation of two other recommendations (1 and 5). The first recommendation supported increasing industry's role in the development of applications in response to the needs of federal departments and agencies. Several actions were identified in response to this recommendation. In 2013-2014, the program team entered into negotiations to create a new partnership and held an information and consultation session with departments and industries. This year, the program team considered developing an umbrella agreement between the CSA and its federal partners. However, following discussions, the program management team agreed to keep the standard agreement template and to amend the appendices according to the project being funded. In response to the fifth recommendation, a communication strategy was developed and discussed with the Communications Branch and it was agreed that it would be implemented as resources permitted.

The remaining recommendations (3, 4, 7 and 8) are in progress and are each more than 50% complete. With respect to Radarsat Constellation Mission (RCM) data access and sharing, consultations with other government departments are under way to develop the RCM data and use policy. This action item is set to be completed by the time the first RCM satellite is launched, which is slated for 2018.

The eighth recommendation is twofold, aimed at both the development and the application of a performance measurement strategy. The performance measurement strategy was finalized in 2014-2015. The recommendation will be deemed to be complete once all the indicators have been measured at least once, which should be the case before the end of this year.

Recommendations 3 and 7 were also twofold, proposing the development of governance mechanisms and a strategic plan in response to needs expressed. Over the past year, new governance mechanisms were created with the intention of setting program priorities, documenting user needs, and developing commitments. A draft strategic plan for the Space Utilization Branch was also drawn up and a decision in this regard is currently pending.

To do ≤ 10% 0
In progress 11% - 50% 0
In progress > 50% 4
Completed 100% 4

Evaluation project: # 12/13 02 – 01

Evaluation of the Advanced Exploration Technology Development Program (AETDP) for the period from April 1st, 2008 to March 31st, 2013

Evaluation objective

The evaluation of the Advanced Exploration Technology Development Program (AETDP) was completed internally by the CSA's Evaluation function. The report was approved by the Executive Committee in February 2014. As per the Treasury Board Secretariat's Directive on the Evaluation Function (2009), the evaluation responded to the five core issues related to program relevance and performance.

Project Management Project support and communication Program Performance and Monitoring Priority Alignment
20% 20% 20% 40%

Nature of recommendations

The evaluation report brought to light that, as a whole, the AETDP demonstrates continued relevance and its performance has been generally successful, effective and economical. However, the report's findings point to opportunities for program improvement. Specifically, the recommendations for program improvement were as follows:

  1. Conduct an analysis of the optimal level of resources that should be dedicated to scientific development by the program;
  2. Either create a clearer distinction between the Enabling Technology Development Program (ETDP) and the AETDP or merge the two programs;
  3. Clarify contracting processes regarding AETDP R&D activities;
  4. Implement a systematic method of communicating plans and priorities to all AETDP staff; and
  5. Include baseline data and targets in the AETD program's performance measurement strategy and make all performance data available in an accessible format.

Implementation status

Two of the five recommendations in the AETDP evaluation report (1 and 4) were fully implemented over the past year. The program team finished drawing up the new Exploration Core Program (2014-2017). The plan includes technological, scientific and operational activities and was circulated to all stakeholders in the Exploration Core Program. The program team also conducted a quarterly review of AETDP activities, enabling communication of plans and priorities to all staff.

The other three recommendations are currently in progress, with two of them over 50% complete (2 and 5). The CSA Executive Committee approved a new framework for governance and investment oversight, and clarifications on the contracting processes regarding R&D activities are expected in the near future. The program performance measurement strategy was approved during the past year and a data collection tool is under development. Finally, a proposal for consolidation of technological development activities was developed and is awaiting a decision.

To do ≤ 10% 0
In progress 11% - 50% 1
In progress > 50% 2
Completed 100% 2

Evaluation project: # 13/14 02 – 01

Evaluation of the CASSIOPE Contribution Program for the period from December 2003 to December 2013

Evaluation objective

A consortium headed up by Kelly Sears Consulting Group conducted the evaluation of the CASSIOPE Contribution Program. The purpose of this evaluation was to establish the relevance and performance of the CSA CASSIOPE Contribution Program. The evaluation report was approved by the CSA Executive Committee in February 2014.

Project support and communication Resources Management Program performance and monitoring
33% 33% 34%

Nature of recommendations

The evaluation report indicated that missions such as CASSIOPE serve important commercial, scientific and societal needs of Canadians. However, the report contained recommendations pertaining to program performance monitoring, communication of strategic directions and management of funding mechanisms.

More specifically, the three recommendations aimed at improving the program advised that the CSA

  1. Make efforts to track the impacts of its projects on industrial capacity;
  2. Explore ways to better ensure the Canadian space industry has the technological capacity to undertake space projects sponsored by the agency; and
  3. Ensure that it has the internal capacity to both assess the merits of the business case presented by the recipient and to manage the repayment process, if the CSA intends to use repayable contributions in future space projects.

Implementation status

Over the past year, efforts have been made to develop a space strategy and implement communication activities with key stakeholders in the space sector. These efforts will continue throughout the coming year in response to the second recommendation in the CASSIOPE Contribution Program evaluation report.

Preliminary discussions were also undertaken in fiscal year 2014-2015 in anticipation of the review of the CSA Program Alignment Architecture and of its associated performance indicators, and the refinement of data collection mechanisms. All these measures should eventually allow for better documentation of project impacts. In response to the third recommendation, a review of the CSA's Grants and Contributions Centre of Expertise mandate, role and responsibilities and of the repayment process is scheduled to get under way in the next year.

To do ≤ 10% 2
In progress 11% - 50% 0
In progress > 50% 1
Completed 100% 0

Evaluation project: # 13/14 02 – 02

Evaluation of the NEOSSAT project for the period from February 2005 to December 2013

Evaluation objective

A consortium headed up by Kelly Sears Consulting Group conducted this evaluation in 2014. The purpose of the evaluation was to assess the relevance and performance of the CSA NEOSSAT project. The evaluation report was approved by the CSA Executive Committee in February 2014.

Resources Management Project support and communication Project Management
33% 33% 34%

Nature of recommendations

The evaluation showed that a sound rationale for NEOSSAT-type satellite missions exists and that the project was implemented in accordance with activities in the initial plan. However, the evaluation did point to lessons learned from the experience acquired throughout NEOSSAT.

More specifically, the three recommendations aimed at improving the project advised that the CSA

  1. Establish a process to address risk factors in cases where deficiencies are observed regarding the financial, technical and/or project management of contractor performance during contract execution;
  2. Continue to maintain technical capacity, either internally or available on an as needed basis, in order to mitigate potential risks related to contractor capacity; and
  3. Define, in collaboration with other federal partners and industry, a strategic direction to establish "niche" capacities and capabilities within the Canadian space industry.

Implementation status

Various activities have been undertaken over the past year to implement the three recommendations arising from the NEOSSAT project evaluation report. As of March 31, 2015, all action items pertaining to the first recommendation were complete, while the other two recommendations were more than 50% complete.

The CSA Executive Committee approved a new framework for governance and investment oversight along with the monitoring procedures and tools associated with this framework, thereby responding to the first recommendation in the evaluation report.

Analysis of key technical expertise, as well as the extension of standing offers for engineering services, was completed recently. Project control guidelines are to be developed by the end of 2015.

Finally, over the past year, efforts have been made to develop a space strategy and to implement communication activities with key stakeholders in the space sector. These efforts will continue throughout the coming year in response to the third recommendation of the NEOSSAT evaluation report.

To do ≤ 10% 0
In progress 11% - 50% 0
In progress > 50% 2
Completed 100% 1

Evaluation project: # 14/15 02 – 03

Evaluation of the Qualifying and Testing Services Program (David Florida Laboratory) for the period from April 2009 to September 2014

Evaluation objective

A consortium led by Kelly Sears Consulting Group conducted the evaluation of the CSA's Qualifying and Testing Services Program (DFL) in 2014. The evaluation report was approved by the Executive Committee in December 2014. As per the Treasury Board Secretariat's Directive on the Evaluation Function (2009), the evaluation responded to the five core issues related to program relevance and performance.

Program performance and monitoring Resources Management Program objectives Project support and communication
33% 33% 17% 17%

Nature of the recommendations

Overall, the evaluation report highlighted the relevance of the program and the achievement of expected results. However, the evaluation identified areas for improvement in terms of program performance monitoring, communication with space sector stakeholders and resource optimization. Specifically, it was recommended that the CSA:

  1. Set up a process for consulting with stakeholders to provide guidance on current and future testing needs;
  2. Revise the existing performance measurement strategy;
  3. Explore a program of sharing staff, particularly engineers, between CSA HQ and DFL;
  4. Develop and put in place a consistent and robust approach to measuring utilization at the DFL;
  5. Undertake a study to identify realistic potential markets for the DFL beyond the space sector; and
  6. Review and revise the costing model for the DFL.

Implementation status

Three of the six recommendations arising from the DFL evaluation report are already more than 50% complete (recommendations 2, 4, 5). In this regard, the performance measurement strategy, and the approach to measuring utilization at the DFL are currently under review and approval is expected by year-end. The program team began reflecting on new management models for the DFL and this process will continue over the coming year with a view to optimizing laboratory use.

The program team's ongoing efforts in this regard over the coming year will include consultations with various stakeholders in the space sector and updating the DFL costing model.

Finally, no actions were taken this year to implement a labour-sharing program. The program team acknowledges that this would be an appropriate initiative and will continue working to this end over the coming year.

To do ≤ 10% 3
In progress 11% - 50% 0
In progress > 50% 3
Completed 100% 0