Management Action Plans Follow-Up - Evaluation - Annual Report as of March 31, 2014

Download the PDF version (465 KB)

Audit and Evaluation Directorate

May 2014

Implementation summary

This follow-up report on the implementation of management action plans outlines the measures taken by the various entities concerned in response to findings and recommendations. As part of the follow-up process, management action plans are reviewed annually until they are fully implemented, and the extent of implementation is assessed and reported to the Executive Committee, which serves as the Canadian Space Agency's (CSA's) Departmental Evaluation Committee.

This annual report contains the follow-up findings, as of March 31, 2014, of three evaluation projects for which reports and management action plans have been submitted to and approved by the Executive Committee. The following charts give an overview of the implementation status of the management action plan elements.

To do (0 management action plan element) In progress ≤ 50% (0 management action plan element) In progress > 50% (9 management action plan elements) Completed 100% (10 management action plan elements)
Management action plan elements 0 0 9 10

The following pages set out in detail the progress of the action plans for each of the evaluation projects.

Summative evaluation of the 2000-2009 Canada / European Space Agency (ESA) Cooperation Agreement

Evaluation project: Goss Gilroy

Evaluation objective

This summative evaluation was conducted by Goss Gilroy Inc. (GGI) Management Consultants in 2008 and was designed to assess the relevance, success and cost-effectiveness of the Canada/ESA cooperation agreement. The evaluation report was presented to the Executive Committee in June 2010 and subsequently approved in September 2010.

Program support and communication Policy monitoring Program renewal Priority alignment
40% 20% 20% 20%

Nature of recommendations

The evaluation report provided recommendations relating to program renewal, strategic priority alignment, program support and communication to Canadian industry and, finally, policy monitoring. Specifically, the five recommendations for program improvement consisted in renewing the agreement, clarifying ESA's role within the CSA's strategic priorities, developing policies and procedures to support ESA industrial opportunities and formally communicate them to industry, and monitoring impacts of space trends in the European Union (EU) on Canada.

Implementation status

The ESA management team at the CSA fully implemented the majority of recommendations stemming from the evaluation. There are two remaining action items that are still in progress, and each of them is more than 50% complete. Although the implementation of these action items has been progressing, a shortage of resources has impeded the ESA management's efforts to complete these remaining items.

The recommendation to support potential Canadian bidders included an action item that pertains to the development of a Guide for Canadian ESA delegates. In order to ensure that Canadian delegates have easy access to relevant information relating to the ESA procurement process and understand their role on ESA program boards, a draft guide has been developed and is currently in the consultation process.

Implementation status
To do 0% 0
In progress ≤ 50% 0
In progress > 50% 2
Completed 100% 3

The other remaining recommendation calls for the development of a coherent plan of action for communicating targeted ESA industrial opportunities to Canadian companies. The ESA management team has been working on the update of the ESA section of the CSA website to include relevant information and web links that could be useful to potential Canadian bidders and new entrants. This work is still in progress and is slated to be completed this year.

Summative Evaluation of the STDP for the Period from 2002-03 to 2007-08

Evaluation project: 570-2800

Evaluation objective

Government Consulting Services (GCS) conducted the evaluation of the Space Technology Development Program (STDP) (now referred to as Enabling Technology Development) in 2009-10, and the report was subsequently approved by the Executive Committee in June 2011. The evaluation focused on four main areas: program relevance, program design and delivery, program success, and cost-effectiveness.

Project support and communication Program objectives Project magagement
33% 34% 33%

Nature of recommendations

The evaluation report shed light on the need to employ the appropriate funding mechanism and identify R&D technology requirements in order to attain the program's objectives, to improve project management through more efficient procurement processes and service standards, and to provide project support and establish communication strategies with companies and CSA clients. The evaluation report contained six recommendations for program improvement.

Implementation status

Four of the six recommendations were completed prior to 2013-2014. Over the last year, the program team continued to work on two of the remaining recommendations, which pertain to project support and communication.

Of the remaining two recommendations, one was fully implemented this past year. Specifically, the program team reviewed its Request for Proposals (RFPs), thereby ensuring that the instructions to bidders are clear and well defined in all upcoming RFPs.

Implementation status
To do 0% 0
In progress ≤ 50% 0
In progress > 50% 1
Completed 100% 5

The implementation of a formal communication strategy to communicate project results to CSA clients is ongoing and is more than 50% complete. The program team has formalized the way project results are communicated to CSA clients by consulting and inviting the client to project kickoff meetings and providing the client with copies of final review packages. A client survey has also been developed, and the results from these surveys are compiled to produce client satisfaction reports. The first client satisfaction report is slated for June 2014.

Evaluation of the Earth Observation Data and Imagery Utilization Program for the Period from April 1, 2005, to March 31, 2010

Evaluation project: # 10/11 02 – 01

Evaluation objective

The evaluation of the Earth Observation Data and Imagery Utilization (EODIU) Program was completed internally by the CSA's Evaluation function. The report was approved by the Executive Committee in January 2012. As per the Treasury Board Secretariat's Directive on the Evaluation Function (2009), the evaluation responded to the five core issues related to program relevance and performance.

Program objectives Project support and communication Partner contributions and objectives Program performance and monitoring
62% 12% 13% 13%

Nature of recommendations

The evaluation report provided recommendations relating to program objectives, project support and communication to stakeholders, coordination of partner contributions, and monitoring of program performance. Specifically, the eight recommendations for program improvement were to:

  1. Increase industry's role in the development of applications for federal departments' needs;
  2. Support academia's participation in the development of applications;
  3. Determine the proportion of resources that should be allocated to applications developed from foreign data;
  4. Ensure that data access and sharing is facilitated by RADARSAT Constellation Mission's (RCM's) Data Policy;
  5. Allocate a greater share of program resources to awareness and knowledge transfer activities;
  6. Increase funding for the development of products and services that will improve market competiveness;
  7. Introduce mechanisms to identify development clusters and coordinate partners' contributions; and
  8. Identify program objectives and monitor the performance with the development of a PM strategy.

Implementation status

Implementation status
To do 0% 0
In progress ≤ 50% 0
In progress > 50% 6
Completed 100% 2

In 2013-2014, the program team continued to work on the implementation of its action items. The team made significant efforts to improve the way it manages and delivers its products and services. In 2014-2015, the team will review and adjust some of its action items to reflect the changes at the CSA in light of the implementation of the newly approved Space Policy Framework.

Two recommendations (#2 and #6) have been fully implemented in accordance with their target dates. The mechanisms to increase both industry's and academia's role in the development of applications have been implemented through the use of the CSA's Class Grant and Contributions program.

The remaining six recommendations are in progress and are each more than 50% complete. There are new governance mechanisms in place at the CSA , such as the Space Capacity and Review Board, to identify the program's priorities, record user needs and establish commitments. With respect to RCM data access and sharing, consultations with other government departments are underway to develop the RCM data and use policy. This action item is set to be completed by the time the first RCM satellite is launched, which is slated for 2018. The program team has also developed a draft communication plan to promote awareness-raising and knowledge transfer activities. Also, examples of project summaries provided by departments are compiled by the program team and published in the EO Express section of the CSA's website. Finally, a draft performance measurement strategy has been developed (with a completion target due date of March 2015), and some data collection mechanisms are already in place.