Management Action Plans Follow-Up - Evaluation - Annual Report as of March 31st, 2013
Audit and Evaluation Directorate
Table of Contents
This follow-up report on the implementation of management action plans outlines the measures taken by the various entities concerned in response to findings and recommendations. As part of the follow-up process, management action plans are reviewed annually until they are fully implemented, and the extent of implementation is assessed and reported to the Executive Committee, which serves as the Canadian Space Agency's (CSA) Departmental Evaluation Committee.
This annual report contains the follow-up findings, as of March 31, 2013, of three evaluation projects for which reports and management action plans have been submitted to and approved by the Executive Committee. The following charts give an overview of the implementation status of the management action plan elements.
The following pages set out in detail the progress of the action plans for each of the evaluation projects.
Summative Evaluation of the 2000-2009 Canada / ESA Cooperation Agreement
This summative evaluation was conducted by Goss Gilroy Inc (GGI) Management Consultants in 2008 and was designed to assess relevance, success and cost-effectiveness of the Canada/ESA agreement. The evaluation report was presented to the Executive Committee in June 2010 and subsequently approved in September 2010.
Nature of Recommendations
The evaluation report provided recommendations relating to program renewal, strategic priority alignment, program support and communication to Canadian industry and finally, policy monitoring. Specifically, the five recommendations for program improvement were:
- Renew the Canada-ESA agreement;
- Clarify ESA's role within the Long Term Space Plan;
- Develop and publicize policies to support organizations and new entrants;
- Develop a plan for communicating ESA industrial opportunities; and
- Monitor impacts of European Space Policy and European Union (EU) trends on Canada.
Over the last three years, the ESA management team at the CSA have addressed and implemented the majority of recommendations that have stemmed from the evaluation. Although there are a few remaining action items associated with recommendations that have yet to be fully implemented, progress has been made on a yearly basis.
The Canada-ESA Agreement was signed in December 2010 by the CSA's President and ESA's Director General and the treaty was then ratified and came into force on March 28, 2012.
In late 2012, final program decisions have been made at the ESA Ministerial. Industrial opportunities are slated to be communicated to Canadian organizations at an industrial forum in the near future. This recommendation is still in progress, more than 50% complete.
Progress towards developing policies to support potential Canadian bidders and updating the ESA management framework documents have been delayed due to limited resources. This recommendation is still in progress, less than 50% complete.
There has also been regular and continuous monitoring of the impacts the European Space Policy may have on Canada's participation in European space activities. The CSA's Policy and External Relations (PER) group is actively invovled in the development of various Canada-EU agreements and program files that have an impact on Canada-Europe space relations. The action details associated with this recommendation have been assessed over a two year timeframe. The actions taken to implement this recommendation have resulted in practices, policies and initiatives that are now integrated in CSA's policy and corporate functions and therefore deemed 100% complete.
Summative Evaluation of the STDP for the Period of 2002/03 to 2007/08
Government Consulting Services (GCS) conducted the evaluation of the Space Technology Development Program (STDP) (now referred to as Enabling Technology Development) in 2009-10 and the report was subsequently approved by the Executive Committee in June 2011. The evaluation focused on four main areas: program relevance, program design and delivery, program success, and cost-effectiveness.
Nature of Recommendations
The evaluation report shed light on the need to employ the appropriate funding mechanism and identify R&D technology requirements in order to attain the program's objectives, to improve project management through more efficient procurement processes and service standards, and to provide project support and establish communication strategies with companies and CSA clients. The evaluation report contained six recommendations for program improvement.
- Clarify program objectives;
- Map research and development (R&D) technology requirements with all proposed missions;
- Clarify procurement processes between CSA and Public Works and Government Services Canada (PWGSC) to reduce delays;
- Consider the viability of a two-step application process for grants and contributions (G&Cs);
- Provide support to companies for the preparation of proposals; and
- Implement a formal communication strategy to communicate project results to CSA clients.
The majority of the recommendations have been fully implemented over the last three years with a few pending actions that are slated to be finalized in 2013.
Program objectives have been clarified within the Program Alignment Architecture (PAA) and a Performance Measurement (PM) Strategy includes both contracts and contributions as outputs to achieve expected outcomes. The contribution mechanism has been established and the program awarded contributions in the first quarter of 2013. A standardized prioritization process through which R&D technology needs are identified and prioritized has been developed and implemented throughout 2012-13.
Recommendations relating to program management have been fully implemented. In addition to an operational contribution mechanism, the CSA also developed tools to clarify the major steps in the contract approval process, as well as the roles and responsibilities of each party involved.
In terms of project support and communication, guidance and support to companies will be provided at the Technology Forum in 2013. The program has also formalized the way project results are communicated to CSA clients by consulting and inviting the client to project kickoff meetings and providing the client with copies of final review packages. A client survey has also been developed. Results from these surveys will be compiled to produce client satisfaction reports. Recommendation 5 and 6 are in process, more than 50% complete.
Evaluation of the Earth Observation Data and Imagery Utilization Program for the Period from April 1, 2005 to march 31, 2010
The evaluation of the Earth Observation Data and Imagery Utilization Program (EODIU) was completed internally by the Evaluation Directorate. The report was approved by the Executive Committee in January 2012. As per the Treasury Board Secretariat's Directive on the Evaluation Function (2009), the evaluation responded to the five core issues related to program relevance and performance.
Nature of Recommendations
The evaluation report provided recommendations relating to program objectives, project support and communication to stakeholders, coordinating partner contributions, and monitoring program performance. Specifically, the eight recommendations for program improvement were to:
- Increase industry's role in the development of applications for federal departments' needs;
- Support academia's participation in the development of applications;
- Determine proportion of resources that should be allocated to applications developed from foreign data;
- Ensure data access and sharing is facilitated by Radarsat Constellation's Data Policy;
- Allocate a greater share of program resources to awareness and knowledge transfer activities;
- Increase funding for the development of products and services that will improve market competiveness;
- Introduce mechanisms to identify development clusters and coordinate partners' contributions;
- Identify program objectives and monitor the performance with the development of a PM strategy.
The program has begun to address many action items associated with recommendations, and in some cases, ahead of their timeline. Although the EODIU program is currently undergoing a strategic restructuring, it has made significant efforts in adjusting the way it manages and delivers its products and services while ensuring the essence of the recommendations are maintained.
Mechanisms to increase industry and academia's role in the development of applications have been developed. An interdepartmental coordination body for needs, priorities and commitments is being formed. This body will ensure industry plays a greater role in the development of applications and will also establish the proportion of resources required to fulfill the needs of each initiative. The G&C mechanism is operational with pending funding.
A governance mechanism to catalogue user needs has been initiated, which will help identify development clusters and coordinate partner contributions. The program is developing a comprehensive compilation of major projects and applications that it has funded over the last 5 years, which will highlight the benefits of the program. Finally, upon the completion of the program's current strategic review, the performance of the program will be monitored with an updated PM strategy.
- Date modified: