Evaluation of the Advanced Exploration Technology Development Program

Download the PDF version (1.57 MB)

For the period from April 1st, 2008 to March 31st, 2013

Project # 12/13 02-01

Prepared by the Audit and Evaluation Directorate

February, 2014

Table of Contents

  1. Introduction
  2. Background
  3. Evaluation Approach and Methods
  4. Results
  5. Conclusions and Recommendations
  6. References

List of Tables and Figures

Acronyms and Definitions Used in the Report

Acronyms

  • AETD: Advanced Exploration Technology Development
  • CSA: Canadian Space Agency
  • CSEP: Canadian Space Exploration Plan
  • DG: Director General
  • EDT: Enabling Technology Development
  • ESM: Exploration surface mobility
  • FTE: Full time equivalent
  • HQP: Highly qualified personnel
  • IP: Intellectual property
  • ISECG: International Space Exploration Coordination Group
  • ISS: International Space Station
  • MEPAG: Mars Exploration Program Analysis Group
  • MOU: Memoranda of understanding
  • NASA: National Aeronautics and Space Administration
  • NGC: Next generation Canadarm
  • OSM: Operational Space Medicine
  • PAA: Program alignment architecture
  • PER: Policy and External Relations
  • PM: Performance measurement
  • PMF: Performance measurement framework
  • PRET: Performance readiness evaluation technology
  • PWGSC: Public Works and Government Services Canada
  • R&D: Research and development
  • RFP: Requests for proposals
  • SME: Small and medium-sized enterprises
  • SD: Standard deviation
  • SP: Sub-program
  • SSP: Sub-sub-program
  • S&T: Science and technology
  • TB: Treasury Board of Canada
  • TBS: Treasury Board of Canada Secretariat
  • TRL: Technology readiness level

Definitions

Highly qualified personnel:
Individuals with university degrees at the bachelors' level and above.
Infrastructure:
Facilities used for AETD program activities. For example, analogue test sites, operations centres, and prototype and specialized equipments storage facilities.
Large enterprise:
Enterprises with 500 or more employees.
Operational capability:
An organization's capability to maintain, inspect, and operate tasks of complex space systems in preparation for future space missions.
Proof of concept:
A demonstration whose purpose is to verify that certain concepts or theories have the potential for real-world application.
Scientific capability:
An organization's capability to enhance scientific expertise enabled by technologies, which can include scientific publications, reports, demonstrations, etc.
Signature technologies:
Well-established or emerging Canadian products or product lines for which Canada is or has the potential to become a world leader and that are useable for multiple space missions. Examples of signature technologies include optics, robotic servicing, spectrometers, rovers, planetary drilling and extractions, advanced crew medical systems, etc.
Small and medium-sized enterprises:
Enterprises with fewer than 500 employees.
Solutions:
Include, but are not limited to, designs, reports, software, or construction, utilization, or operation of scientific or technological instruments and tools needed to work in space. Solutions may refer to a complete set of instructions and protocols that are required for the successful use of a prototype. A solution can be at a preliminary phase of development, but with sufficient information available to reach a decision concerning the furthering of its development.
Standard deviation:
The extent of deviation or dispersion from the average. A low standard deviation indicates that the data points tend to be very close to the mean, whereas a high standard deviation indicates that the data points are spread out over a large range of values.
Technological capability:
An organization's capability to produce a technology that performs in space to achieve space science and human spaceflight objectives.
Technology readiness level:
Management metrics used to assess the maturity of a particular technology, regardless of the discipline.

Executive Summary

The mandate of the Canadian Space Agency's (CSA) Advanced Exploration Technology Development (AETD) program is to carry out the technological, operational, and scientific development needed to prepare Canada for potential space exploration missions. The program is designed to shape the nature of Canada's contribution to potential international exploration and astronomy missions, and to manage the first stages of development of new solutions that are likely to be needed for space exploration.

This report contains the evaluation of the AETD program's relevance and performance over the course of the evaluation period (2008-2009 to 2012-2013). In addition, the performance of the Space Robotics Stimulus Initiative (implemented from 2009-2010 to 2012-2013 as part of the Government of Canada's Economic Action Plan) is examined. In carrying out this evaluation, a participatory and utilization-focus approach was employed that entailed the use of mixed quantitative and qualitative methods. Specifically, document and archival data reviews were performed, and interviews, focus groups, and online questionnaires were administered to over 60 key informants, including AETD contract recipients, CSA senior executives, AETD managers, and AETD staff. Methodological limitations (particularly with respect to data availability and accessibility, as well as to lack of baseline data and performance targets) were mitigated where possible.

As a whole, the AETD program demonstrates continued relevance and its performance has been generally successful, effective, and economical. However, evidence-based findings point to opportunities for program improvement. In addition, in order for the Canadian space exploration community to continue benefitting from the results that the program has achieved to date, there is a need for sufficient and long-term program funding.

In terms of relevance, the AETD program aligns with federal and departmental priorities, and is consistent with federal roles and responsibilities. In addition, the AETD program responds to the need for a federal program that leads the planning and coordination of Canada's space exploration activities and that supports the development of technological, operational, and scientific capabilities for future space exploration.

In terms of performance, the available data suggest that the AETD program has successfully achieved most, but not all, of its expected results. At the program output level, exploration plans and a roadmap have been developed (via the Canadian Space Exploration Plan) and contracts awarded to private enterprises and academic institutions have produced numerous technological solutions, though fewer science solutions. (Of note, the extent to which solutions stemming from the AETD program are suitable for public engagement could not be determined due to a lack of data and, by extension, nor could the extent to which public interest in space exploration has increased.)

As a result of the outputs produced, the AETD program has directly contributed to (a) improving the CSA's ability to make well-informed decisions for future missions and program development, (b) maintaining Canada's HQP and strengthening its infrastructure for space exploration, (c) maintaining Canada's position in the international space exploration scene, and (d) increasing Canada's space exploration technological and, to a somewhat lesser extent, operational capabilities. In addition, AETD-based solutions have increased Canada's space exploration scientific capabilities in a few niche areas, though the scope of the scientific capabilities developed is limited.

The successful achievement of most of the AETD program's immediate outcomes has led to generally favourable intermediate results, including increases in Canada's leadership in space exploration signature technologies (though not as notable with regard to science-based technologies) and transfers of dozens of AETD-based solutions to new or improved space and terrestrial applications. In addition, the AETD program has led to socio-economic benefits for the Canadian space exploration sector, such as encouraging numerous organizations to engage in space exploration endeavours and making space exploration a viable business area for many private enterprises. However, reductions in AETD program funding have recently threatened the economic viability of the Canadian space exploration sector, as evidenced in 2012 by a sharp decrease in the annual revenues generated by AETD contract recipients active in space exploration and by the Canadian space exploration sector at large. Furthermore, though preliminary evidence suggests that the AETD program has successfully positioned Canada for participation in future space exploration missions, diminished AETD program funding curtails Canada's ability to demonstrate its solutions in space or to commit to international partners. Consequently, despite the AETD program's initial success in positioning Canada for future space exploration missions, Canada risks losing these opportunities if sufficient long-term funding is not made available.

Given that this program was created only six years prior to the end of the evaluation period, it is too soon to ascertain whether it will achieve its ultimate outcome of ensuring successful Canadian participation in space exploration missions. Furthermore, the extent of Canada's influence on international space exploration decision-making could not be determined from the available data. However, preliminary evidence suggests the potential for success in both cases, assuming sufficient long-term program funding. In addition, considerable evidence shows that the AETD program has contributed to producing socio-economic benefits, such as increased access to new markets and new organizations joining the space exploration sector. Also, some of the AETD-based solutions that were transferred to other applications have been commercialized, though it was not possible to determine whether the generated revenues are aligned with AETD program targets.

With respect to the AETD program's overall efficiency and economy, the AETD program has delivered outputs and outcomes efficiently and has used its resources in an economically sound manner. For example, the program has produced good value with respect to use of public funds, especially given its small budget (relative to other space agencies). In addition, the proportion of total AETD program spending allocated to salaries and program management is within an appropriate range. However, by the end of the evaluation period, there were too few resources dedicated to the program to ensure the continued achievement of several expected results.

A more detailed analysis revealed certain aspects of the AETD program that are less efficient and economical. For example, inconsistent communication of AETD plans and priorities to AETD staff has at times hindered their ability to align their work with program objectives. Also, though the majority of CSA employees perceive that the AETD program and the CSA's Enabling Technology Development (ETD) program complement each other well, other key informants perceive potential redundancies in resource utilization and in the needs addressed by these two programs. In addition, inconsistencies in the turn-around times and in the interpretations of processes and procedures from Public Works and Government Services Canada have cause delays in the AETD contracting process and, on occasion, have resulted in missed opportunities for international collaboration.

Overall, the Stimulus Initiative achieved its two main objectives. First, the Canadian space exploration sector's key research and development (R&D) personnel were retained over the course of those years in which the majority of the Stimulus Initiative funding was spent. Second, the level of activities in space robotics was sustained, as evidenced by the development of over 30 Stimulus-based space exploration solutions. These solutions have accelerated the development of space exploration technologies and have increased the visibility of Canada's space exploration capabilities. In addition, through subcontracts and supplier contracts, the Stimulus Initiative contributed to the economic viability of the Canadian space exploration sector. Moreover, several contract recipients used Stimulus funds to increase their organizations' space exploration R&D expenditures and one Stimulus-based solution has already been flown on space missions. Of note, it was not possible to draw definitive conclusions regarding the efficiency of the Stimulus Initiative. However, evidence suggests that both external and internal factors contributed, to varying extents, to delays in the initiative's implementation and that, despite these challenges, the Stimulus Initiative was completed within its budget and schedule. With respect to economy, evaluation findings show that Stimulus financial resources were used in an economically sound manner.

Based on the evaluation findings and conclusions, the CSA's Evaluation function recommends that the AETD program:

  1. Conduct an analysis of the optimal level of resources that should be dedicated to scientific development by the AETD program and, based on the results of this analysis, clearly communicate to program stakeholders the AETD program objectives with regard to scientific development and allocate resources accordingly;
  2. Either create a clearer distinction between the ETD program and the AETD program or merge the two programs while ensuring that the planning and execution of technological, operational, and scientific developments remain integrated and aligned with future space exploration opportunities;
  3. Clarify the contracting processes regarding the AETD program's R&D activities and communicate these clarifications to AETD staff involved in contracting processes;
  4. Implement a systematic method of communicating plans and priorities to all AETD staff; and
  5. Include baseline data and targets in the AETD program's performance measurement (PM) strategy and make all performance data available in an accessible format.

1. Introduction

This document constitutes the final evaluation report of the Canadian Space Agency's (CSA) Advanced Exploration Technology Development (AETD) program. The evaluation was conducted during the 2012-2013 and 2013-2014 fiscal years by the CSA's Audit and Evaluation Directorate (specifically, the CSA's Evaluation function) in response to the Treasury Board of Canada's (TB) Policy on Evaluation (2009a), which requires that all federal government programs be evaluated every five years. The evaluation covers the period from 2008-2009 to 2012-2013.

2. Background

2.1 Program History and Description

In 2007-2008, the CSA created a program named Exploration Core (otherwise known as ExCore) to carry out the technological, operational, and scientific development needed to prepare Canada for potential space exploration missions. Following restructuring of the CSA in 2010-2011 and the implementation of the CSA's new Program Alignment Architecture (PAA) in 2011-2012, the AETD program was created, which includes both ExCore and space exploration planning. Since 2011-2012, the AETD program has been identified as sub-sub-program (SSP) 1.2.2.3 of the CSA's PAA, which falls under the Exploration Missions and Technology sub-program (SP) 1.2.2, which in turn falls under the CSA's Space Exploration program 1.2.

As per the CSA's PAA description (2012-2013a), the mandate of the AETD program is to develop advanced Canadian signature technologies to be used in potential astronomy and planetary missions that could be destined for the Moon, Mars, asteroids, or other celestial bodies. The program is designed to shape or determine the nature of Canada's contribution to potential international exploration and astronomy missions and could lead to spin-offs. In addition, the program includes terrestrial deployments in analogue sites that offer geological similarities with Martian or Lunar surfaces, where this technology and its operational aspects are tested and where exploration-related science is conducted for proof of concepts.

According to the AETD performance measurement (PM) strategy (approved in March 2013; CSA, 2013b), the program carries out this mandate by:

  1. developing the Canadian Space Exploration Plan (CSEP) for the CSA;
  2. anticipating the science, technological and operational needs for future missions likely to be needed in Canada, within the context of the Canadian government priorities and the international space exploration community, which is summarized in the CSEP; and
  3. investing in and managing the first stages of development of new technology, science and operational solutions that are likely to be needed for space exploration.

The program's target population is the Canadian Space Exploration Sector, in all geographic areas of Canada, which includes (CSA, 2013b):

  • private enterprises involved in development of science and technology used for space exploration, and commercialization of those technologies;
  • academic institutions involved in research and development of science and technology used for space exploration, and commercialization of those technologies; and
  • other units within CSA, other than the AETD program, involved in the planning and implementation of exploration missions.

The programs' stakeholders are the same as the target population, as well as:

  • foreign space agencies involved in collaborative arrangements with the CSA;
  • other federal government departments; and
  • the Canadian public, as it pertains to the dissemination of information resulting from AETD activities and outputs.

From 2009-2010 to 2012-2013, the Government of Canada implemented the Space Robotics Stimulus Initiative (hereafter referred to as the Stimulus Initiative) through the AETD program, as part of the Government of Canada's Economic Action Plan. The Stimulus Initiative's purpose was to respond to the needs of the space exploration industry to sustain the level of activities in space robotics and, in particular, maintain key research and development (R&D) personnel in Canada. To this end, an envelope of $110M was allocated by the Government of Canada's Budget 2009 to develop terrestrial prototypes for space robotic vehicles and to further develop robotics and other technologies. Specifically, two main projects fell under the Stimulus Initiative:

  1. Exploration Surface Mobility (ESM), aimed at designing and developing terrestrial prototypes of rovers and advanced technologies that can be initiated rapidly ($60M); and
  2. Next Generation Canadarm (NGC), aimed at developing terrestrial prototypes of the next generation of Canadarm for on-orbit servicing, that is, for use of robotic spacecraft to perform maintenance tasks on satellites in space or to assemble space structures ($50M).

A-base funding was used for project and risk management, as well as to complete the projects over four years (rather than the three years of Stimulus funding allocated by Budget 2009).

2.2 Governance, Roles and Responsibilities

As stipulated in the Canadian Space Agency Act, the objectives of the CSA are "… to promote the peaceful use and development of space, to advance the knowledge of space through science and to ensure that space science and technology provide social and economic benefits for Canadians" (Canadian Space Agency Act of 1990, sec. 4). The AETD program's mandate and functions were designed to align with the roles and responsibilities accorded to the CSA through this Act.

The Director General (DG), Space Exploration is accountable to the President of the CSA for the conduct of the AETD program (CSA, 2013b). The Director, Space Exploration Development reports directly to the DG, Space Exploration and is responsible for the AETD program. The Director's roles and responsibilities include:

  • interfacing with colleagues responsible for the International Space Station program (SP 1.2.1) and the Human Space Missions and Support program (SP 1.2.3), as well as for the Space Astronomy Missions and Planetary Missions programs (SSPs 1.2.2.1 and 1.2.2.2, respectively), in order to take into account their needs as they pertain to advanced technology, scientific, and operational solutions needed for space exploration;
  • allocating resources to the initial phases of developing science, technology, and operational solutions for space exploration; and
  • advising the DG, Space Exploration on the overall strategic direction that the CSA should take in space exploration.

AETD program managers report directly into the Director, Space Exploration Development and manage the AETD program's financial and human resources. Via matrix management, CSA staff support the work related to the AETD program.

2.3 Resource Allocation

Table 1 shows the total human and financial resources allocated to the AETD program, as well as the forecasted budget, over the course of the evaluation period. In addition, the resources allocated to the AETD program without including the Stimulus Initiative (i.e., A-base funding) and those allocated specifically to the Stimulus Initiative are presented.

Table 1: Resources allocated to the AETD program over the course of the evaluation period.

Type of Resource 2008-2009 2009-2010 2010-2011 2011-2012 2012-2013
FTEsNote a of table 1
Without Stimulus Initiative 47 33 20 28 25
Stimulus Initiative - 11 23 24 14
Program Total 47 44 43 52 39
Forecasted Budget ($)Note b of table 1 Note c of table 1
Without Stimulus Initiative 18,216 21,193 13,578 14,237 7,028
Stimulus Initiative - - 57,541 54,322 8,510
Program Total 18,216 21,193 71,119 68,559 15,538
5-year total = 194,625
Actual Spending ($)Note b of table 1
Without Stimulus Initiative
Salary Note d of table 1 4,841 3,544 2,069 2,945 2,736
O&M 11,960 12,496 7,802 9,298 2,774
Capital 4,593 1,425 1,985 413 631
Sub-total 21,394 17,465 11,856 12,656 6,141
Stimulus Initiative
Salary - 1,277 2,542 2,557 1,451
O&M - 14,196 47,587 56,767 7,067
Sub-total - 15,473 50,129 59,324 8,518
Program Total 21,394 32,939 61,985 71,980 14,658
5-year total = 202,956

Source: The CSA's Finance Directorate

Notes of table 1

Note a

FTEs are full time equivalents.

Back to footnote reference a of table 1

Note b

Values are represented in thousands of dollars.

Back to footnote reference b of table 1

Note c

As approved in annual work plans.

Back to footnote reference c of table 1

Note d

Excludes employee benefit plan.

Back to footnote reference d of table 1

2.4 Program theory

The logic model in Figure 1 depicts the theory behind the AETD program by providing a visual representation of how the resources allocated to the program are used to achieve expected outcomes. Links between logic model outcomes and expected results of the CSA's PAA are noted in the diagram. Also, because not all elements of the AETD program logic model pertain to the Stimulus Initiative, those that do (as per the logic model found in the Stimulus Initiative's Program Brief; CSA, 2009) are identified by thicker text box boarders.

The narrative that follows the logic model draws from the AETD program's PM strategy (CSA, 2013b) to explain in detail the various elements of the program theory.

Figure 1: AETD program logic model

Inputs: Inputs are the financial and non-financial resources used to deliver activities, produce outputs, and accomplish outcomes (Treasury Board of Canada Secretariat [TBS], 2010). The AETD program's inputs include both financial and human resources (i.e., FTEs).

Activities: Activities are the actions that a departmental organization undertakes to produce one or more outputs under the program (TBS, 2010). There are seven AETD program activities, including:

  1. Manage concept and prototype development projects (A1)
    • Under this activity, the preparation and management of concept studies and prototype development is performed. It is conducted to assess the feasibility of a number of Canadian solutions that could contribute science and technology concepts to space exploration missions. Among other tasks, this activity allows for maintaining contact with selected companies, ensuring that costs and schedules of projects are followed, and ensuring adherence to project scopes.
  2. Plan and execute procurement for AETD contracts (A2)
    • In order to plan procurement, the AETD team analyzes the priorities of the Canadian government and the strategic needs for space exploration at the international level, as well as the resources available among the program's target populations and other federal government departments.
    • Analyses of data from past and current missions are performed in order to anticipate new science, technology, or operational solutions likely to be required for future space exploration missions of interest to the CSA.
    • Requests for proposals (RFPs) are issued on an as needed basis, in line with Public Works and Government Services Canada (PWGSC) regulations. Thereafter, the AETD team follows the implementation of the contract, manages supplier relationship, and ensures that the desired deliverables are produced.
  3. Integrate and utilize advanced exploration prototypes (A3)
    • This activity deals with the integration of different prototypes in order to verify their performance when integrated and to perform end-to-end characterizations.
    • All AETD prototypes are maintained and certified in order to ensure safe operations. Specifically, prototypes maintenance strategies are established and then applied via processes, procedures, and tools. In addition, the configuration, utilization, and prioritization of analogue missions are managed, sometimes in collaboration with international partners through joint activities (such as international analogue deployment campaigns).
  4. Conduct analogue deployments (A4)
    • This activity entails the testing of science, technology, and operational solutions in earth-based or orbital environments that contain key features of the mission environment in which the solutions will eventually be used.
  5. Maintain operational infrastructure (A5)
    • Facilities used for AETD program activities (e.g., analogue test sites, operations centres, and prototype and specialized equipments storage facilities) are constructed and maintained.
  6. Participate in international meetings related to space exploration (A6)
    • Participation in international meetings related to space exploration is carried out in order to contribute to the CSA's knowledge and expertise, and to keep abreast of the latest orientations concerning space exploration. Examples of international meeting include, but are not limited to, the International Space Exploration Coordination Group (ISECG) and its sub-committees, National Aeronautics and Space Administration's (NASA) Mars Exploration Program Analysis Group (MEPAG), and International Space Station (ISS) expert working groups.
  7. Perform strategic planning (A7)
    • Scans of national and international space exploration activities and trends are performed and information derived from attendance at international meeting is drawn upon in order to:
      • Prepare major revisions of and regular updates to the CSEP;
      • Prepare strategic advice and recommendations for CSA senior management;
      • Develop the AETD program's PM strategy; and
      • Consult with Canadian communities related to astronomy and planetary exploration, with other stakeholders internal to the CSA, and with other government departments.

Outputs: Outputs are the direct products or services generated from program activities. They are usually within the control of the program (TBS, 2010). There are four AETD program outputs, including:

  1. Contracts for advanced exploration science, technology, and operational development (Op1)
    • Contracts are issued to ensure that the Government of Canada's identified needs for space exploration are fulfilled.
    • Contracting out the development of science, technology, and operational solutions creates opportunities for private companies and academic institutions involved in space exploration.
  2. Science, technology, and operational solutions for space exploration (Op2)
    • In the present context, the word "solution" refers to the different means of solving problems related to space exploration. The means used to solve problems could include, but are not limited to, concept studies, design, construction, utilization, and operation of scientific or technological prototypes, and instruments and tools needed to work in space. A solution may encompass a complete set of instructions and protocols that are required for the successful use of a prototype. A solution can still be at a preliminary phase of development, but with sufficient information available to reach a decision concerning the furthering of its development. For example, a solution could be a rover and any of its sub-systems, a sensor for docking a space craft, fuel cells, or a scientific instrument.
    • In response to the needs of the government of Canada and in coordination with the needs of international partners, the activities performed by the AETD program produce a portfolio of scientific, technological, and operational solutions at different level of maturity that are candidates for missions of interest to Canada or that can contribute to international space exploration missions.
  3. Solutions suitable for public engagement (Op3)
    • One of the goals for space exploration is to engage the Canadian public by increasing opportunities for interactive participation in space exploration. One of the ways that the AETD program seeks to achieve this goal is by adapting solutions, such as a planetary rover prototype, for use on earth to demonstrate the capabilities to the public. Another method is to create animations of concepts for public use.
  4. Exploration roadmaps and exploration plan (Op4)
    • The AETD program produces the CSEP, which identifies, within a 10-year planning horizon, high-level direction on the types of space exploration missions in which the CSA should participate, the outcomes to be achieved for Canadians from space exploration, the technologies and science solutions for space exploration in which the CSA should invest, and the nature of the CSA's interaction with industry, academia, and the public in relation to space exploration.

Immediate outcomes: Immediate outcomes are those that are directly attributable to the outputs delivered (TBS, 2010). In terms of timeframe, these are short-term outcomes. The AETD program has five immediate outcomes, including:

  1. Maintain or strengthen Canada's HQP and infrastructure capacity for future space exploration (Oc1)
    • The work contracted out by the AEDT program to industry or academia is expected to either maintain or strengthen Canada's highly qualified personnel (HQP).
    • The AETD program's public engagement activities should also help in building interest among young citizens to choose careers in science, technology, engineering, or mathematics, thereby increasing HQP.
    • The infrastructure used for space exploration includes analogue test sites, operations centres, and maintenance and storage facilities. These infrastructures are expected to be maintained or improved in order to meet industry, government, or academia's needs.
  2. Increase Canadian space exploration scientific, technological, and operational capability (Oc2)
    • The Canadian scientific and technological capability, as well as the operational know-how related to space exploration, is expected to be increased in academia, industry, and governmental segments.
    • Scientific capability: By working with the Canadian scientific community to set the scientific context for future space exploration missions that address selected science objectives, the AETD program is expected to lead to scientific discoveries, enabled by technologies.
    • Technological capability: The innovative engineering developed via the AETD program is expected to produce technologies that survive in the extremes of space environment, thereby enabling the realisation of space science and human spaceflight objectives.
    • Operational capability: The AETD program is expected to lead to increased expertise and readiness in maintenance, inspection, and operation tasks for complex space systems, in preparation for future missions.
  3. Maintain or improve Canada's position in the international space exploration scene (Oc3)
    • To maintain or improve its position on the international scene, Canada proposes scientific and technological solutions at different international meetings. The importance of maintaining favourable international relationships is crucial in space exploration because many space agencies cannot afford to carry out every mission on their own. Hence, international partnerships are sought.
  4. Maintain or improve the CSA's decision-making for future mission and program development (Oc4)
    • The CSEP (CSA, 2012a) is expected to allow for analyses of global opportunities in order to position the CSA's Executive Committee for making well-informed decisions about allocating resources to future missions and program development.
  5. Increase public interest in space exploration (Oc5)
    • The solutions suitable for public engagement developed via the AETD program are expected to have an impact on the awareness and interest that the Canadian public demonstrate vis-à-vis space exploration.

Intermediate Outcomes: Intermediate outcomes are those that are logically expected to occur once one or more immediate outcomes have been achieved (TBS, 2010). The AETD program has four intermediate outcomes, including:

  1. Maintain or increase the economic viability of the Canadian space exploration sector (Oc6)
    • Work performed by AETD program, along with contributions from the private and academic sectors, is expected to allow for the production of space technologies that can be integrated into space hardware, needed by national or international partners. It is expected that total revenue and number of employees in private enterprises in the Canadian space sector will be maintained or increased, thus contributing to its economic viability.
  2. Maintain or increase Canadian leadership in science and signature technologies (Oc7)
    • Along with other partners, Canada reaches the level of preparedness required to meet international exploration mission levels. Canada is then in a position to contribute space science expertise or instruments, and/or advanced technologies.
  3. Maintain or increase Canadian participation in international space exploration missions (Oc8)
    • Well-informed decision-making with regard to solutions development and mission selection, combined with Canada's capacity to influence decision-making by other space agencies, is expected to increase the likelihood of Canada participating in space exploration missions of strategic interest.
  4. Maintain or increase space technology transfers to new applications (Oc9)
    • Work performed by private sector and academia on space exploration projects lead to building connections between space and terrestrial technologies.

Ultimate outcomes: Ultimate outcomes are the highest-level outcomes that can be reasonably and causally attributed to a program as a consequence of one or more intermediate outcomes having been achieved (TBS, 2010). The AETD program has three ultimate outcomes, including:

  1. Maintain or increase socio-economic benefits of space exploration R&D and of Canadian participation in space exploration missions (Oc10)
    • Following investment by the AETD program in the industrial and academic sectors, as well as funding by other economic actors, socio-economic benefits for Canadians are expected to unfold, such as increasing numbers of organizations joining the space exploration sector, the commercialization of new applications, and industry's increased access to new markets.
  2. Ensure successful Canadian participation in space exploration missions (Oc11)
    • In order to be successful, a space exploration mission has to be performed according to precise specifications. It is not enough to deliver a scientific instrument or a new technology if it does not perform according to established requirements in space. Through Canada's leadership in science and signature technologies, it is expected that Canadian participation in space exploration missions will be successful.
  3. Maintain or increase Canada's influence in international space exploration decision-making (Oc12)
    • Canada's influence in international space exploration decision-making is maintained or increased by participating in different international space exploration decision-making fora. The fora can take the shape, for example, of advisory groups, consultative committees, working groups, and boards, but are not limited to these examples. Furthermore, the functions carried out in the different fora can be as an observer, participant, or chair; there may be voting rights or not, under the different functions.

2.5 Performance Measurement and Prior Evaluation of the Program

This report constitutes the first evaluation of the AETD program. At the time that the work for this evaluation began (April, 2012), a PM strategy had not yet been developed for the program, though expected results for the Stimulus Initiative had been identified in the initiative's Program Brief (CSA, 2009). The PM strategy that was approved in March 2013 was modelled after the program theory and evaluation strategy framework developed in the context of the current evaluation and, as such, this PM strategy contains a retrospective approach that incorporates Stimulus Initiative expected results. At the time that the present report was written, a revised PM strategy with a prospective approach was being developed to support ongoing program monitoring and subsequent evaluations of the AETD program.

3. Evaluation Approach and Methods

3.1 Purpose, Evaluation Questions, and Scope

In accordance with the TB Policy on Evaluation (2009a), the purpose of this evaluation is to provide an evidence-based, neutral assessment of the AETD program's value for money, with respect to both the program's continued relevance and performance. Evaluation findings and recommendations aim to support accountability to Parliament and Canadians and to support decision-making regarding managing for results, program improvements, and resource allocation. The intended users of this evaluation include the AETD program's Director, managers, and staff, the CSA President and Executive Committee, the Minister of Industry, Parliamentarians, the Canadian space exploration sector, and the Canadian public.

In keeping with the purpose of the evaluation, the approach adopted in its development was utilization-focused and participatory. Thus, a consultative group comprised of key stakeholder representatives (12 AETD program managers and staff, and four representatives of organizations that have received AETD funding) was formed at the onset of the evaluation to offer insight into stakeholder's information needs and to provide guidance and feedback throughout the evaluation process. The President of CSA's information needs were also solicited, resulting in evaluation questions that were then grouped according to the five core evaluation issues stipulated by the TB's Directive on the Evaluation Function (2009b):

  1. Continued relevance
    1. Continued need for the program
    2. Alignment with federal and departmental government priorities
    3. Alignment with federal roles and responsibilities
  2. Performance
    1. Achievement of expected outcomes
    2. Demonstration of efficiency and economy

Specific evaluation questions pertaining to each of these evaluation issues are listed in the left-most column of the evaluation strategy framework (Appendix A).

The resources invested in the AETD program, the activities carried out, and the results obtained between 2008-2009 and 2012-2013 comprise the scope of the evaluation. A case study on the performance of the two Stimulus Initiative projects, ESM and NGC, is also included.

3.2 Methods

The evaluation design was calibrated to align with the risks associated with the AETD program, which are identified as medium (high in corporate risk and low in materiality Footnote 1) in the CSA's 2012-2013 to 2016-2017 Departmental Evaluation Plan (2012b). A mixed-methods approach that combines qualitative and quantitative analyses was employed to produce multiple lines of evidence upon which to base evaluation findings and recommendations. The evaluation strategy framework (Appendix A) outlines the indicators, methods and data sources, and the responsibility for data collection for each evaluation question, grouped by core evaluation issues.

Specifically, the methods employed included document reviews, archival data reviews, and key informant survey instruments (specifically, interviews, online questionnaires, and focus groups).

Documents produced both internally (i.e., by the CSA) and externally (e.g., by Parliament, other government departments, the Canadian press, etc.) were reviewed primarily to evaluate the program's continued relevance, though they also informed the evaluation of program performance in some cases. For a complete list of documents reviewed, refer to the References section of this report. In addition, a search of the Internet was conducted in order to identify programs similar to the AETD program, with the aim of informing the assessment of the program's efficiency and economy.

Archival data provided by various units within the CSA (including the AETD program, Finance, Policy and External Relations [PER], Communications, and Library Services) were reviewed to evaluate the AETD program's performance. Because complete contract-relevant data were not readily accessible at the onset of the evaluation (see the Limitations section below), they were compiled from various archival sources, including AETD program files and the CSA's SAP, UNITAS, and ORIS databases. This compilation exercise resulted in the following types of data about AETD contracts funded during the evaluation period (not including contracts awarded for administrative purposes):

  • the timing and financing of these contracts (including contract amendments);
  • the nature of the contracting process (i.e., direct or competitive);
  • whether or not the contract was part of the Stimulus Initiative;
  • contract recipient demographics (i.e., type, size, and geographic location of organization);
  • the solutions developed from AETD contract; and
  • intellectual property (IP; belonging to the Crown or the contractor).

Four groups of key informants were surveyed in order to evaluate program performance and relevance:

  1. AETD contract recipients: Senior executives (in the case of private enterprises) and professors (in the case of academic institutions) working for organizations that received prime AETD contracts geared specifically toward technical, operation, or science solutions development during the evaluation period – not including contracts that (a) began prior to the evaluation period and ended in 2008-2009 or (b) contracts that were ongoing at end of the evaluation period.
  2. AETD staff: CSA employees who worked at least a moderate amount on the AETD program (often via matrix management) during the evaluation period.
  3. AETD managers: CSA employees who were either managers of the AETD program or the Director of the program during the evaluation period.
  4. CSA senior executives: DGs of the CSA's Space Exploration, Space Utilization, and Future Canadian Space Capacity programs, the current and former Presidents of the CSAFootnote 2, and the Director of the Government Liaison Office. (Note that the latter was asked a sub-set of questionnaire items focus specifically on program relevance.)

Of note, at the time of the interviews with AETD program managers and CSA senior executives, some individuals were in acting roles at higher levels of the organizational chart. For the purpose of this evaluation, they were interviewed in the capacity associated with their substantive positions.

Table 2 presents the number of key informants surveyed, the response rate relative to those solicited for their participation, and the type of survey instrument(s) administered, for each group of key informants.

Interviews were conducted either by telephone or face-to-face in cases where respondents worked at the CSA's headquarters in Saint Hubert, Quebec. Interview guides and online questionnaires items contained a mix of open- and closed-ended response options, whereas the focus group guide contained only open-ended questions. Appendices B through G contain copies of the interview guides, online questionnaires, and focus group guide. Each item in these documents is coded to indicate with which evaluation question it corresponds, as per the evaluation strategy framework in Appendix A. In addition to these items and in order to better contextualize the evaluation findings, all four groups of key informants were asked (a) to rate their overall satisfaction with the AETD program, (b) to identify those aspects of the program that that they value most and least, and (c) to describe any unintended outcomes of the program.

Table 2 : Number of key informants surveyed, response rates, and type of survey instrument(s) administered, by group of key informant.

Group of Key Informant Online Questionnaire Interview Focus Group
Number of Respondents Response Rate Number of Respondents Response Rate Number of Respondents Response Rate
AETD contract recipients 28Note a of table 2 76% 14Note b of table 2 50%
AETD staff 17 77% 16 73%
AETD managers 11 100%
CSA senior executives 5 83%

Notes of table 2

Note a

Among AETD contract recipient who responded to the online questionnaire, 19 (68%) represented private enterprises and nine (32%) represented academic institutions. In total, six (21%) received at least one Stimulus Initiative contract.

Back to footnote reference a of table 2

Note b

Only AETD contract recipients who responded to the online questionnaire were invited to participate in interviews. Among those interviewed, 12 (86%) represented private enterprises and two (14%) represented academic institutions. In total, three (21%) received at least one Stimulus Initiative contract.

Back to footnote reference b of table 2

The following factors pertaining to the analysis and treatment of data should be considered when interpreting the results presented below:

  • Percentages do not always add up to 100 due to number rounding;
  • When key informants did not respond to specific questionnaire items, the valid percent was reported. This is particularly noteworthy for the interview responses of CSA senior executives, one of whom was only asked a subset of the questionnaire items;
  • Quantitative data provided by AETD contract recipients were analyzed using t-tests, where applicable, in order to compare (a) responses provided by private enterprises with academic institutions and (b) Stimulus Initiative data with data pertaining to AETD contracts that were not part of the Stimulus Initiative (referred to as non-Stimulus contracts). In most cases, only statistically significant differences are reported; and
  • The explanations provided by AETD staff regarding some of the closed-ended responses they provided in the online questionnaire indicated a wide range of questionnaire item interpretations. In these cases, only open-ended, qualitative responses are reported.

3.3 Limitations

AETD Contract Information: Complete archival data pertaining to the contracts awarded by the AETD program during the course of the evaluation period were not readily accessible, in part due to the challenge of accurately associating those contracts awarded prior to the CSA's restructuring of the PAA in 2011-2012 with the current AETD program. In addition, discrepancies (e.g., differences in the number of contracts, in contract identification codes, etc.) were noted between the list of contracts provided by the AETD program, the information contained in the SAP system, and the information contained in UNITAS. Furthermore, data provided by AETD contract recipients in their interim or project closure reports had not been compiled on a consistent basis and could not be extracted within a timeframe that would be conducive to carrying out this evaluation. In light of these limitations, two strategies were employed to obtain the contract-relevant data that are presented in the Results section below:

First, in consultation with AETD program managers and the CSA's Finance and PER directorates, the Evaluation function triangulated contract information from three separate sources (AETD program files, SAP, and UNITAS) to create a compiled set of data pertaining to those AETD contracts funded during the evaluation period, excluding contracts awarded for administrative purposes (e.g., translation services). However, five contracts funded by the Stimulus Initiative were initially miscategorised as non-Stimulus contracts and this error was not discovered until after the online questionnaire for AETD contract recipient had been administered. Consequently, Stimulus-specific online questionnaire data were not collected for five of the 34 Stimulus contracts. Though the Stimulus Initiative findings based on the online questionnaire data represents a close approximation of the total Stimulus contracts, they should nonetheless be interpreted with caution.

Second, the online questionnaire that was administered to AETD contract recipients asked them to provide contract-specific information (e.g., HQP, FTEs, subcontracts, suppliers), as well more general information about the impact of receiving AETD contracts on their organizations (e.g., publications and conference presentations, investments in infrastructure and R&D, follow-on contracts, transfers to new applications, etc.). According to AETD managers, there has been no standard template for the type of information that AETD contract recipients should include in their reports. However, some types of information that were gathered through the online questionnaire had been occasionally, if not often, requested. Thus, AETD contract recipients who responded to the online questionnaire were asked to provide some information that they had already provided in their reports, thereby increasing the reporting burden. Furthermore, the validity of data collected retrospectively through the evaluation's online questionnaire was likely lower than the validity of the data provided initially in reports, due to a retrospective biasing effect and the potential for loss of data over time.

Baseline Data and Targets: The AETD program's PM strategy (approved in March 2013, at the end of the evaluation period; CSA, 2013b) does not include baseline data and targets for performance indicators. In an attempt to compensate for the lack of targets, AETD managers were asked for their opinions regarding whether the quantitative data provided by AETD contract recipients through the online questionnaire fall within the program's target range. However, AETD managers most frequently responded that they could not provide an informed opinion without knowing the target range. In cases where opinions were provided, they differed across AETD managers. Therefore, it was often not possible to determine whether the AETD program achieved its expected results based on these quantitative data. However, data provided by AETD contract recipients are provided below in order to facilitate establishing baselines for ongoing program monitoring and subsequent evaluations of the AETD program.

Key Informant Comparison Group: In order to assess program relevance and performance as it is perceived by diverse segments of the AETD program's target population, the original methodology designed for this evaluation included a comparison of data provided by AETD contract recipients with data provided by organizations that had bid on AETD request for proposals but that were never awarded an AETD contract. However, archival data pertaining to the latter group of organizations were incomplete, resulting in the identification of very few such organizations. Given that the sample size was too small to provide adequate data reliability and generalizability, this comparative methodology was removed from the evaluation design.

Onsite Interviews: Onsite visits to a sample of AETD contract recipient's organizations had originally been planned in order for the Evaluation function to better understand the AETD program context and to improve the quality of the data resulting from key informant interviews. However, reductions over time in the Evaluation function's resources necessitates that telephone interviews be conducted instead.

Public Engagement Data: Though indicators, data sources, and the responsibility for data collection were identified by the CSA's Communications Directorate at the onset of the evaluation, most of these performance data were not provided by the time that this report was written. Consequently, it was not possible to draw conclusive findings regarding the extent to which the AETD program has produced solutions suitable for public engagement (Op3) and the extent to which public interest in space exploration has been increased (Oc5).

Technology Readiness Level (TRL) Data: Increases in TRLs for technological solutions produced as a result of AETD contracts were initially identified as an indicator of the extent to which Canadian space exploration technological capabilities have increased (Oc2). However, with the exception of the NGC project, TRLs were not accessible for solutions produced by the AETD program and, therefore, this indicator was removed from the evaluation strategy framework. However, the AETD program is implementing a systematic method for tracking TRLs in the future, which should effectively support subsequent evaluations of the program.

Performance Measurement Framework (PMF) Data: As indicated in the AETD program's logic model and in Appendix A, the CSA's PAA identifies performance indicators associated with two of the AETD program's expected results, namely the production of science, technology, and operational solutions for space exploration (Op2) and the extent to which Canada's influence in international space exploration decision-making has been maintained or increased (Oc12). However, data for the latter indicator had not yet been collected at the time that this report was written.

IP Data: IP data pertaining to the number of patents (e.g., trade secrets and invention disclosures or declarations), as well as to foreground and background IP and licenses granted, were initially identified as indicators of the extent to which Canadian space exploration capabilities have increased (Oc2). However, these data could not be readily extracted from archival sources. Therefore, only key informants' qualitative responses and quantitative data about whether IP belongs to the Crown or to the contract recipient were available for contracts awarded by the AETD program.

4. Results

This section presents the AETD evaluation findings, first with respect to program relevance (i.e., the extent to which the program is appropriate to the federal government and addresses the needs of Canadians) and second with respect to program performance (the extent to which effectiveness, efficiency, and economy are achieved by the program).

Codes that identify specific elements of the evaluation strategy framework (Appendix A) and, where applicable, that link to the program logic model are included in the text boxes that introduce each evaluation question and finding below.

4.1 Relevance

The AETD program's relevance was evaluated with regard to (1) the linkages between program objectives and federal government priorities, (2) the linkages between program objectives and departmental strategic outcomes, (3) the role and responsibilities for the federal government in delivering the program, and (4) the extent to which the program continues to address a demonstrable need and is responsive to the needs of Canadians.

4.1.1 Alignment with Federal Government Priorities

Evaluation Question # 1: Are the program's objectives aligned with federal government priorities? (R1)

Evaluation Finding # 1: The objectives of the AETD program are aligned with federal government priorities, including the Government of Canada's Whole-of-Government Framework, the overarching priorities and core principles in science and technology (S&T), and the government's commitment to space exploration.

Canada has traditionally been an important player in space and, through its past investments and successes, Canada and its space sector have established a strong reputation for specialized expertise in several areas of space research, space robotics, and space exploration (Paradis, 2012). A review commissioned by the federal government concluded that,

"Space has been important to Canada over the last half century, but not nearly as important as it will be over the next half century. Simply put, it will be an essential tool of nationhood for a country that aspires to provide long-term prosperity and security to its people, protect its natural environment, and discharge its international responsibilities" (Industry Canada, 2012, p.2)

In particular, the report found that, "...participating in space exploration and science missions make nations richer, safer, smarter, and better-respected" (Industry Canada, 2012, p.3). Furthermore, the Parliamentary-Secretary to the Minister of Industry recently stated that, "Canada is a global leader in advanced research and space technology development. Our government recognizes Canada's contribution to innovative space technologies, and we are committed to supporting space exploration, commercialization and development" (Industry Canada, 2014, para. 2).

Recognizing the importance of space exploration and building on Canada's heritage in space exploration, the objectives of the AETD program support the government's priority to build an innovative, knowledge-based economy through science, technology, and research excellence, as well as through long-term competitiveness (Department of Finance Canada, 2012).

Further evidence of alignment with federal government priorities was found through the Government of Canada's identified outcomes, its science and technology priorities, and its commitment to the next phase of Canada's space program.

The Government of Canada's Whole-of-Government Framework: The AETD program's activities align with the economic affairs area of spending represented in the Whole-of-Government framework (TBS, 2011). More specifically, the program aligns with the government's expected outcomes of creating an environment conducive to economic growth and of preparing Canada for future challenges by investing in innovative research and development.

Canada's Science &Technology Strategy: The AETD program's objectives also correspond to three of the government's S&T priorities as identified in Mobilizing Science and Technology to Canada's Advantage (Industry Canada, 2007). These three S&T federal priorities entail developing an entrepreneurial advantage, a knowledge advantage, and a people advantage.

First, the AETD program's objective to maintain or increase space technology transfers to new applications directly supports the development of an entrepreneurial advantage where "Canada must translate knowledge into commercial applications" (Industry Canada, 2007, p.11). Moreover, some of the CSA's senior executives pointed to promoting innovation as one of the ways in which the program is aligned with federal priorities. Commercialization, access to new markets, and terrestrial applications of space science solutions were also mentioned by AETD contract recipients as important outcomes of the AETD program. The federal government has also referred to the Canadian space industry as a "sophisticated research and innovation leader, successfully turning its investment in knowledge into a global advantage in several niches areas, including robotics..." (Department of Finance Canada, 2010a, p. 82).

Second, by supporting the development of science solutions and technological and operational capabilities, the objective of the AETD program to increase Canadian leadership in science and signature technologies corresponds to the government's S&T priority to develop a knowledge advantage (Industry Canada, 2007). This alignment was further noted by the Minister of Industry when the rovers and scientific instruments produced under the Stimulus Initiative were unveiled. At this unveiling, the Minister said:

"These model rovers are a stellar example of how our Government's investments in space are strengthening Canadian S&T excellence, fostering industrial innovation and commercialization, and positioning Canada for continuing economic growth in the knowledge economy" (CSA, 2012c, para. 4).

Third, the AETD program's objective to maintain or increase Canada's HQP capacity in space exploration is aligned with the government's priority to develop a people advantage, where "Canada must be a magnet for highly skilled people" (Industry Canada, 2007, p. 11). This alignment was further outlined in Canada's Economic Action Plan: 6th Report to Canadians, which stated that the Stimulus Initiative for advanced space technologies was intended to provide jobs for highly skilled personnel (Department of Finance Canada, 2010b). In a similar vein, when asked about the AETD program's alignment with federal priorities, some CSA senior executives referred to developing HQP and a knowledge economy as ways in which the program supports government priorities.

In addition, the AETD program's objectives align with two federal government S&T core principles. First, Canada's Science and Technology Strategy notes the importance of "focusing on strategic areas where Canada can be a world leader, [...] by strategically targeting funding in areas of opportunity that build on national strengths" (Industry Canada, 2007, p. 46). This core principle of focusing on strategic areas where Canada can be a world leader is reflected in the strategic planning activity of the AETD program. The program aims to focus its investments on signature technologies and science solutions in which Canada is or has the potential of becoming a world leader. Moreover, the Stimulus Initiative pertained to investing in the development of emerging and established signatures technologies, such as robotics, optics, and drills.

The second core principle of Canada's Science and Technology Strategy is to foster partnerships with private enterprises and universities, both within Canada and abroad (Industry Canada, 2007). The AETD program aligns directly with this core principle because the program is designed to collaborate with Canadian industry and academia involved in space exploration by contracting out the development of science, technology, and operational solutions. Finally, given that Canada's participation in space exploration missions are based on international collaboration with other space agencies, the AETD program aims to foster international partnerships by participating in international space exploration committees and by conducting joint analogue deployments with international partners.

Space Policy Framework: In terms of the government's priorities in the next phase of Canada's space program, the recently released Space Policy Framework (CSA, 2014) refers to space exploration as a strategic area for action. The AEDT program aligns with this strategic area of action by developing plans and roadmaps for Canadian space exploration. The AETD program's focus on increasing Canada's leadership in science solution and signature technologies also aligns with the Space Policy Framework's commitments to "invest in the development of Canadian contributions in the form of advanced systems and scientific instruments as part of major international endeavours" and to "ensure that Canada is a sought-after partner in the international space exploration missions" (CSA, 2014, p.12).

In addition, one of the core principles of Canada's Space Policy Framework is to inspire Canadians. As explained by many of the AETD contract recipients and CSA senior executives interviewed for this evaluation, preparing for and taking part in future space exploration missions inspires Canadians and instils a sense of national pride.

Of note, when CSA senior executive were asked to comment specifically on the extent of alignment with federal government priorities, all interviewees indicated that they perceive strong alignment between the AETD program's objectives and federal priorities.

4.1.2 Alignment with the Departmental Priorities

Evaluation Question # 2: Are the program objectives aligned with departmental strategic outcomes? (R2)

Evaluation Finding # 2: The program's objectives are aligned with the CSA's strategic outcome and with Canadian space exploration priorities and goals.

All of the CSA senior executives who were interviewed for this evaluation reported that the AETD program's objectives are aligned with the CSA's strategic outcome, which is to ensure that "Canada's exploration of space, provision of space services and development of its space capacity meets the nation's need for scientific knowledge, innovation and information" (CSA, 2012d, p. 12). Specifically, the AETD program's objectives align with the CSA's strategic outcome by focussing on the development of space exploration capabilities and capacities.

Furthermore, the activities and objectives of the AETD program directly contribute to the achievement of the CSA's Space Exploration program's priority, which is outlined in the 2012-2013 Report on Plans and Priorities as follows (CSA, 2012d):

"Foster knowledge and innovation through space exploration [by supporting] the development of valuable Canadian science, signature technologies and qualified astronauts to international space exploration endeavours. [The Space Exploration program] will support the development of space technologies and knowledge with potential for terrestrial benefits mainly through the optimal utilization of the International Space Station (ISS) and the demonstration of advanced robotics technologies" (CSA, 2012d, p.13).

Other important Canadian space priorities are outlined in the CSEP (CSA, 2012a), which was developed by the AETD program and which provides Canada's vision, goals, and priorities in space exploration. As per the CSEP, Canada's goals in space exploration are to (a) gain fundamental knowledge of the universe, (b) expand Canadian presence in space and other planets, (c) maintain and strengthen Canadian signature technologies and, (d) increase the space exploration stakeholder base, which includes supporting the commercial expansion to new markets (CSA, 2012a). The AETD program's activities and outcomes support the goals set out in this document, notably including the goal of maintaining and strengthening Canadian signature technologies.

4.1.3 Alignment with Federal Roles and Responsibilities

Evaluation Question # 3: Is the program consistent with federal roles and responsibilities? (R3)

Evaluation Finding # 3: The AETD program is aligned with federal roles and responsibilities, as outlined in the Canadian Space Agency Act. The program falls within the jurisdiction of the federal government because the CSA is the federal organization responsible for space exploration activities. It is also within the federal government's roles to (a) lead the planning and coordination of Canadian space exploration, (b) maintain the Canadian space sector's technological capabilities by supporting R&D and industry's competitive advantage, and (c) negotiate international agreements for Canadian participation in space exploration missions.

According to the Canadian Space Agency Act, the CSA has the legislated mandate to:

"Promote the peaceful use and development of space, to advance the knowledge of space through science and to ensure that space science and technology provide social and economic benefits for Canadians" (Canadian Space Agency Act, 1990, sec. 4).

More specifically, the activities and objectives of the AETD program are consistent with several of the CSA's responsibilities, including to:

  • "plan, direct, manage and implement programs and projects relating to scientific or industrial space research and development and the application of space technology;
  • promote the transfer and diffusion of space technology to and throughout Canadian industry; and
  • encourage commercial exploitation of space capabilities, technology, facilities and systems" (Canadian Space Agency Act, 1990, sec.5).

Consistency with federal roles and responsibilities was reaffirmed by all the CSA senior executives interviewed for this evaluation. In addition, according to the Science and Technology Strategy, it is the government's role to "ensure a competitive marketplace and create an investment climate that encourages the private sector to compete against the world on the basis of their innovative products, services, and technologies" (Industry Canada, 2007, p. 11). Furthermore, the role of government in the space sector is articulated in the Aerospace Review, which states that the government supports the Canadian space industry by "supporting R&D that might take years to produce marketable results but has the potential to generate substantial benefit to the public good, in part through risk sharing" (Industry Canada, 2012, p.5).

It is also within the government's role to lead the planning and coordination of Canadian space exploration activities. As nations' space exploration programs are led by national governments, it is within the role of national governments to lead their country's participation international committees related to space exploration. For example, the international coordination of mutual efforts in space exploration is conducted by fourteen space agencies, including the CSA, via the ISECG. Furthermore, the negotiation and creation of bilateral and multilateral partnerships are spearheaded by national governments, which is consistent with the AETD program's role to position Canada as a partner for future space exploration missions.

4.1.4 Continued Need of the Program

Evaluation Question # 4: Does the program continue to address a demonstrable need and is it responsive to needs of Canadians? (R4)

Evaluation Finding # 4: The AETD program addresses two main needs: (a) the need to plan and coordinate Canadian space exploration activities and (b) the need to maintain the key space capabilities of the Canadian space exploration sector. The program is designed to meet these needs by identifying and supporting scientific, technological, and operational solutions, thereby preparing and positioning Canada for future space exploration missions. More specifically, the program responds to these needs via strategic planning, integrating and operating technologies, and supporting industry to maintain its technological capacities and competitiveness. In addition, the program addresses the scientific needs of the Canadian space exploration community.

Multiple lines of evidence, including information collected via a document review and key informant interviews with CSA senior executives, AETD managers, and AETD contract recipients, were examined in order to ascertain whether the AETD program continues to address a demonstrable need and whether it is responsive to the needs of Canadians. Of note, the vast majority of key informants (88% of AETD contract recipients, 100% of CSA senior executives, and 100% of AETD managers) strongly agreed that there is a continued need for this program.

Based on the analyses, two main needs addressed by the AETD program were identified:

  1. The need to provide leadership, planning, and coordination of Canadian space exploration activities. This need stems from the long periods of preparation required for space initiatives and the advantages of integration and operations for attaining mission readiness; and
  2. The need to maintain the technological and scientific capabilities of the Canadian space exploration sector. This need relates to strengthening Canada's competitive advantage in niche areas and addressing the research needs of the Canadian scientific community.

Need to Provide Leadership, Planning, and Coordination of Canadian Space Exploration Activities: The AETD program was created in 2007 to address the need to unify space exploration activities in order to better prepare Canada for future space exploration missions (Piedboeuf, 2010). As former Minister of Industry Tony Clement pointed out, "space missions are like marathons: it takes years of preparation, development, and dedication to achieve gold" (CSA, 2010, para. 2). In addition, given that space initiatives require long periods of preparation (which includes requirements analyses, concept definitions, and technology developments), there is a need for the government to lead the planning and coordination of space exploration activities (Industry Canada, 2012). Through strategic planning with national stakeholders and by coordinating mutual efforts with international partners, the program aims to anticipate the science, technological, and operational needs for future missions of interest to Canada (CSA, 2013b). Indeed, CSA senior executives indicated that the AETD program is primarily needed in order to prepare Canada for participation in future space missions, which necessitates strategic planning in the development of R&D for space exploration.

Integrating, testing, and operating solutions are important activities for preparing Canada for future space exploration missions. Several AETD managers pointed to the importance of integrating the scientific, technological, and operational aspects of prototypes. They explained that this integration helps reduce certain risks associated with technology development and potential cost overruns, which in turn contributes to mission readiness. Furthermore, AETD managers and AETD contract recipients emphasized that demonstrating end-to-end operations of prototypes in analogue sites helps to further showcase Canadian expertise, increase Canada's credibility, and position Canada as a potential partner.

In the absence of the AETD program, CSA senior executives said that the lack of strategic and coordinated planning would notably diminish Canada's preparedness for future space missions, which would weaken Canada's international credibility. In a similar vein, the majority of AETD contract recipients and AETD managers explained that, without the program, there would be missed international space exploration opportunities for two main reasons. First, the likelihood for partnerships is increased when risks associated with space technology development are reduced.. Second, in the absence of the AETD program, participation in planning groups with the international space exploration community would be considerably reduced, making it difficult to identify potential Canadian contributions.

Need to Maintain the Technological, Operational and Scientific Capabilities of the Canadian Space Exploration Sector: The need to sustain the Canadian space exploration sector's expertise in science solutions and its technological capability in signature technologies for space exploration can be further divided into two categories of needs:

  1. The need for Canadian industry to stay competitive and maintain its technological capabilities; and
  2. The need to address the research requirements of the Canadian scientific community.

With respect to the first of these categories, there are two main reasons why maintaining technological capability in space exploration is needed: first, to ensure that Canada can deliver on advanced space exploration technologies when exploration missions arise, and, second, to support Canada's competitive advantage in technology development through both space and terrestrial applications in new markets. In explaining the rationale for the Stimulus Initiative, the former Minister of Industry outlined the need to "ensure that Canada retains its enviable reputation as a world-class leader in space robotics and advanced space technologies [by...] investing in innovative Canadian companies so that they retain their competitive edge, grow our specialized workforce, and position Canada on the cutting-edge of technology to play a key role when international opportunities arise" (CSA, 2010, para. 2).

The AETD program is designed to address the need of the Canadian space exploration sector to remain innovative. Doing so requires maintaining HQP capacity and technological and operational capabilities in niche areas, which also contribute to the sector's competitiveness. By advancing technology and maintaining its key capabilities and highly skilled workforce, the AETD program aims to ensure that the space exploration sector is not standing still in the absence of missions.

When asked about the Canadian needs addressed by the AETD program, the most common response provided by AETD contract recipients was that the program helps maintain a competitive high-tech economy and a dynamic space industry, which enables Canadians to occupy highly qualified jobs and participate in advanced science and technology development. Several AETD contract recipients also stated that the AETD program responds to the need to expand Canadian space companies' business profile and global branding, which impacts their competitiveness in their core business areas. This competitive advantage is not only applicable to the space market. AETD contract recipients explained that the human and technical skills required for developing technologies and scientific instruments are at the leading edge of innovation and are transferrable to other markets, such as mining and transportation. Developing Canadian technologies for terrestrial use and creating new opportunities for Canadian industry, both in space exploration and in other markets, were also mentioned by both CSA senior executives and AETD managers as needs addressed by the program. In addition, space technology transfers for terrestrial use have been identified as beneficial to Canadians (e.g., electric rovers can contribute to Canada's sustainable development transportation efforts; Grenier et al., 2008).

When asked what gaps would exist without the program, the majority of AETD contract recipients pointed to the difficulty in investing in advanced technology development without a federally run program because the associated risks would be too high. According to AETD contract recipients, this barrier would impede efforts to maintain the space exploration industry's expertise and key capabilities, and it would diminish the extent of commercialization of high tech developments. Similar gaps were identified by AETD managers, who said that there would be a shortfall in key expertise, knowledge, and technological capabilities in Canada.

Turning now to the research needs of the Canadian scientific community, the Aerospace Review (Industry Canada, 2012) notes that space exploration and science respond to a fundamental need for knowledge. The AETD program is designed to work with the Canadian scientific community to set the scientific context for future space exploration missions by addressing selected scientific objectives (CSA, 2013b). Specifically, the program aims to respond to the research needs of the Canadian space exploration community by contracting out the development of new science solutions and scientific instruments to both university and industry. In line with this objective, a few AETD contract recipients reported that the program addresses their need to develop a comprehensive capacity in science solutions, which at times can foster academic-industrial collaborations and follow-on research for Canada's research sector. In addition, both AETD managers and CSA senior executives reported that sustaining space expertise in Canada and generating new knowledge about space and the universe were important needs addressed by the program. Moreover, some CSA senior executives and AETD contract recipients reported that space exploration programs, such as the AETD program, encourage Canadians to pursue science, technology, engineering, and mathematics (STEM) careers. In the absence of the program, some AETD contract recipients said there would be decreased Canadian capacity for advanced scientific research.

4.2 Performance

The AETD program's performance was evaluated with regard to (1) its progress toward producing outputs and achieving expected outcomes, and (2) its resource utilization in relation to the production of outputs and progress toward expected outcomes. The evaluation of these core performance issues are conducted sequentially below. At the end of this section, a case study on the Stimulus Initiative and key informants' overall levels of satisfaction with the AETD program are presented.

4.2.1 Achievement of Expected Outputs and Outcomes

The extent to which the AETD program has achieved each of the outputs and outcomes identified in the logic model is evaluated in this section of the report, which is divided into output, immediate outcome, intermediate outcome, and ultimate outcome sub-sections.

4.2.1.1 Outputs

Evaluation Question #5: Have contracts for advanced exploration science, technology and operational development been awarded? (Op1)

Evaluation Finding #5: The vast majority of AETD contract spending was allocated to space exploration technological development and, to a lesser extent, to operational development. One quarter of all AETD contract spending was allocated to scientific development, though science-related contracts represented about two-fifth of non-Stimulus contract spending.

As indicated in Table 1 (Section 2.3), the entire AETD program forecasted budget over the five years of the evaluation period was fully invested. Over this period of time, a total of $153M (including amendments) was spent on 145 contracts, not including contracts pertaining to program administration.

In about half of the 145 cases (48%), AETD contracts pertained to a combination of more than one type of space exploration development (i.e., technological, operational, and scientific development). Table 3 shows that the vast majority of AETD contract spending was allocated to technological development and approximately three quarters was allocated to operational development. However, only about one quarter was allocated to scientific development. Given that the Stimulus Initiative targeted space robotics, Table 3 also presents the percent of contract spending by type of space exploration development for non-Stimulus contract. These data show that the majority of non-Stimulus contract spending was allocated to technological development and that scientific development represented approximately two-fifths of non-Stimulus contract spending.

Table 3: Percent of contract spending by type of contract for all AETD contacts and for only non-Stimulus contracts.

Type of ContractNote a of table 3

  Technological Operational Scientific OtherNote b of table 3
Percent of AETD contract spending 96% 73% 26% 1%
Percent of non-Stimulus contract spending 70% 48% 43% 4%

Notes of table 3

Note a

Type of contract was determined based on original contracts, not on amendments.

Back to footnote reference a of table 3

Note b

The "other" category includes, for example, contracts geared toward developing space exploration plans and roadmaps, overseeing the construction of infrastructure, developing databases, etc.

Back to footnote reference b of table 3

Among the 145 AETD contracts awarded:

  • 23% were funded through the Stimulus Initiative, representing 84% of total contract spending;
  • 63% were awarded through a competitive process, representing 54% of total contract spending;
  • 15% were awarded to academic institutions, representing 3% of total contract spending;
    • not including Stimulus contracts intended for industry, 19% were awarded to academic institutions, representing 16% of contract spending;
  • among private enterprises, most contracts were awarded to small and medium-sized enterprises (SMEs; 72%, representing 35% of contract spending).
  • All but one contract were awarded to Canadian organizations, the vast majority of which were located in Ontario or Quebec (94%, representing 99% of total contract spending). The remaining contracts were awarded to organizations in British Columbia (4%), Alberta (1%), and Nova Scotia (<1%).

Though key informants were not specifically asked about their views on the contracting process, they spontaneously offered thoughts and suggestions on this matter. One concern raised by CSA senior executives and AETD staff was that the contracting process requires detailed specification of methods, deliverables, and due dates, which can at times be incompatible with contracts geared toward R&D and innovation. A second concern raised by CSA senior executives, AETD management, and AETD staff pertained to inconsistencies in the interpretation of processes and procedures regarding the contracting process. Archival program data showed that the length of time between when a requisition was sent to PWGSC and the contract award date ranged across contracts from 22 to 753 days, with an average of 193 days (SD = 138.2).

Evaluation Question #6: Have science, technology, and operational solutions for space exploration been developed and delivered? (Op2)

Evaluation Finding #6: Solutions for space exploration have been developed and delivered as a result of the AETD program. Most solutions were geared toward technological development, though operational and scientific solutions were also produced.

According to the CSA's Departmental Performance Reports (2012e and 2013c), 47 AETD-funded solutions were under development in 2011-2012 and 34 were under development in 2012-2013, thereby exceeding that program's PMF targets for the latter year. Because the CSA introduced a new PAA in 2011-2012, PMF data were not available for previous years.

Of the 145 AETD contracts which received AETD funding during the evaluation period, 89% had ended by the end of the evaluation period. In total, 175 AETD-based solutions were developed and delivered during the evaluation period. The majority (75%) of AETD contracts geared toward solution development produced between one and three solutions, with an average of 1.4 (SD = 1.7) solutions per contract.

Based on the type(s) of contracts awarded, it was estimated that:

  • 85% of these solutions pertained to technological development (70% not including Stimulus Initiative solutions);
  • 52% pertained to operational development, (39% not including Stimulus Initiative solutions); and
  • 30% pertained to scientific development for space exploration (44% not including Stimulus Initiative solutions).

Evaluation Question #7: Have solutions been suitable for public engagement? (Op3)

Evaluation Finding #7:Though evidence suggests that some AETD-based solutions have been used to engage the public, the extent to which solutions stemming from the AETD program are suitable for public engagement cannot be determined based on the available data.

As mentioned in the Limitations section of this report, insufficient data were available concerning which of the 175 AETD-based solutions were suitable for public engagement or the percent of these that were utilized for public engagement.

Partial data provided by the CSA's Communications Directorate showed that:

  • One AETD-based solution was used to inform a speech about rovers delivered by the Minister of Industry in October 2012;
  • A section of the CSA's website that is intended to inform the public included information about three AETD-based solutions; and
  • A web section containing seven videos about rovers developed via the AETD program was being developed at the time that this report was written.

Evaluation Question #8: Have space exploration roadmaps and plans been produced? (Op4)

Evaluation Finding #8: The CSEP outlines plans and a roadmap for Canadian space exploration initiatives. The CSEP was developed following an extensive consultative process that was valued by the majority of the Canadian space exploration community, though some organizations would have preferred that their input had had more of an impact on the CSEP.

The CSEP (CSA, 2012a) was produced by the AETD program to outline the CSA's Space Exploration program's goals, priorities, and roadmap, with the objective of providing the Canadian space exploration community with the context in which Canadian space exploration initiatives are identified and pursued. This document was developed following ongoing discussions with international partners and consultation with over 200 Canadian stakeholders via four separate consultation process that spanned a two-year period. Specifically, the stakeholders consulted included representatives of Canadian industry, academia, and other government departments interested in space exploration, as well as CSA employees and senior executives.

Information provided by key informants about the extensive consultations that took place in developing the CSEP indicates that they were generally well-received by members of the space exploration community. For example, some AETD staff said that this consultative process led to the unintended, positive outcome of strengthening the Canadian space exploration community's relationships and allowing for a shared, broader understanding of the Canadian space sector. As another example, when AETD contract recipients were asked for their opinions about this consultation process, only a minority (29%) of those who had been consulted (representing a full 93% of interviewees) indicated that they would have preferred to have been consulted somewhat more extensively. The majority (57%) reported satisfaction with the extent to which they had been consulted and some even cited these community engagement efforts as an aspect of the AETD program that they value the most.

Despite predominant satisfaction with the extent of consultation, contract recipients were less unified in their views regarding the extent to which their organization's input had an impact on the CSA's space exploration plans and roadmaps. A little over one third of interviewees (39%) felt their input had a considerable impact, a little under one third (31%) reported a moderate impact, and approximately one quarter (23%) reported only a slight impact of their organization's input on the CSA's space exploration planning. The reasons cited by those who were less satisfied with the extent of their input's impact included (a) that some of the consultation formats did not lend themselves well to dialogue, (b) that the planning was presented to them as a fait accompli rather than an opportunity to contribute to its development, and (c) that academia's interests were not well address, resulting in plans that do not adequately outline long-term research projects.

Looking to the future, the need for an ongoing consultation process with the Canadian space exploration community was identified by some AETD managers and CSA senior executives as a means of ensuring well-informed decisions for future program development.

4.2.1.2 Immediate Outcomes

Evaluation Question #9: Has Canada's HQP and infrastructure capacity for future space exploration been maintained or strengthened? (Oc1)

Evaluation Finding #9: The AETD program has contributed to maintaining Canada's HQP and strengthening its infrastructure for future space exploration.

HQP Capacity: Data provided by contract recipients show that the AETD program has contributed to maintaining Canada's space exploration HQP capacity. The majority of contract recipients (71%) reported that the program's beneficial impact on their organization's HQP capacity has been moderate to extreme, in that AETD contracts allowed them to attract, hire, and retain HQP, as well as to develop HQP expertise. However, based on data they provided regarding workforce adjustments both within their own organizations and within their subcontractors' organizations, the net average of HQP hired and let go across AETD contracts was close to zero.Footnote 3 The finding that the AETD program has contributed to maintaining, but not increasing, the number of Canadian HQP is perhaps not surprising given that Canada underwent an economic recession during the evaluation period.

Table 4: Range and average number of HQP and university students per AETD contract, for prime contracts and subcontracts.

Type of worker Mean (SD) Range
HQP
Prime Contracts 10.4 (26.9) 0 to 250
Subcontracts 4.7 (6.3) 0 to 35
University Students
Prime Contracts 1.2 (2.2) 0 to 12
Subcontracts 0.9 (3.2) 0 to 25

Table 4 shows that the number of HQP and university students who worked on AETD prime contracts and subcontracts varied widely across contracts. Note that these data represent total HQP and university students, regardless of the number of hours they spent working on prime AETD contracts.

Infrastructure Capacity: Evidence provided by both the CSA and contract recipients suggests that, over the course of the evaluation period, Canada's infrastructure capacity for future space exploration has been strengthened by the AETD program.

To facilitate the development, integration, and testing of space exploration solutions, the AETD program contributed $4M (76% of the total cost) on the construction of the following, CSA-owned infrastructure during the course of the evaluation period: Exploration Development and Operations Center, Rover Integration Facility, Planetary Analogue Terrain, and Portable Command and Control Shelter. In addition, an Exploration Storage Facility was built in which to securely store rover prototypes. Though data were not available regarding the percent of infrastructure utilization or the frequency with which this infrastructure was used by the various members of the AETD program's target population, AETD staff reported that, following its construction, this infrastructure was available to them when needed.

In addition to increasing the CSA's infrastructure capacity, the AETD program contributed toward strengthening the infrastructure capacity of at least some private enterprises and academic institutions. Approximately one quarter of contract recipients (29%) reported that the AETD program greatly benefited their organization's infrastructure capacity by allowing them to further develop their existing facilities and/or to build new facilities or laboratories for qualifying and testing technologies. Though the majority of contract recipients (65%) did not directly apply the money they received from AETD contracts to developing their organization's infrastructure, those that did allocated an average of 24.7% (SD = 29.1) of the total monetary value of their AETD contracts to infrastructure.

Evaluation Question #10: Has the Canadian space exploration scientific, technological, and operational capability increased? (Oc2)

Evaluation Finding #10: The AETD program has increased Canadian space exploration capabilities, primarily with respect to technological capabilities but also with respect to operational capabilities. However, the AETD program has not contributed as extensively to increasing Canada's scientific capability for space exploration.

Two main data sources were drawn upon to assess the extent to which the AETD program has contributed to increasing Canadian space exploration capabilities: information provided by key informants and archival data pertaining to IP generated by AETD contracts.

In terms of IP, archival data stored in the CSA's SAP database showed that IP was generated for 77% of AETD contracts during the course of the evaluation period. Among these contracts, a little over one third (38%) of the IP belonged to the prime contractor and the remaining IP (62%) belonged to the Crown. When asked about the program's effectiveness in enabling organizations to increase their IP, 50% of contract recipients stated that AETD contracts greatly increased their IP through the development of trade secrets, in-house expertise, know-how, and/or patents. Another 40% of contract recipients reported slight to moderate increases in their organization's IP as a result of having been awarded AETD contracts.

As shown in Figure 2, the majority of key informants perceived that the AETD program contributes greatly to increasing Canadian space exploration technological capabilities, though somewhat fewer expressed similar views concerning the program's contribution to operational capabilities and only about one quarter perceived that the program contributes greatly to increasing scientific capabilities.

In terms of the program's contribution to space exploration technological capabilities, the majority of key informants explained that the AETD program has successfully improved upon the performance of existing capabilities (e.g., vision systems) and carved out new niches for Canadian space exploration (e.g., rovers). Moreover, Canada's demonstrated technological competence was mentioned by both senior executives and AETD staff as an aspect of the program that they value the most. They explained that it is due to the AETD program's emphasis on building technological capability that Canada is now ready to provide international partners with critical elements for future space missions.

A sizable minority of key informants (between 25 and 45%) reported that developing technological capability is the primary focus of the AETD program and that less emphasis is placed on developing operational capabilities. However, the majority of key informants expressed the divergent view that the AETD program has notably increased Canada's operational capability for space exploration, particularly through analogue deployments of rover prototypes and medical equipment testing. In addition, several AETD contract recipients added that their operational capabilities have been greatly enhanced because AETD contracts allowed them to gain real operational experience.

Perhaps not surprisingly given the previously reported finding that only one quarter of AETD contract spending was allocated to scientific development, the predominant view among key informants was that building scientific capabilities is not an integral component of the AETD program. To illustrate this perception, when AETD managers were asked what aspects of the program they value least, one of the common responses was the lack of scientists assigned to work on the AETD program at the CSA. Moreover, the minimal emphasis on scientific development was most frequently cited by AETD managers as an aspect of the program that they value least. Similarly, lack of resources for scientific instrument development was commonly cited by AETD contract recipients as an aspect of the program that they value least. However, in the minority of cases where key informants perceived that the AETD program does contribute meaningfully to increasing scientific capabilities, it was explained that the program has focused on developing scientific capabilities in a few niche areas, rather than expanding scientific capabilities in new directions. Of note, the development of scientific instruments in collaboration with universities was mentioned by some AETD contract recipients as being their most valued aspect of the program because it fostered a better understanding of the science drivers of missions.

Figure 2: Key informant's opinions regarding the extent to which the AETD program has contributed to increases in Canadian space exploration technological, operational, and scientific capabilities.

When AETD managers were asked, more generally, to rate the overall quality of the space exploration solutions produced by AETD contracts, the vast majority (91%) indicated that they were either satisfied or very satisfied with the solutions produced. They explained that, though some solutions did not pan out (as is to be expected in R&D), many will be useful for future space exploration opportunities and a few may even be "game changers" (e.g., a performance readiness evaluation technology [PRET] used in space to assess astronauts' neurocognitive readiness to perform critical tasks). Due to a lack of baseline data and targets, findings and conclusions regarding the AETD program's success in increasing space exploration capabilities cannot be derived from available data concerning publically available publications and presentations that have stemmed from AETD contracts. However, the following information is provided in order to facilitate establishing baselines for ongoing program monitoring and subsequent program evaluations. During the five-year period covered by this evaluation:

  • CSA employees published at least 41 peer-reviewed publications and delivered at least 90 conference presentations related to the AETD program.Footnote 4
  • Approximately 159 presentations and 164 publically available publications related to AETD contracts were produced by the 27 contract recipients who provided relevant information. Forty-four percent of the publications were peer-reviewed.

Evaluation Question #11: Has Canada's position in the international space exploration scene been maintained or improved? (Oc3)

Evaluation Finding #11: Canada remains well-positioned in the international space exploration scene. However, decreases in the AETD program's budget over the past five years threaten the program's ability to maintain Canada's positive international space exploration reputation.

The strength of Canada's international reputation could not be assessed via interviews with representatives of international partners due to limitations in the resources available for conducting this evaluation. However, as shown in Figure 3, most AETD contract recipients, CSA senior executives, and AETD managers reported that the AETD program has been moderately to extremely instrumental in ensuring that Canada is well-positioned with the international space exploration context.

Key informants explained that, as a result of the program's strategic planning (including global coordination efforts), as well as the suite of solutions developed and analogue deployments, Canada is viewed as an active player in the international scene and remains sought after by international partners. Several key informants cited examples of Canadian organizations having been approached by international partners with opportunities to participate in future space exploration mission. To further illustrate the strength of Canada's reputation, one senior executive explained that he received considerable praise at international conferences for the AETD program's strategic planning approach and for the program's ability to "punch above its weight" with respect to program efficiency and economy.

Figure 3 : Key informant's opinions regarding the extent to which the AETD program has been instrumental in ensuring that Canada is well-positioned within the international space exploration context.

Despite the predominant view that the AETD program has demonstrated success in positioning Canada within the international context, some AETD managers cautioned that this determination cannot yet be made because a key element for positioning Canada on the international scene is flight heritage (i.e., past flight successes in space missions) and very few AETD-based solutions have been flown to date. Indeed some CSA senior executives expressed concern that interest in partnering with Canada on future space missions will wane if the AETD program's budget is not increased to allow for continued development of space exploration solutions. This concern was reiterated by AETD staff, who explained that, going forward, stable and predictable long-term program funding will be essential for participating in international opportunities.

Evaluation Question #12: Has the CSA's ability to make well-informed decisions for future missions and program development been maintained or improved? (Oc4)

Evaluation Finding #12: Through advice and guidance provided by the AETD program and through the CSEP, the program has contributed to the CSA's ability to make well-informed decisions. However, communication of AETD plans and priorities to AETD staff has been inconsistent, which can have an adverse impact on decisions regarding the execution of AETD program activities.

Though a few CSA senior executives and AETD managers reported that the space exploration roadmaps and plans produced by the AETD program are not fully integrated into decision-making processes, the majority (75% of CSA senior executives and 73% of AETD managers) said that the roadmaps and plans are either very or extremely useful. Indeed, when asked what aspects of the AETD program they value most, senior executives most frequently cited the emphasis placed on space exploration strategic planning. They explained that these plans and roadmaps allowed the CSA to make well-informed decisions regarding what to invest in so as to maximize budget utilization.

In addition, all senior executives agreed that the advice and guidance provided by the AETD program has been either very or extremely useful to the CSA's decision-making. As evidence, one senior executive pointed to the notable influence that this advice and guidance has had on the development of the new Space Policy Framework (which was completed at the time that this report was written; CSA, 2014). Incidentally, anticipation of this new framework's implementation was spontaneously expressed by several AETD managers and staff who said that, even though the CSEP is useful, there is still a need for clear federal priorities with which to align AETD program planning.

Though the AETD program's strategic planning was deemed useful by most senior executives and program managers, lack of communication regarding program plans and priorities was mentioned by several AETD staff as a hindrance to their ability to align the projects they work on with AETD program objectives. AETD staff explained that, as a consequence, errors and duplication of efforts with respect to setting deliverable requirements have occurred.

Further evidence of miscommunication between managers and staff regarding program plans and priorities surfaced on a number of occasions throughout the interviews and focus group discussions conducted for the purpose of this program evaluation. For example, some AETD staff perceived that lack of clarity surrounding how certain solutions would ultimately be used has resulted in inefficiencies in program spending. Conversely, AETD managers explained that there are often multiple applications (both space and terrestrial) for solutions under development and that taking into account the unpredictable nature of R&D geared toward innovation is part of the AETD planning process.

In order to better align the work carried out by AETD staff with the program's plans and priorities, some AETD managers and AETD staff suggested implementing a more systematic internal knowledge transfer mechanism.

Evaluation Question #13: Has public interest in space exploration increased? (Oc5)

Evaluation Finding #13: The extent to which the AETD program has increased public interest in space exploration could not be assessed due to lack of data.

Though some data were provided by the CSA's Communication Directorate for the purpose of this evaluation, there were not enough data upon which to draw any findings or conclusions regarding the extent to which the AETD program has increased public interest in space exploration.

4.2.1.3 Intermediate Outcomes

Evaluation Question #14: Has the economic viability of the Canadian space exploration sector been maintained or increased? (Oc6)

Evaluation Finding #14: Over the course of the evaluation period, the AETD program has contributed to ensuring the participation of private enterprises and academic institutions in the Canadian space exploration sector. The program also has a strong influence on organizations' likelihood of continuing to work in this sector. However, following the Stimulus Initiative, reductions in AETD program funding hindered the economic viability of the Canadian space exploration sector.

Due to a lack of baseline data and targets for quantitative data pertaining to most of the AETD program's impact on the economic viability of the Canadian space exploration sector, findings regarding the program's success in maintaining or increasing the economic viability of this sector are based primarily on opinions and experiences shared by key informants. However, quantitative data regarding total annual revenues for both the Canadian space exploration sector and AETD contract recipients lend support to the qualitative findings.

The majority of AETD contract recipients (71%) reported that the AETD program has had a strong influence on their organizations' likelihood of remaining in the space exploration sector. They explained that the program has exposed their organizations to new space exploration opportunities, venues of research, and business opportunities. For some private enterprises, AETD-based signature technologies and new applications have contributed to making space exploration a viable business area.

When asked more specifically about the likelihood that their organizations will be engaging in future space endeavours, almost three quarters (71%) of AETD contract recipients said that they are very or extremely likely to do so. The most frequent explanation offered was that space exploration is a fundamental part of their organization's core business or research. In addition, some AETD contract recipients said that they are already working on future space exploration opportunities. Those contract recipients who indicated that their organizations are less likely to engage in future space endeavours explained that there would first need to be specific opportunities to which their organizations could contribute.

Though not explicitly requested to speak to the topic of future economic viability, several AETD contract recipients emphasized, throughout their interviews, the adverse ramifications that reductions in the AETD program's funding have had on their organizations' financial stability. As indicated in Table 1 above (Section 2.3), though the Stimulus Initiative allowed for a notable increase in AETD program spending, the program's spending in 2012-2013 (not including Stimulus funds) was a little under 30% of what it had been in 2008-2009. AETD contract recipients explained that, following the Stimulus Initiative, they have been struggling to financially sustain their organizations, which in some cases rely heavily on the AETD program (e.g., for contracts, for furthering space exploration research and solutions, for opportunities to participate in space missions, etc.). Some contract recipients explained that, in an effort to compensate for the reduction in AETD program funding, they have been attempting to diversity their business beyond space exploration. Others said that their organizations have undergone significant cut-backs.

Interview and focus group data provided by CSA senior executives, AETD managers, and AETD staff lend further credence to the concerns voiced by contract recipients. They explained that, although the injection of Stimulus funding did stimulate the Canadian space exploration sector, there is currently not enough funding to continue developing some of the viable solutions funded by the Stimulus Initiative and to sustain the capabilities developed to date.

The estimated annual revenues generated by the Canadian space exploration sector and, more specifically, by AETD contract recipients also point to an economic downturn following the Stimulus Initiative. As shown in Table 5, these revenues increased by a notable amount in 2011 (perhaps due to the Stimulus Initiative) and then decreased sharply in 2012. Moreover, according to the State of the Canadian Space Sector report (CSA, 2012f), the percentage change in annual revenues between 2008 and 2012 was -9.4% for the space exploration sector (defined as robotics in the report) and -22% for the space sciences sector.

Of note, the revenue data presented in Table 5 should be interpreted with caution because (a) the Canadian space exploration sector's revenues were estimated by combining available data for the Canadian robotics and space science sectors, and (b) annual revenues for AETD contract recipients were not available for 12% of contract recipients because their organizations are not active in space. Also note that no precise data were available concerning the number of FTEs for either the Canadian space exploration sector or for AETD contract recipients.

Table 5 : Estimated annual revenues (in thousands of dollars) generated by the Canadian space exploration sector and by AETD contract recipients between 2008 and 2012.

Scope 2008 2009 2010 2011 2012 Total
Canadian space exploration sector $177,186 $175,083 $167,930 $254,806 $152,028 $927,034
AETD contract recipients $148,155 $149,973 $148,013 $185,577 $115,395 $747,113

Source: The CSA's PER Directorate (data collected for the Annual Canadian Space Sector Survey)

Due to a lack of baseline data and targets, no findings or conclusions pertaining to the general economic viability of the Canadian space exploration sector could be deduced from the remainder of the quantitative data that were collected for the purpose of this evaluation. However, the following information is provided in order to facilitate establishing baseline data for ongoing program monitoring and subsequent program evaluations.

AETD Subcontracts: Contract-specific data provided by AETD contract recipients shows that subcontracts were issued for approximately two-thirds of prime AETD contracts (65%) during the five-year period covered by this evaluation. When subcontracts were issued, the average number of subcontracted organizations per prime contract was 3.4 (SD = 2.6), with a range of one to 13 subcontracted organizations. The total dollar value of the work subcontracted varied greatly from approximately $4K to $5M per prime contract, with the majority of total subcontracts per prime contract (90%) costing less than $800K. Figure 4 shows the proportions of AETD subcontracts' total dollar value that were given to SMEs, large enterprises, and academic institutions.

Type of organization
Academic institutions SMEs Large enterprises
11% 55% 34%

Figure 4: Percentage of total AETD subcontract dollars by type of subcontractor organizations.

AETD Prime Contract Suppliers: According to AETD contract recipients, the total dollar value allocated to suppliers per prime AETD contract ranged widely from zero to $4.3M, though 43% of prime contracts did not entail any supplier contracts. On average, there were 11.3 (SD = 26.1) contracted suppliers per prime contract, about half of which were Canadian (55%) and half of which were foreign (45%).

AETD Follow-On Contracts: Over half (56%) of AETD contract recipients reported having been issued at least one follow-on contract as a consequence of prime AETD contracts received during the course of the evaluation period. The mean number of follow-on contracts issued by the CSA (not including AETD contracts) to prime contractors was 0.9 (SD = 1.5), with a range of zero to four. The mean number of follow-on contracts issued by other organization was 1.5 (SD = 2.9), with a range of zero to 10. Figure 5 shows the approximate total value of follow-on contracts issued by the CSA and by other organizations.

<$500K $500K to $1M $1M to $5M >$5M
From CSA 4 2 2 0
From other organizations 4 1 2 2

Figure 5: Value of follow-on contracts issued by the CSA (not including the AETD program) and by other organizations to AETD contract recipients as a consequence of prime AETD contracts.

Evaluation Question #15: Has Canadian leadership in science and signature technologies been maintained or increased? (Oc7)

Evaluation Finding #15: The AETD program has contributed toward increasing Canada's leadership in space exploration signature technologies. The program has also carved out a Canadian niche for a few scientific solutions, though the scope of Canada's contribution to space exploration science solutions is minimal.

Signature Technology Leadership: Data provided by AETD contract recipients indicates that the number of AETD-based signature technologies developed over the course of the evaluation period ranged from zero to 28 per organization. Most commonly (52% of cases), contract recipients reported having developed either one or two signature technologies, though another 32% did not develop any. In total, the 25 AETD contract recipients who provided relevant information reported having developed 61 signature technologies. However, in the absence of baseline data and targets, it was not possible to draw conclusive findings from these data.

Figure 6: Key informant's opinions regarding the extent to which Canada's leadership in signature technologies has been increased as a result of the AETD program.

When asked for their opinions about the extent to which Canada's leadership in signature technologies has been increased as a result of the AETD program, approximately three quarters of AETD contract recipients, CSA senior executives, and AETD managers said either "very" or "extremely" (Figure 6). In particular, they pointed to vision systems, spectrometers, drills, orbital servicing robotics, and rovers (the latter being a recent addition to the suite of Canada's signature technologies) as examples. However, some key informants cautioned that, until signature technologies have flown successfully on space missions, it is too soon to profess Canadian leadership in them.

Science Leadership: Perhaps not surprisingly given the common perception reported above that building scientific capabilities is not an integral component of the AETD program, only about one third of AETD contract recipients, CSA senior executives, and AETD managers indicated that the AETD program has contributed considerably to increasing Canada's leadership in science solutions for space exploration (Figure 7). These key informants explained that, though only a few science solutions have been developed, they have successfully created a Canadian niche and that Canadian science teams are adept at proposing advanced science solutions. AETD contract recipients also cautioned that this leadership may be lost if sufficient financial support for implementing space exploration science solutions is not made available.

Figure 7 : Key informant's opinions regarding the extent to which Canada's leadership in science solutions for space exploration has been increased as a result of the AETD program.

Most key informants who expressed doubts about the AETD program's contribution to developing Canadian leadership in science solutions reiterated their view that these are not the main focus of the program. However, others said it is simply too soon to tell whether the scientific instruments developed via the AETD program will perform successfully on flight missions.

Evaluation Question #16: Has Canadian participation in international space exploration missions been maintained or increased? (Oc8)

Evaluation Finding #16: Though it is too soon to ascertain definitively whether the AETD program will have a meaningful impact on Canadian participation in international space exploration missions, preliminary evidence suggests that the program has successfully positioned Canada for participation in space exploration missions. For example, two AETD-based solutions have already been flown on multiple space missions, Canada has participated in numerous terrestrial deployments and technical collaborations, international partners have expressed interest in collaborating with Canada on up-coming space missions, and AETD contract recipients report several other upcoming international space exploration opportunities for their organizations. However, without increased funding to commit to international space missions, Canada risks losing opportunities for future space missions.

AETD contract recipients reported that five of their space exploration solutions have already been deployed on space exploration missions. Among these five solutions, two stemmed from the AETD program:

  1. Neptec Technologies' TriDAR, a guidance system for unmanned vehicle during rendezvous and docking operations, was flown on space shuttles on three separate missions.
  2. Xiphos Technologies' Q6, a miniature computer processor card for complex image or signal processing, was flown on four microsatellites and one gondola.Footnote 5

However, given that the domain of space exploration tends to necessitate several years of R&D before solutions are flight-ready and that the ExCore component of the AETD program was established only six years prior to the end of the evaluation period, it was too soon to assess whether, overall, AETD-based solutions have had a meaningful impact on Canada's participation in international space exploration missions. Therefore, the remainder of the findings presented in this section are intended to assess whether the AETD program has successfully positioned Canada for participation in future space exploration missions.

According to data provided by the AETD program, the CSA, along with its Canadian industry and academic partners, have participated in a number of collaborative efforts that could result (or have already resulted, in some cases) in space exploration missions. Specifically, over the course of the evaluation period, there were:

  • 14 joint deployments, three with national partners and 11 with international partners. Most of these deployments were terrestrial (i.e., analogue missions), though two pertained to the TriDAR space missions mentioned above;
  • Four coordinated technology developments with international partners; and
  • Two technical-level collaborations established through memoranda of understanding (MOU). One of these MOUs was with an international partner and the other was with a federal government department.

AETD contract recipients were asked to identify solutions that their organizations have developed in preparation for specific space missions. However, based on the information provided, it appeared that some may have interpreted the question as asking whether they had developed solutions intended more generally for space exploration and it was not possible to tease apart their online questionnaire responses further. Though these findings should be interpreted with caution, half of AETD contract recipients (50%) reported that, since 2008, they have developed at least one solution in preparation for a specific space exploration mission and that, among these solutions, 74% were funded by AETD.

More reliable information was provided concerning upcoming international space exploration opportunities, such as international invitations, requests to participate, requests for collaborations, etc. The majority of AETD contract recipients (61%) indicated that their organizations have up-coming international opportunities. Among those for whom opportunities have been presented, the average number of opportunities identified per organization was 1.9 (SD = 1.2), for a total of 27 opportunities among 14 AETD contract recipients. Some notable examples of upcoming opportunities for participation in space missions include AETD-based navigation cameras (developed by Neptec Technologies) and rover components (developed by MacDonald Dettwiler & Associates) currently being developed for use on a mission to Mars in 2018. Another example is the integration of Q6 computer processor cards (developed by Xiphos Technologies) on three microsatellites being launched in 2014, as well as on the ISS in 2015.

In the absence of baseline data and targets with which to evaluate the information presented above, AETD managers and CSA senior executives were asked to predict the effect of the AETD program on Canada's future participation in international space missions. In response, all interviewees indicated that the suite of solutions that stem from the program has better positioned Canada for future missions. They explained that AETD-based solutions have increased Canada's credibility, technological capability, and expertise in the eyes of international partners. As evidence, they pointed to the fact that Canada has already been approached by international partners to explore the possibility of including a couple of the AETD-based solutions in up-coming space missions.

However, most AETD managers and CSA senior executives also said that the likelihood of future participation hinges on whether or not the program budget is increased. Specifically, they cautioned that, without funding to demonstrate solutions in space or commit to international space missions, Canada risks losing its opportunity to apply the solutions it has developed. As one interviewee explained, "The suite of solutions developed is providing a counter-weight to our decreased ability to commit to our international partners."

Evaluation Question #17: Have space technology transfers to new applications been maintained or increased? (Oc9)

Evaluation Finding #17: According to AETD contract recipients, AETD-based solutions have been transferred to dozens of space and terrestrial applications.

Figure 8: Number of AETD-based solution transfers per AETD contract recipient.

Approximately two-thirds of AETD contract recipients (65%) reported that their organizations have transferred AETD-based solutions to new or improved applications since the onset of the evaluation period. Among these contract recipients, most (87%) transferred between one and three solutions to new or improved applications (Figure 8), for a total of 38 applications among 23 AETD contract recipients. Approximately half of these applications were space-related (46%) and the other half were not space-related (54%).

Though there seem to have been a considerable number of solutions transferred to new or improved applications, it was not possible to determine whether the extent to which solutions have been transferred to other applications has been maintained or increased due to a lack of baseline data and targets.

An exhaustive list of new or improved applications was not available. However, key informants highlighted some of the terrestrial applications that have already been put to use on Earth:

  • A fuel cell technology initially developed for planetary rovers has been adapted for use on aeroplanes and submarines;
  • A planetary rover has been developed into a new line of electric, all-terrain, recreational vehicles;
  • A medical interface initially developed for use in space has been adapted to perform ultrasounds in remote locations (e.g., for use by medical doctors with patients in northern Canada and for use by veterinarians with clients on farms and racetracks); and
  • Vision systems technologies initially developed for planetary rovers have been adapted for use as laser sensors for navigating mines and as a guidance system for landing helicopters;
  • Robotics developed for orbital servicing have been adapted for use as medical applications, including neurosurgery and breast cancer screening; and
  • Robotics developed for orbital servicing have been adapted for use as nuclear automation solutions that inspect, maintain, and remediate nuclear reactor equipment.

4.2.1.4 Ultimate Outcomes

Evaluation Question #18: Has space R&D and Canadian participation in space exploration missions led to socio-economic benefits? (Oc10)

Evaluation Finding #18: Approximately 55 organizations joined the Canadian space exploration sector over the course of the evaluation period, the vast majority of which have remained active. In addition, the AETD program has enabled Canadian organizations to expand their commercial portfolios in the domain of space exploration and to gain access to new markets, most notably in health, mining, and transportation. Furthermore, several AETD contract recipients generated revenue from the commercialization of applications transferred from AETD-based solutions, though it was not possible to evaluate whether the amount of revenue generated is aligned with AETD program targets.

Evidence of the extent to which the AETD program has contributed to creating socio-economic benefits was derived from multiple indicators, including the extent of commercialization of AETD-based solutions and access to new markets, as well as space exploration-specific indicators such as commercial portfolio expansion in space exploration and organizations joining the space exploration sector.

Commercialization and Access to New Markets: Only a little over half (57%) of AETD contract recipients who reported having transferred AETD-based solutions to new or improved applications provided further information about revenues generated from commercialization and about access to new markets associated with these applications. Therefore, the data that follow should be interpreted with caution as they do not fully represent all AETD contract recipients who completed the online questionnaire.

Figure 9: Approximate commercialization revenue generated from AETD-based solutions transferred to new or improved applications per AETD contract recipient, over the five-year evaluation period.

As shown in Figure 9, the revenues generated since 2008 from the commercialization of AETD-based solutions (among those who provided relevant information) ranged widely from no revenue to up to $3M in revenues. Not surprisingly, the vast majority (92%) of the organizations that reported having generated commercialization revenue were private enterprises.

Over the five years of the evaluation period, an approximate total of $6.5M in combined revenue was reported by 13 AETD contract recipients. However, in the absence of baseline data and targets, it was not possible to determine whether the revenues generated from the commercialization of AETD-based solutions have been maintained or increased over time.

As a result of having received an AETD contract(s), approximately three-quarters (77%) of those AETD contract recipients who provided relevant information indicated that their organizations have gained increased access to new markets. According to AETD contract recipients and AETD program managers, those new markets accessed most frequently include health, mining, and transportation, though defense, energy, and aerospace have also been tapped into, albeit less frequently.

Space Exploration-Specific Socio-Economic Benefits: Forty-six percent of AETD contract recipients whose private enterprises fall within the space exploration sector said that their organization's space exploration commercial portfolio expanded considerably as a result of having been awarded an AETD contract. Another 45% reported a slight to moderate commercial portfolio expansion. In most cases, the work done for the AETD program helped private enterprises develop new or existing product lines, secure new contracts, and/or better position them for participation in international space exploration opportunities. However, even the few AETD contract recipients who reported little or no expansion in their commercial portfolio explained that the work that their organizations did for the AETD program added value and credibility to their organizations, which is instrumental for attracting new clients.

To estimate the number of organizations that have joined the Canadian space exploration sector, data regarding organizations that received space exploration-related contracts, grants, or contributions for the first time from the CSA's Space Exploration program (including the AETD program and other SSPs) were examined. As shown in Table 6, a total of 55 organizations joined the space exploration sector over the five-year evaluation period, primarily between 2008-2009 and 2011-2012. The vast majority of these organizations (96%) were Canadian and most (76%) were Canadian private enterprises.

Table 6: Number and type of organizations that received CSA space exploration funding for the first time during the evaluation period.

Type of Organization 2008-2009 2009-2010 2010-2011 2011-2012 2012-2013 Total
Private enterprises 8 11 8 13 2 42
Universities and research centres 2 0 1 2 0 5
Not-for-profit organizations 1 0 0 0 1 2
Federal departments 1 1 0 1 1 4
Foreign companies 0 1 0 0 1 2
Total 12 13 9 15 5 55

Source: The CSA's PER Directorate (CSA expenditures data)

Due to a transition in the CSA's data collection and storage database, information regarding whether these organizations remained active in the space exploration sector was only available up to December 1, 2012 – four months prior to the end of the evaluation period. Assuming consistency over this four-month period, all but one of the 55 organizations that joined the space exploration sector remained active in this domain (98%).

Evaluation Question #19: Is Canada's participation in space exploration missions successful? (Oc11)

Evaluation Finding #19: To date, the solutions that have been used on space exploration missions by AETD contract recipients, including two AETD-based solutions, have performed successfully in accordance with mission objectives.

According to AETD contract recipients, all five space exploration solutions that had already been deployed on space exploration missions performed according to the space missions' established requirements. AETD managers confirmed the successful space mission performance of the two AETD-based solutions, TriDAR and the Q6.

Evaluation Question #20: Has Canada maintained or increased its influence in international space exploration decision-making? (Oc12)

Evaluation Finding #20: The Canadian space exploration community, including the CSA, private enterprises, and academia, participate in numerous international space exploration advisory, consultation, or working groups. These participations demonstrate a considerable Canadian presence among international decision-making bodies, though the extent of Canada's influence on space exploration decision-making cannot be evaluated based on the available data.

At the end of the evaluation period, the CSA was participating in 68 international space exploration advisory, consultation, working groups, or sub-groups, most of which pertain to the ISS. The nature of the CSA's participation was as a member of the space exploration group in all but two cases (for which the CSA was an elected vice-chair and an observer).

Figure 10: Number of international space exploration advisory, consultation, or working groups per AETD contract recipient.

Figure 10 shows that half (50%) of AETD contract recipients' organizations did not participate in international space exploration advisory, consultation, or working groups. However, those that did most commonly participated in one or two groups. In total, participation on 29 international space exploration groups was reported by 26 AETD contract recipients. Among these international group participations, AETD contract recipients most frequently served as members of international groups (83% of seats), though a small minority also served the roles of chair (1%), observer (1%), and member of the board of directors (<1%).

Based on the information provided by the CSA and AETD contract recipients reported above, the Canadian space exploration community is represented on international space exploration advisory, consultation, or working groups. However, the coverage of this representation and the extent to which it impacts international space exploration decision-making cannot be deduced from the available data.

4.2.2 Demonstration of Efficiency and Economy

Evaluation Question #21: Efficiency: To what extent is the program delivering outputs and achieving outcomes in the most efficient manner? (Ee1)

Evaluation Finding #21: The AETD program is efficient in its resource utilization and in the value produced by the program with respect to use of public funds. In terms of overall program quality compared with other space agencies' programs, most CSA interviewees rate the AETD program as superior, given the favorable results achieved with a relatively small budget. However, several AETD contract recipients perceive that, regardless of efficiency, the program's budget is too small to achieve the results required to sustain the Canadian space exploration community. Divergent opinions were expressed regarding whether AETD program spending was allocated efficiently to produce an appropriate breadth of space exploration solutions.

Key informants' opinions and experiences regarding (a) overall efficiency in resource utilization, (b) value with respect to use of public funds, and (c) overall program quality compared with other space agencies' programs were drawn upon to evaluate the AETD program's efficiency. In addition, the appropriateness of the breath of solutions developed by the AETD program was examined because this theme emerged frequently in the initial phases of key informant data collection. In an effort to compare the efficiency of AETD program's costs per output and outcome with other similar programs, several key informants were asked to identify other similar programs and an Internet search for similar programs was also conducted. However, though a few programs run by other space agencies were initially identified, further investigation showed that these programs differ too much from the AETD program in terms of both scope and resources to allow for a meaningful comparison.

Overall Efficiency in Resource Utilization: According to the majority of CSA senior executives (75%) and AETD managers (73%), the AETD program has been delivered either very or extremely efficiently with respect to resource utilization. In particular, they pointed to the efficient production of outputs, especially in light of the program's relatively small budget compared with that of other space agencies. Furthermore, though some CSA interviewees identified alternatives that had been explored for producing program outputs, none of these alternatives would have done so at a lower cost.

Despite general agreement that the AETD program's resources are used efficiently, several suggestions were offered by CSA senior executives and AETD managers, as well as by AETD staff, for increasing program efficiency. Some of these suggestions include:

  • In order to reduce the overhead associated with contract management, introduce a "turnkey" contracting system whereby there is less oversight required after the initial specifications for deliverables have been established;
  • In order to increase the return on investments, support a third party (e.g., a not-for-profit organization) in launching space exploration solution development competitions for university students;
  • In order to avoid duplication of efforts by applying lessons learned, increase AETD staff's access to information about past projects funded by the CSA (via AETD or other CSA programs);
  • In order to more efficiently and effectively support the development of science solutions for space exploration, add a scientist(s) to the team of AETD staff.

Figure 11: Key informant's opinion regarding the value produced by the program with respect to use of public funds.

Value with Respect to Use of Public Funds: As shown in Figure 11, the majority of AETD contract recipients, CSA senior executives, and AETD managers indicated that, overall, the AETD program has produced good value with respect to the use of Canadian public funds. Most interviewees explained that AETD program funding has successfully contributed to advancing space exploration capabilities and HQP capacity, as well as to developing innovative space exploration solutions and planning tools. In addition, some AETD contract recipients spoke of the program's success in positioning their organizations for future space mission opportunities and, in the case of private enterprises, for follow-on contracts and revenue generated by the commercialization of solution transfers to other applications. Those interviewees who reported only slight or moderate value with respect to use of public funds explained that, specifically in relation to the Stimulus Initiative, the breadth of solutions developed was not appropriate. According to some of these interviewees, more solutions could have been developed using the same amount of funds and, according to others, fewer solutions of higher quality could have been developed.

Overall Program Quality Compared with Other Space Agencies' Programs: As shown in Figure 12, key informant groups differed from one another in their ratings of the overall quality of the AETD program compared with other space agencies' advanced space exploration technology development programs.

Most CSA senior executives and AETD managers, but under one-third of AETD contract recipients, rated the AETD program as better than others. Those who rated the program as superior indicated that the program's relatively small budget was invested wisely to produce solutions that successfully contribute to achieving program outcomes. More specifically, some CSA senior executives and AETD managers pointed to the anticipatory planning approach, as well as to the integration of scientific, technological, and operational components, as unique advantages of the AETD program that allow for efficiency in achieving expected results.

Figure 12 : Key informant's opinion regarding the AETD program's overall quality compared with other space agencies' programs.

In contrast, the most common explanation offered by those who rated the AETD program as inferior (primarily AETD contract recipients) was that space exploration in Canada is under-prioritized and under-funded, thereby leaving Canada behind other agencies in terms of the breadth of solution developed and the opportunities to participate in space exploration missions.

Breadth of Solutions Developed: The question of whether the breadth of solutions developed via the AETD program has been appropriate was not initially included in the survey instruments developed for this evaluation. However, several AETD contract recipients spontaneously mentioned that the solutions developed are too narrowly focused on robotics and rovers, and that more emphasis should be placed on other space exploration solutions, such as scientific instruments or a more diversified range of technologies. Conversely, several AETD staff spontaneously mentioned that, given budgetary constraints, too wide a net has been cast with respect to solution development and that, consequently, the focus on developing high quality signature technologies has been diluted. However, it was not possible to gauge the extent to which either of these views was endorsed across the sample of AETD contract recipients or AETD staff.

AETD managers and CSA senior executives were specifically asked to rate the appropriateness of the breadth of solutions developed via the AETD program. In response, the majority (82% and 75%, respectively) said that the breadth of the types of solutions funded by the AETD program has been appropriate because the solutions address an international need for specific types of robotics and they build upon existing Canadian capabilities. For example, speaking specifically of the Stimulus Initiative, one interviewee explained that rovers will be needed for most future space exploration missions, so the decision to build Canadian capabilities in this domain was well-founded. However, even among AETD managers and CSA senior executives, divergent perspectives were expressed. Some said that the focus on robotics is too narrow and others said that, retrospectively, the breadth of funded solutions may have been too broad because it created an expectation that Canada will continue to develop these solutions and this may not be possible due to reductions in the AETD program's budget.

Evaluation Question #22: Economy: To what extent has resource use been minimized in the implementation and delivery of the program? (Ee2)

Evaluation Finding #22: Overall, AETD funds have been spent in an economically sound manner, though qualitative data suggest that there may be redundancies with the Enabling Technology Development Program that have not yet been addressed. In terms of the appropriateness of the program's inputs, the predominant view among key informants is that there are too few resources dedicated to the AETD program for supporting the achievement of expected results, though the resources allocated to the Stimulus Initiative were sufficient.

Three facets of economy were examined for the purposes of this evaluation: (a) economy in program spending, (b) the appropriateness of program inputs for supporting expected results, and (c) redundancy or overlap with other programs.

Economy in Program Spending: Based on the financial data presented in Table 1 (Section 2.3), the AETD program's spending came to within 4% of the forecasted budget across the five years of the evaluation period.

Total Spending
Salary O&M Capital
12% 84% 4%

Figure 13 : Percent of AETD program spending allocated to salaries, O&M, and capital costs over the 5 years of the evaluation period.

Financial data regarding spending allocated specifically to each of the seven activities or to each of the four outputs identified in the AETD program logic model was not available. However:

  1. As shows in Figure 13, O&M constituted the majority (84%) of total program spending.
  2. The $153M spent on these 145 contracts during the evaluation period represents 75% of total program spending, including salaries, O&M, and capital costs, and 79% of salaries and O&M costs; and
  3. Across the evaluation period, approximately $0.16 was spent on salaries for every dollar spent on the 145 contracts, though it is worth noting that AETD employee's salaries also contribute to producing outputs other than contracts. Moreover, when tracked year-by-year, the amount spent on salaries for every dollar spent on the 145 contracts decreased considerably in those years when the amount spent on contracts was higher (Table 7).

Thus, while relevant data limitations cannot be ignored, the available data pertaining to overall program spending suggest that AETD funds have been spent in an economically sound manner.

Table 7: Dollar amount spent on salaries for every $1 spent on the 145 contracts across the evaluation period.

  2008-2009 2009-2010 2010-2011 2011-2012 2012-2013 Total
Amount spent on 145 contracts $6,624K $21,686K $51,950K $64,026K $8,944K $153,128K
Amount spent on salaries per $1 spent on contracts $0.73 $0.22 $0.09 $0.09 $0.47 $0.16

Figure 14: Key informant's opinion regarding the appropriateness of program inputs for achieving program delivery.

Appropriateness of Program Inputs for Supporting Expected Results: The findings presented thus far in this report make frequent mention of the fact that the AETD's program spending diminished considerably over the course of the evaluation period, resulting in some adverse consequences following completion of the Stimulus Initiative. Though AETD contract recipients were not specifically asked to rate the appropriateness of the AETD program's inputs, they frequently voiced concern that the program is under-funded. This concern was reflected in the responses of 40 to 75% of CSA senior executives, AETD managers, and AETD staff when they were asked about the appropriateness of program inputs (Figure 14). They explained that, following the Stimulus Initiative, there were too few resources invested in the program to continue supporting the achievement of expected results. Most frequently, they pointed to the inadequacy of the program's current budget, though a shortage of AETD staff was also mentioned by individuals across all three groups of CSA key informants. In line with this common view, unsustainable program funding was mentioned most frequently by CSA senior executives when they were asked which aspect of the AETD program they value least.

Conversely, about half of AETD managers and staff reported that the program's inputs are appropriate for achieving program delivery and two interviewees reported that the inputs are too high. A detailed analysis of the explanations provided for these latter ratings showed that interviewees based their responses on the appropriateness of the inputs specifically for the Stimulus Initiative, rather than for the AETD program as a whole.

Redundancy or Overlap with Other Programs: When asked whether the needs addressed by the AETD program overlap with the needs addressed by other Canadian programs, some key informants identified two CSA programs with which there are potential redundancy in resource utilization:

  • the Enabling Technology Development program (ETD; SSP 1.3.2.2); and
  • the Operational Space Medicine program (OSM; SSP 1.2.3.2).

According to the CSA's PAA (2012-2013), the ETD program falls under the CSA's Future Canadian Space Capacity Program and is designed to produce technology development and demonstration activities that contribute to maintaining or developing a technological edge in promising fields. With respect to the possibility of an overlap with the CSA's ETD program, the majority (75 to 85%) of CSA senior executives, AETD managers, and AETD staff did not perceive redundancy between the ETD and the AETD programs, and instead said that the two programs complement each other well. They explained that, unlike the ETD program's focus on component-level technology, the AETD program develops integrated technologies, including scientific and operational elements, geared toward future space exploration mission opportunities. Though not explicitly asked about a potential overlap between the two programs, one AETD contract recipient reflected the views of the majority of CSA interviewees by saying, "It is very important to have an exploration-focused technology program that includes component-level technology development, system-level prototyping, system-level testing, field testing and operations, and flight relevant concept studies."

Conversely, approximately 15 to 25% of CSA senior executives, AETD managers, and AETD staff said that there is ambiguity surrounding whether it is the ETD or the AETD program that is responsible for carrying out technology development for space exploration. In addition, one interviewee noted that the in-house expertise among AETD staff that was developed for the Stimulus Initiative overlaps with the Future Canadian Space Capacity's mandate of building Canadian HQP capacity. To resolve this ambiguity, some interviewees suggested either merging the two programs or creating a clearer distinction between them.

The OSM program falls under the CSA's Space Exploration Program and is designed to deliver operational and clinical health care activities for Canadian astronauts in order to promote and ensure their physical, mental, and social well-being and safety. However, the development of exploration-enabling medical technologies that feed into the achievement of the OSM program's objectives is carried out by the AETD program. As a result of this structure, a few AETD managers and staff explained that there are redundancies in the approval and reporting process for carrying out medical technology development activities and that these redundancies create duplication in the use of human resources. The interviewees suggested that, though it is necessary to work with the AETD program to develop solutions that mitigate space flight health risks, the responsibility for developing exploration-enabling medical technologies should fall under the purview of the OSM program. When asked for feedback on this suggestion, most AETD managers and some CSA senior executives indicated that they did not have an informed opinion to offer. Among those that did, the predominant view was that the development of exploration-enabling medical technologies should remain within the AETD program because they benefit from the integration of technological, scientific, and operational elements afforded by the program. However, these interviewees also suggested that activities related to approval and reporting processes should be more clearly defined to reduce duplication of efforts.

Following the period of time covered by this evaluation, the AETD program introduced an Exploration Core Governance Structure, in which the new governance structure for activities pertaining to the ExCore component of the AETD program were established. Follow-up discussions with AETD managers and staff indicated that this new structure will likely address the redundancies in carrying out medical technology development activities. However, AETD staff who work on medical technology development suggested that the OSM and AETD programs collaboratively monitor the implementation of the new governance structure in order to identify and address remaining redundancies, should any still exist.

4.3 Stimulus Initiative Case Study

As described in Section 2.1 of this report, an envelope of $110M was allocated by the Government of Canada's Budget 2009 to develop terrestrial prototypes for space robotic vehicles and to further develop robotics and other technologies. This budget was allocated to two projects: (a) $60M for ESM, including terrestrial prototypes of rovers and advanced technologies, and (b) $50M for NGC, including terrestrial prototypes of the next generation of Canadarm.

In this section of the report, a case study of the Stimulus Initiative is presented in order to assess its performance. As with the performance evaluation of the overall AETD program, Stimulus Initiative performance was assessed with regard to (a) progress toward producing outputs and achieving expected outcomes specifically related to the Stimulus Initiative, as identified in the program logic model in Figure 1, and (b) the initiative's resource utilization in relation to the production of outputs and progress toward expected outcomes.

The methods employed in conducting this case study entailed document reviews and the administration of key informant survey instruments. With respect to the latter, Stimulus-relevant information was extracted from the interview, focus group, and online questionnaire responses provided by AETD contract recipients (including Stimulus contract recipients), CSA senior executives, AETD managers, and AETD staff (as per the survey instrument guides in Appendices B through G). In most cases, the information provided was in response to open-ended questions and, as such, quantitative analyses of frequency counts and percentages were not conducted.

4.3.1 Achievement of Expected Outputs and Outcomes

Evaluation Question #23: Have Stimulus Initiative outputs been produced?

Evaluation Finding #23: All planned Stimulus contracts were awarded. In several cases, the contracting process was successfully expedited via pre-existing contract amendments. Due to external factors beyond the AETD program's control, there were major delays in awarding some ESM contracts. The production of several ESM solutions was also hindered by underestimation of contract values and by lack of clarity among AETD managers surrounding the primary objectives of ESM contracts at the onset of the initiative. Despite these challenges, delays were managed within the funding envelope and schedule. In addition, over 30 ESM space exploration technological solutions were developed and the NGC project produced five main prototypes.

Contracts Awarded for Advanced Exploration Technology Development (Op1): All contracts under the Stimulus Initiative that were announced as part of Canada's Economic Plan have been awarded (CSA, 2011). These Stimulus contracts included 33 pertaining to ESM and one pertaining to NGC.

According to the AETD program's Stimulus Initiative's Closure Report (CSA, 2013d), Stimulus Initiative investments were designed to be in line with the CSA's past and current effort to prepare Canada for next opportunities in space exploration and, therefore, situating the initiative under the AETD program allowed for quick start-up of its implementation process. Archival contracting data from the CSA's SAP database affirms that, in approximately 40% of cases, contracts that existed prior to the Stimulus Initiative were built upon to continue solution development and expedite the contracting process. For the remainder of Stimulus contracts, the average length of time between approval of the Stimulus Initiative's TB submission (June 18 2009) and the date upon which a requisition was submitted to PWGSC was 162 days (SD = 187.9), with a maximum period of 820 calendar days.

The length of time between when a requisition was sent to PWGSC and when the contract was awarded did not differ significantly between Stimulus and non-Stimulus contracts. However, in February 2010, an external factor beyond the AETD program's control impacted the ESM contracting process (CSA, 2013d). Specifically, a shift in direction concerning one of Canada's international partners' space exploration plans necessitated realignment of the statement of work and requirements for one of the main ESM rovers under development. Though this realignment led to major delays in awarding some ESM contracts, these delays were managed within the funding envelope and schedule. However, according to some AETD managers and staff, the realignment created an overlap with another ESM rover.

Though not specifically asked to comment on the contract awarding process, several AETD managers and staff reported that awarding many contracts and spending the contracting dollars within the short timeframe afforded by the Stimulus Initiative was challenging, in that the tight deadlines did not allow sufficient time to plan properly. Perhaps as a consequence, it became apparent over time that the ESM budget had been underestimated for many contracts. As one AETD program manager explained, because some of the contracts were for the development of solutions with which the CSA had little prior experience (e.g., rovers), there was a lack of both in-house and contractor expertise in estimating the appropriate costs of contracts. To mitigate risks, requirement de-scoping options were built into several Stimulus contracts.

Perhaps due to the tight deadlines that did not allow for an extensive planning process, several AETD staff and a few AETD managers reported that the primary objectives and related directives for Stimulus contracts were not clear, particularly with respect to ESM contracts. This lack of focus stemmed from senior managers' divergent approaches with regard to two topics. The first topic pertained to whether, in line with the Government of Canada's Economic Action Plan, the primary objective was job creation or whether, in line with the AETD program's objectives, it was a path to flight for future missions. The second topic, a sub-set of the first, pertained to whether ESM prototype development should satisfy requirements for simulation missions or whether it should align with the more stringent procedures for flight missions. Depending on the primary objectives, requests for proposals would need to be adjusted in terms of how much flexibility should be build into design and deliverable requirements.

Over time, the resolution to these divergent approaches was that, in some cases, stringent requirements associated with flight missions were included in contracts, thereby necessitating that contract recipients fulfill additional procedural requirements. According to several AETD staff, the time and money invested in completing procedural requirements was excessive and detracted from a needed focus on fulfilling technical requirements. In other cases, fewer contractual requirements were stipulated, which expedited the onset of the work, but which subsequently resulted in technical challenges that hindered the functionality of the prototypes (e.g., unclear interfacing between prototypes).

Solutions Produced for Space Exploration (Op2): According to the AETD program's archival data (CSA, 2013d), more than 30 ESM prototype technologies were produced as a result of Stimulus contracts. These prototypes included three new and one upgraded rovers, as well as numerous preparatory prototyping and sub-system/payload solutions. In addition, the NGC project produced the Next Generation Small Canadarm, the Next Generation Large Canadarm, The Proximity Operations Station Testbed, the Semi-Autonomous Docking Station, and the Missions Operation Station.

Evaluation Question #24: To what extent did the Stimulus Initiative achieve its immediate outcomes?

Evaluation Finding #24: The Stimulus Initiative enabled the Canadian space exploration sector to maintain its FTEs and HQP, as well as to employ more university students, during a period of economic hardships. In addition, through the development of ESM and NGC solutions, the Stimulus Initiative allowed for accelerated development of space exploration technologies and increased the visibility of Canada's space exploration capabilities on the international scene.

Maintain or Strengthen Canada's HQP Capacity for Future Space Exploration (Oc1): Several CSA senior executives, AETD managers, and AETD staff spoke of the beneficial impact that the Stimulus Initiative had on maintaining Canada's space exploration HQP during a period of economic hardship. According to most Stimulus contract recipients who participated in the interviews conducted for this evaluation (75%), it would have been very to extremely likely that their organization would have undergone workforce downsizing if they had not been awarded Stimulus contracts. Stimulus contract recipients who said that their organization would probably not have undergone downsizing explained that, with the help of the Stimulus Initiative, they were able to increase the size of their workforce.

Figure 15: Number of prime and subcontracted FTEs per Stimulus contract.

Data provided by Stimulus contract recipients showed that the number of prime and subcontracted FTEs who worked on a Stimulus contract was between one and 10 for the majority of Stimulus contracts (Figure 15). In total, approximately 354 FTEs (215 prime and 139 subcontracted FTEs) worked on the 28 Stimulus contracts for which data was available.

In terms of HQP, the net average of HQP hired at contract onset and let go at contract end across both Stimulus prime contracts and subcontracts was close to zero (as it was for non-Stimulus contracts). Also, there were no significant differences between Stimulus contracts and non-Stimulus contracts in the number of HQPs who worked on either prime contracts or subcontracts. However, significantly more university students worked on Stimulus than on non-Stimulus prime contracts, with an average of two (SD = 2.1) students per Stimulus contract and an average of 0.7 (SD = 2.1) students per non-Stimulus contract (t(84) = -2.38, p = .02).

Increase Canadian Space Exploration Technological and Operational Capabilities (Oc2): Several CSA senior executives, AETD managers, and AETD staff made a point of emphasizing that accelerated space exploration technological development was a particularly advantageous aspect of the Stimulus Initiative.

Though TRLs were not available for ESM solutions, NGC-related archival data provided by the AETD program showed that 13 of the 16 target TRLs for NGC were met. Incomplete integration of components was primarily responsible for those target TRLs which were not met, though in one case a proof of concept was developed but not fully demonstrated. In addition, target technical performance measures were met and new functionalities were developed.

Though they were not specifically asked to rate the quality of the Stimulus solutions delivered, several AETD staff noted that ESM solution quality was high in some cases and low in others. Indeed, low product quality resulting from Stimulus contracts was most frequently identified by AETD staff as an unfavorable aspect of the AETD program. However, some AETD staff also said that the Stimulus Initiative provided an opportunity for the CSA to experiment with new techniques, approaches, and technologies, as well as to benefit from lessons learned, in a lower-risk environment where high performance was not critical to the success of a space mission. This sentiment was shared by some AETD managers, who reported that the overall quality of the solutions was good, especially given that some of them were new additions to the suite of Canada's signature technologies.

In the absence of target and actual TRLs for ESM solutions, a quantitative assessment of the quality of the solutions delivered was not within the scope of this evaluation.

Maintain or Improve Canada's Position in the International Space Exploration Scene (Oc3): Though key informants were not specifically asked to speak about the Stimulus Initiative's impact on Canada's position within the international space exploration scene, several CSA senior executives, AETD managers, and AETD staff nonetheless mentioned that the initiative has created increased visibility for Canada's space exploration capabilities among its international partners. In particular, they pointed to the international attention that some of the ESM rovers and other technological solutions have attracted.

Evaluation Question #25: To what extent did the Stimulus Initiative achieve its intermediate outcomes?

Evaluation Finding #25: The Stimulus Initiative contributed to the economic viability of the Canadian space exploration sector by increasing the amount of money spent per prime contract on both subcontractors and suppliers, relative to the amount spent for non-Stimulus prime contracts. In terms of the impact on Canada's participation in international space missions, two ESM solutions have been flown on space missions and opportunities for future missions have been presented by international partners, though it is too soon to know whether any of these opportunities will materialize.

Maintain or Increase the Economic Viability of the Canadian Space Exploration Sector (Oc6):

As shown in Table 8, the total dollar value spent on both subcontracts and suppliers was significantly higher for Stimulus prime contracts than for non-Stimulus prime contracts (t(28.40) = -2.83, p = .009 and t(29.26) = -2.25, p = .032, respectively).

Table 8: Average amount spent on subcontracts and suppliers per AETD prime contract, for Stimulus and non-Stimulus contracts.

Type of worker Mean (SD)
Subcontracts
Stimulus Prime Contracts $809K ($1,261K)
Non-Stimulus Prime Contracts $127K ($207K)
Suppliers
Stimulus Prime Contracts $358K ($814K)
Non-Stimulus Prime Contracts $23K ($64K)

Maintain or Increase Canadian Leadership in Signature Technologies (Oc7): Apart from the finding reported above that the Stimulus Initiative allowed for accelerated development of space exploration technology, no further evidence was available to assess the extent to which Canadian leadership in signature technologies has been advanced by the Stimulus Initiative.

Maintain or Improve Canadian Participation in International Space Exploration Missions (Oc8): According to AETD program archival data, rovers and associated payloads, such as drills, fuel cells, active vision systems, sensors, computer processor cards, and advanced navigation algorithms, developed via the ESM project were included in three analogue deployments with international partners. In addition, one ESM solution (Xiphos Technologies' Q6) was flown on four microsatellites and one gondola. Furthermore, Canada was approached by international partners to explore the possibility of participation in five future space exploration missions (e.g., the navigation cameras and rover components currently being developed for a mission to Mars, a rover, drill, and avionics for a lunar in-situ resource utilization mission, and the upcoming missions in which Q6 cards will be integrated, as mentioned above). At the time that this evaluation report was written, none of these possibilities had yet come to fruition. However, though it is likely that not all five opportunities will materialize, lack of affirmation regarding Canada's participation in these missions is not surprising given that space missions tend to require a long planning process.

Evaluation Question #26: To what extent did the Stimulus Initiative achieve its ultimate outcome?

Evaluation Finding #26: Though the available evidence is primarily anecdotal in nature, it suggests that socio-economic benefits have resulted from the Stimulus Initiative. For example, according to key informants, new organizations joined the space exploration sector and some fruitful partnerships were forged as a result of the Stimulus Initiative. In addition, a few Stimulus-based solutions were commercialized following transfers to other applications, though the extent of the revenues generated could not be ascertained. More robust evidence indicates that Stimulus funds were used by organizations to increase their space exploration R&D expenditures.

Maintain or Increase Socio-economic Benefits of Space Exploration R&D (Oc10): The impact of the Stimulus Initiative on the socio-economic benefits of space exploration was assessed in relation to Stimulus contract recipients' R&D expenditures, organizations that joined the space exploration sector, partnerships forged among Stimulus contract recipients, and solutions transferred to other applications. However, only anecdotal evidence was available for these indicators, with the exception of R&D expenditures.

Among those Stimulus contract recipients who provided relevant information, two-thirds (67%) reported that their organization's space exploration R&D expenditures increased as a direct result of this funding. This percent increase ranged from 10 to 50%, with a mean increase in R&D expenditures of 26.5% (SD = 16.9).

Though the available data did not allow for pinpointing which organizations that joined the Canadian space exploration sector did so as a result of the Stimulus Initiative, several CSA senior executives and AETD staff reported that some organizations with no prior experience in the domain of space exploration were awarded Stimulus contracts. In addition, AETD contract recipients, CSA senior executives, and AETD staff explained that new partnerships were formed among organizations that had never worked together before. In many but not all of these cases, these partnerships resulted in good synergies between organizations, especially when new players and space organizations were paired.

No data were available concerning the revenues generated by technology transfers of Stimulus solutions to new or improved applications. However, based on information provided by the AETD program, at least four ESM solutions were commercialized following transfers to terrestrial or space applications, and the NGC project produced at least two commercialized applications.

4.3.2 Efficiency and Economy

Evaluation Question #27: To what extent were Stimulus Initiative resources used efficiently and economically?

Evaluation Finding #27: The available data did not allow for an evaluation of Stimulus Initiative efficiency. Though the value of several ESM contracts had been underestimated, financial data suggest that, overall, resources were used in an economically sound manner for the Stimulus Initiative's implementation and delivery.

Efficient Delivery of Outputs and Achievement of Outcomes (Ee1): At the onset of the Stimulus Initiative, the AETD program transferred funds to PWGSC to pay for the salaries of three PWGSC employees assigned to manage the Stimulus Initiative contracting process. However, like those opinions expressed by CSA employees concerning the AETD program as a whole, several CSA senior executives, AETD managers, and AETD staff noted inconsistencies in the interpretation of processes and procedures regarding the contracting process. Interviewees explained that these inconsistencies reduced the efficiency of the Stimulus Initiative contracting process, which was time-sensitive. Notably, challenges were experienced in expediting those changes in the statement of work and rover requirements that needed to be effected in order to realign deliverables with the shift in direction taken by a key international partner (as explained above). Thus, external factors impacting the delivery of the Stimulus Initiative contributed to the delays in the initiative's implementation.

As explained in a previous section of this report, divergent opinions were expressed by all four groups of key informants regarding whether AETD program spending and, more specifically, Stimulus Initiative spending were allocated efficiently in order to produce an appropriate breadth of space exploration solutions. Similarly, opinions varied with respect to the quality of the Stimulus solutions produced. Based on the available information, it was not possible to draw any conclusive findings regarding the efficiency of the Stimulus Initiative with respect to either the appropriateness of the breadth or the quality of the solutions produced.

Minimization of Resource Use in Program Implementation and Delivery (Ee2): As reported previously in this report, several AETD managers and staff indicated that, generally, the financial resources invested in the Stimulus Initiative were appropriate. However, the dollar value of many ESM contracts had been underestimated due to a lack of both in-house and contractor experience in estimating appropriate contract costs, particularly for new types of solutions.

As shown in Table 9, a total of $133M was spent of the Stimulus Initiative between 2009-2010 and 2012-2013. Of this total amount, $110M was allocated by the Government of Canada's Budget 2009, all of which was spent on Stimulus contracts. The remaining $23M was allocated by the AETD program for project and risk management, as well as to complete some of the contracts.

Broken down by the two Stimulus Initiative projects:

  • A total of $79M was spent on the ESM project, with authorities to spend up to $85M; and
  • A total of $55M was spent on the NGC project, with authorities to spend up to $59M.

Thus, actual Stimulus Initiative spending fell within the initiative's budget.

In total, the $122M in contracts awarded to Stimulus contract recipients constituted the vast majority (92%) of the total dollar amount spent on the Stimulus Initiative. Combined, approximately $0.07 was spent on salaries and project management costs for every dollar spent on Stimulus contracts.

Table 9: Financial resources allocated to the Stimulus Initiative between 2009-2010 and 2012-2013.

Type of Resource ESM NGC Total
Salaries $6.4M $1.4M $7.8M
Project Management Costs $2.6M $0.6M $3.2M
AETD-funded contracts $9.3M $3.1M $12.4M
Stimulus-funded contracts (as per the Government of Canada's Budget 2009) $60M $50M $110M
Total $78.3M $55.1M $133.4M

Source: The CSA's Finance Directorate

5. Conclusions and Recommendations

Conclusions drawn from the relevance and performance findings reported above are presented in this section, as are evidence-based recommendations for program improvement. These recommendations appear in bold font in the text below and are summarized at the end of this section.

With respect to relevance, the AETD program's objectives align with the Government of Canada's priorities in S&T and in space exploration. Canada's continued involvement in space exploration builds on a strong reputation and heritage in space. In recent years, the Canadian government has recognized not only the importance of space exploration, but also the beneficial impact it has had on the country's knowledge-based economy in terms of long-term prosperity, innovation, and competitiveness. The AETD program also falls within the federal government's jurisdiction, as per the legal mandate accorded to the CSA through the Canadian Space Agency Act (1990), and the program aligns with the CSA's strategic outcomes and departmental priorities. In terms of continued need, the AETD program's objectives demonstrate relevance in addressing the need of Canadians for a federal program that leads the planning and coordination of Canada's space exploration activities and that supports the development of technological, operational, and scientific capabilities for future space exploration.

With respect to performance, the AETD program has successfully achieved most, but not all, of its expected results. In terms of the production of outputs, plans and a roadmap for Canadian space exploration initiatives have been outlined via the CSEP. The strategies laid out in this document, as well as other advice and guidance provided by the AETD program, have effectively supported decision-making at the CSA, according to the majority of senior executives. However, communication of AETD plans and priorities to AETD staff has been inconsistent, indicating that a systematic method of communicating plans and priorities from AETD managers to all AETD staff should be implemented.

In addition to having developed space exploration plans and a roadmap, the AETD program has successfully awarded contracts for advanced exploration technology and operation development to Canadian private enterprises and academic institutions. As a result, numerous technology solutions for space exploration have been produced. However, considerably fewer contracts geared toward scientific development have been awarded and, consequently, fewer scientific solutions for space exploration have been produced.

The scientific solutions produced by the AETD program have increased Canada's space exploration scientific capabilities in a few niche areas. However, the limited amount of financial and human resources invested by the AETD program in developing science solutions has not adequately positioned Canada's scientific community for strong leadership in the field of space exploration science. Therefore, the AETD program should conduct an analysis of the optimal level of resources that should be dedicated to scientific development by the AETD program and, based on the results of this analysis, clearly communicate to program stakeholders the AETD program objectives with regard to scientific development and allocate resources accordingly.

In contrast, the technological and (to a somewhat lesser extent) operational developments produced by the AETD program have led to notable successes in achieving program outcomes, particularly with respect to increases in both Canada's technological capabilities and its leadership in space exploration signature technologies. In addition, the AETD program's investment in technological and operational development has strengthened the Canadian space exploration communities' infrastructure capacity and has allowed the Canadian space sector to maintain its HQP during an economic downturn in the Canadian economy. Furthermore, the technological solutions developed as a result of AETD contracts have been transferred to dozens of space and terrestrial applications, some of which have been commercialized – thereby enabling AETD contract recipients to expand their commercial portfolios. Though it was not possible to determine whether the extent of the generated revenues is aligned with AETD program targets, other socio-economic benefits that resulted from the production of space exploration solutions include increased access to new markets for AETD contract recipients and new organizations joining the space exploration sector.

Evidence suggests that AETD-based solutions have also successfully positioned Canada for participation in space exploration missions. For example, Canada has participated in several terrestrial deployments and technical collaborations, AETD contract recipients report many upcoming international space exploration opportunities, and two AETD-based solutions have successfully been flown on space missions. However, given that the AETD program has only existed since 2007-2008, it is too soon to assess definitively the extent to which AETD solutions will have a meaningful impact on Canadian participation in future space exploration missions. Furthermore, Canada's participation in future missions hinges upon the long-term stability of the AETD program's budget. Though Canada currently maintains a favourable reputation in the international space exploration scene and international partners have expressed interest in collaborating with Canada on a few up-coming space missions, decreases in the AETD program's A-base funding over the past five years threaten to deter other space agencies from inviting Canada to participate in future missions. Without sufficient long-term funding to demonstrate solutions in space or to commit to international space missions, Canada risks losing its opportunity to apply the viable solutions it has developed. In tandem, Canada's influence on international space exploration decision-making may diminish.

Reductions in the AETD program's funding have already begun to have an adverse effect on the economic viability of the Canadian space exploration sector. Revenues generated by this sector decreased considerably after the majority of the Stimulus Initiative funding had been spent. Many AETD contract recipients reported that they have since been struggling to financially sustain their organizations. They explained that the AETD program has strongly influenced their organization's decision to work within the space exploration sector and that most will likely continue to do so. However, as further evidenced by the drop in revenues for 2012 among private enterprises that were awarded AETD contracts, the future economic viability of Canada's space exploration is precarious.

With respect to overall efficiency and economy, the AETD program has delivered outputs and outcomes efficiently and has used its resources in an economically sound manner for program implementation and delivery. More specifically, efficiency has been demonstrated by the value produced by the program with respect to use of public funds, especially given the favorable results achieved with a small budget (relative to the budgets allocated to other space agencies for achieving their space exploration results). In addition, the proportion of total AETD program spending allocated to salaries and program management is within an appropriate range. However, as explained above, by the end of the evaluation period there were too few resources dedicated to the program to ensure the continued achievement of several expected results. In addition, some key informants raised questions about the breadth of AETD solutions that could not be answered within the scope of this evaluation.

Of note, some key informants expressed concerns about redundancies in the AETD program's use of resources and those of two other CSA programs, namely the ETD and the OSM programs. Based on the available data for this evaluation, the CSA should either create a clearer distinction between the ETD program and the AETD program or merge the two programs in such a way that the planning and execution of technological, operational, and scientific space exploration developments remain integrated and aligned with future space exploration opportunities. In the case of the OSM program, the development of exploration-enabling medical technologies can benefit from the integration of technological, scientific, and operational elements afforded by the AETD program. Following the period of time covered by this evaluation period, a new governance structure for the ExCore component of the AETD program was put in place that will likely address duplication of efforts between the OSM and AETD programs.

To further improve upon the efficiency of program delivery, the AETD program would benefit from a means of reducing the reported inconsistencies in the interpretation of processes and procedures regarding the contracting process. Therefore, the AETD program should clarify the contracting processes regarding the AETD program's R&D activities and communicate these clarifications to AETD staff involved in contracting processes.

As mentioned in the Limitations section of this report, insufficient evidence was available to evaluate the performance of the AETD program with respect to some of the evaluation questions. For example, due to a lack of data, it was not possible to determine the extent to which solutions suitable to public engagement were produced or the effect of these solutions on the Canadian public's interest in space exploration. In other cases, lack of baseline data and targets precluded assessments of quantitative data, resulting in a heavy reliance on the opinions expressed by key informants. In order to support subsequent evaluations of the AETD program, its PM strategy should be revised to include baseline data and targets. In addition, all performance information should be made available in an accessible format.

With regard to the Stimulus Initiative, case study findings indicate that, though some setbacks in delivery of the initiative were experienced, it achieved its main objectives of maintaining key research and development personnel, and of sustaining the level of activities in space robotics. Specifically, AETD contract recipients (the majority of whom would have had to undergo downsizing without Stimulus funding) were able to maintain the level of FTEs and HQP across both their organizations and those of subcontracted organizations. With respect to activities in space robotics, $122M in Stimulus contracts were awarded, resulting in over 30 space exploration solutions that accelerated the development of space exploration technologies and increased the visibility of Canada's space exploration capabilities on the international scene. However, the quality and the appropriateness of the breadth of the solutions produced could not be ascertained from key informants' divergent perspectives.

The Stimulus Initiative also contributed to the economic viability of the Canadian space exploration sector by augmenting the amount of money spent on subcontracts and suppliers per prime Stimulus contract, relative to non-Stimulus contracts. In addition, Stimulus contract recipients used Stimulus funds to increase their organizations' space exploration R&D expenditures. Furthermore, the development of one of the two AETD solutions that has been flown on space missions was funded by the Stimulus Initiative.

Though the evaluation findings show that Stimulus resources were used in an economically sound manner, the available data did not allow for an assessment of the extent to which the initiative was efficient. However, evidence suggests that both external factors (e.g., an international partner's shift in priorities and challenges associated with the PWGSC contracting process) and internal factors (e.g., lack of consensus regarding the initiative's primary objectives) contributed to delays in the initiative's implementation and completion.

In summary, the AETD program demonstrates continued relevance and its performance to date has been generally successful, effective, and economical. However, evidence-based findings also point to opportunities for program improvement. These conclusions were echoed in key informants' overall assessments of the program, whereby the majority indicated that they are either satisfied or very satisfied with the AETD program (86% of AETD contract recipients, 100% of CSA senior executives, 82% of AETD managers, and 31% of AETD staff). Looking to the future, sufficient long-term program funding will be essential for maintaining the Canadian space sector's capabilities and economic viability, for participating in future international space missions, and for maintaining Canada's favourable reputation among international partners.

Based on the evaluation findings and conclusions, the CSA's Evaluation function recommends that the AETD program:

  1. Conduct an analysis of the optimal level of resources that should be dedicated to scientific development by the AETD program and, based on the results of this analysis, clearly communicate to program stakeholders the AETD program objectives with regard to scientific development and allocate resources accordingly;
  2. Either create a clearer distinction between the ETD program and the AETD program or merge the two programs while ensuring that the planning and execution of technological, operational, and scientific developments remain integrated and aligned with future space exploration opportunities;
  3. Clarify the contracting processes regarding the AETD program's R&D activities and communicate these clarifications to AETD staff involved in contracting processes;
  4. Implement a systematic method of communicating plans and priorities to all AETD staff; and
  5. Include baseline data and targets in the AETD program's PM strategy and make all performance data available in an accessible format.

6. References

  • Canadian Space Agency (2009). Exploration Core program: Space robotics stimulus initiative program brief. St. Hubert, QC: Government of Canada.
  • Canadian Space Agency (2010). Canadian Space Agency awarding contracts for terrestrial prototypes: Lunar exploration rovers [Press release November 25]. St. Hubert, QC: Government of Canada. Retrieved from news.gc.ca/web/article-en.do?nid=576159
  • Canadian Space Agency (2011). Departmental performance report: 2010-2011. St. Hubert, QC: Government of Canada. Retrieved from www.tbs-sct.gc.ca/dpr-rmr/2010-2011/inst/csa/csa00-eng.asp
  • Canadian Space Agency (2012a). Canadian space exploration plan. St. Hubert, QC: Government of Canada.
  • Canadian Space Agency (2012b). Departmental evaluation plan: 2012-2013 to 2016-2017. Audit and Evaluation Directorate. St. Hubert, QC: Government of Canada.
  • Canadian Space Agency (2012c). Minister Paradis unveils Canada's rover prototypes for the Moon and Mars. [Press release October 19]. St. Hubert, QC: Government of Canada. Retrieved from www.asc-csa.gc.ca/eng/media/news_releases/2012/1019.asp
  • Canadian Space Agency (2012d). Reports on plans and priorities: 2012-2013 estimates. St. Hubert, QC: Government of Canada. www.asc-csa.gc.ca/eng/publications/rpp-2012.asp
  • Canadian Space Agency (2012e). Departmental performance report: 2011-2012. St. Hubert, QC: Government of Canada. Retrieved from www.asc-csa.gc.ca/pdf/pr-2012.pdf
    Canadian Space Agency (2012f). State of the Canadian space sector 2012. St. Hubert, QC: Government of Canada. www.asc-csa.gc.ca/pdf/eng/industry/state-2012.pdf
  • Canadian Space Agency (2013a). Program activity architecture: 2012-2013. St. Hubert, QC: Government of Canada.
  • Canadian Space Agency (March, 2013b). The Advanced Exploration Technology Development program performance management strategy. St. Hubert, QC: Government of Canada.
  • Canadian Space Agency (2013c). Departmental performance report: 2012-2013. St. Hubert, QC: Government of Canada. Retrieved from www.asc-csa.gc.ca/pdf/eng/publications/pr-2013.pdf
  • Canadian Space Agency (2013d). Exploration Core: Exploration surface mobility EMS project closure report (Initial release). St. Hubert, QC: Government of Canada.
  • Canadian Space Agency (2014). Canada's space policy framework. St. Hubert, QC: Government of Canada.
  • Canadian Space Agency Act of 1990, S.C. 13. (1990 & Supp., 2012). Retrieved from laws-lois.justice.gc.ca/eng/acts/C-23.2/page-1.html
  • Department of Finance Canada (2010a). Canada's economic action plan year 2: Leading the way on jobs and growth. Ottawa, ON: Government of Canada. Retrieved from www.budget.gc.ca/2010/pdf/budget-planbudgetaire-eng.pdf
  • Department of Finance Canada (2010b). Canada's economic action plan year 2: 6th report to Canadians. Ottawa, ON: Government of Canada. Retrieved from www.fin.gc.ca/pub/report-rapport/2010-09-27/pdf/ceap-paec-2010-09-eng.pdf
  • Department of Finance Canada (2012). Economic action plan 2012: Jobs, growth and long-term prosperity. Ottawa, ON: Government of Canada. Retrieved from www.budget.gc.ca/2012/plan/pdf/Plan2012-eng.pdf
  • Grenier, L.L., Macintyre, I., Davidson, I., Nimmo, G., Zhang, M., Lalonde, P-G., …Cormier, A. (2008, July). An integrated approach towards electric drive transportation in Canada. Paper presented at the 6th International Energy Conversion Engineering Conference (IECE), Cleveland, OH. Retrieved from: arc.aiaa.org/doi/abs/10.2514/6.2008-5676
  • Industry Canada (2007). Mobilizing science and technology to Canada's advantage. Ottawa, ON: Government of Canada. Retrieved from www.ic.gc.ca/eic/site/icgc.nsf/vwapj/SandTstrategy.pdf/$file/SandTstrategy.pdf
  • Industry Canada (2012). Aerospace review Vol.2: Reaching higher: Canada's interests and future in space. Retrieved from aerospacereview.ca/eic/site/060.nsf/vwapj/Space-e-online.pdf/$file/Space-e-online.pdf
  • Industry Canada (2014). Parliamentary secretary Mike Lake celebrates space innovation with industry leaders in Québec City [Press release January 22]. Ottawa, ON: Government of Canada. Retrieved from news.gc.ca/web/article-en.do?mthd=tp&crtr.page=1&nid=810519&crtr.tp1D=1
  • Paradis, C. (2012, February 29). Speech given by the Minister of Industry at the Canadian Space Agency event. Québec, QC: Government of Canada. Retrieved from www.ic.gc.ca/eic/site/064.nsf/eng/07078.html
  • Piedboeuf, J.-C. (2010, September). Preparatory exploration activities: An overview of the Canadian space agency exploration core program. Paper presented at the International Astronautical Conference (IAC), Prague, Czech Republic.
  • Treasury Board of Canada (2009a). Policy on evaluation. Ottawa, ON: Government of Canada. Retrieved from www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15024&section=text
  • Treasury Board of Canada (2009b). Directive on evaluation function. Ottawa, ON: Government of Canada. Retrieved from www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15681&section=text
  • Treasury Board of Canada Secretariat (2010). Supporting effective evaluations: A guide to developing performance measurement strategies. Ottawa, ON: Government of Canada. Retrieved from www.tbs-sct.gc.ca/cee/dpms-esmr/dpms-esmr02-eng.asp
  • Treasury Board of Canada Secretariat (2011). Whole-of-government framework: Descriptions for Government of Canada outcome areas. Ottawa, ON: Government of Canada. Retrieved from www.tbs-sct.gc.ca/ppg-cpr/descript-eng.aspx

Management Response and Action Plan

  Responsibility Organization / Function Management response Details of action plan Schedule
Recommendation #1
Conduct an analysis of the optimal level of resources that should be dedicated to scientific development by the AETD program and, based on the results of this analysis, clearly communicate to program stakeholders the AETD program objectives with regard to scientific development and allocate resources accordingly. Director, Exploration Development and Director, Astronauts, Life Science and Space Medicine Use the exploration plan and roadmaps to determine the appropriate scientific content required under AETD. In the new ExCore Plan that is currently being developed, explicitly include some targets for technology, science, and operation activities under ExCore and communicate the plan to all stakeholders involved in ExCore. December 15, 2014
Recommendation #2
Either create a clearer distinction between the ETD program and the AETD program or merge the two programs while ensuring that the planning and execution of technological, operational, and scientific developments remain integrated and aligned with future space exploration opportunities. DG, Space Exploration and DG, Space Science and Technology and DG, Space Utilization We agree that there are a number of technology development activities carried out by CSA and that there is a need to better define the scope and implementation of each to ensure efficient program delivery. DG-level direction on the respective programs' scope is required in order to review the implementation approach.
  • The DG, Space Exploration and DG, Space Science and Technology and DG, Space Utilization will propose guidelines for technology development at the CSA, taking into account all the technology development activities to optimize CSA resources. If appropriate, the DGs will propose modifications to existing programs to make sure that they serve the best interests of the CSA.
  • The Directors under Space Exploration and Space Science and Technology (and Space Utilization) will implement the changes in their respective programs.
  • October 1 2014
  • March 31, 2015
Recommendation #3
Clarify contracting processes regarding AETD R&D activities and communicate these to AETD staff involved in these contracting processes. CSA's Chief Financial Officer and DG, Space Exploration We agree that issuing a contract is a relatively long process and that the process may be better documented and communicated to all those involved so everyone understand why it may take time to issue R&D contracts.
Note: This is applicable to all CSA's R&D contracts.
The CSA's Chief Financial Officer and the DG, Space Exploration will work together to document the process by which a R&D idea could become a R&D contract, including steps within CSA and PWGSC. October 1, 2014
Recommendation #4
Implement a systematic method of communicating plans and priorities to all AETD staff. Director Space Exploration Development The AETD program had the ambitious objective of integrating technology, science, and operation within a pre-mission program requiring people from various backgrounds working together. The concept is a success as demonstrated by the results. However, it was not always clear which aspects are priorities.

A new ExCore governance is now in place to make the decision process more transparent and to help communicating the information.

  • Present the overview of the AETD program's plans and results at each DG Program Review.
  • Organize a yearly AETD forum where detailed plans and results will be presented to all staff involved in AETD activities.
  • Twice a year during the DG Program Review
  • Fall every year
Recommendation #5
Includes baseline data and targets in the AETD program's PM strategy and makes all performance data available in an accessible format. DG, Space Exploration As documented in the Audit Management Report for the AEDT program, November 2013, the AETD program's PM strategy will be updated and will include baseline data and targets.

Making all performance data available in an accessible format is not solely the responsibility of AETD, it is a CSA-wide issue. AETD will make performance indicator measurements accessible.

Space Exploration will work with the corporate sector to have a database that can capture all the indicators. This database will be useful to all programs.

  • The AETD program's PM strategy will be updated and will include baseline data and targets.
  • Tools will be developed in order to collect and analyze data.
  • Data will be collected in accordance to the approved PM strategy. In the meantime, ad hoc tools will continue to be used to collect and stock data.
  • March 31 2014
  • March 31 2015
  • March 31 2016

Appendices

Appendix A: AETD Evaluation Strategy Framework

Evaluation question Indicator Data sources / methods Responsibility for data collection
Relevance
Are the program objectives aligned with federal government priorities? (R1)
  • Program's objectives correspond to current federal government priorities
  • Document review
  • Survey instrument for CSA senior executives
  • Evaluation
  • Evaluation
Are the program objectives aligned with departmental strategic outcomes? (R2)
  • Program's objectives correspond to current departmental strategic outcomes
  • Document review
  • Survey instrument for CSA senior executives
  • Evaluation
  • Evaluation
Is the program consistent with federal roles and responsibilities? (R3)
  • Program mandate aligned with federal government jurisdiction
  • Extent of appropriateness of federal involvement
  • Document review
  • Survey instrument for CSA senior executives
  • Evaluation
  • Evaluation
Does the program continue to address a demonstrable need and is it responsive to the needs of Canadians? (R4)
  • Strength of connection of program objectives with needs of Canadians
  • Gaps would exist in addressing societal/technological/economic need in absence of the program
  • Document review
  • Survey instrument for AETD contract recipients, AETD managers, and CSA senior executives
  • Evaluation
  • Evaluation
  • Evaluation
Performance: Achievement of Expected Results
Have contracts for advanced exploration science, technology and operational development been awarded? (Op1)
  • Forecasted budget fully invested
  • Number, type (competitive and directed) and value of new contracts issued
  • Percent of contracts (and amendments) awarded to SMEs, large enterprises, and academia
  • Regional distribution target (best effort)
  • Time of submitting the requisition to PWGSC to the time of contract award date
  • Stimulus Initiative only: Time from TB Submission approval to the time of submitting the requisition to PWGSC
  • Archival data review
  • AETD Program
  • Finance
  • Policy and External Relations
Have science, technology and operational solutions for space exploration been developed and delivered? (Op2)
  • Number and percent of solutions that have been delivered by type of solution
  • Number of science, technology, and operational solutions that are under development in conformity with the orientations and conclusions of the CSEP (PMF 1.2.2.3 R-1)
  • Archival data review
  • AETD program
Have solutions been suitable for public engagement? (Op3)
  • Number and percent of delivered solutions suitable for public engagement by type of solution and by both general public and space-community users
  • Number and percent solution utilized based on the number provided above
  • Number and percent of AETD-related CSA pages dedicated to public awareness
  • Archival data review
  • Communications
Have space exploration roadmaps and plans been produced? (Op4)
  • Number of plans and roadmaps
  • Number of major external stakeholder consultations (domestic and international) related to strategic planning (workshops, conferences, consultation communities, questionnaires)
  • Extent of external consultation
  • Extent of perceived impact of consultations on roadmaps and plans
  • Document review
  • Archival data review
  • Survey instrument for AETD contract recipients
  • AETD program
  • AETD program
  • Evaluation
Has Canada's HQP and infrastructure capacity for future space exploration been maintained or increased? (Oc1)
  • Opinions about AETD impact on HQP and infrastructure
  • Number of HQP and university students working on AETD contracts (for prime contractors and subcontractors)
  • Number of HQPs hired at beginning of contract and let go at end of contract
  • Percent of AETD funding allocated to infrastructure by contract recipients
  • Stimulus Initiative only: Opinions about potential lay-offs had Stimulus not been available
  • Percent of utilization of CSA infrastructure for AETD-related purposes (% realized by capacity per year) broken down by number of days available and type of Infrastructure
  • Frequency of utilization by type of AETD program target population
  • Extent to which CSA infrastructure was available when needed for the development and delivery of solutions
  • Survey instrument for AETD contract recipients
  • Archival data review
  • Survey instrument for AETD staff
  • Evaluation
  • AETD program
  • Evaluation
Has the Canadian space exploration scientific, technological and operational capability increased? (Oc2)
  • Opinions regarding extent to which AETD contracts have increased Canadian space exploration scientific, technological and operational capabilities
  • Opinions regarding the quality of the solutions produced
  • Opinions about the extent to which the AETD program has enabled organizations to increase their intellectual property
  • Intellectual property generated by AETD contracts belonging to the Crown and to the contractor
  • Number of publications and presentations related to solutions produced by AETD
  • Survey instrument for AETD contract recipients, CSA senior executives, and AETD managers
  • Survey instrument for AETD managers
  • Survey instrument for AETD contract recipients
  • Archival data review
  • Archival data review
  • Survey instrument for AETD contract recipients
  • Evaluation
  • Evaluation
  • Evaluation
  • Finance
  • CSA Library Services
  • Evaluation
Has Canada's position in the international space exploration scene been maintained or improved? (Oc3)
  • Extent to which AETD program is perceived to contribute to Canada's international positioning
  • Survey instrument for AETD contract recipients, CSA senior executives, AETD managers
  • Evaluation
Has the CSA's ability to make well-informed decisions for future missions and program development been maintained or improved? (Oc4)
  1. Extent to which AETD advice/analysis is perceived to contribute to decision-making
  2. Extent to which roadmaps and plans are perceived to be useful for decision-making
  • Survey instrument for CSA senior executives
  • Survey instrument for CSA senior executives and AETD managers
  • Evaluation
  • Evaluation
Has public interest in space exploration increased? (Oc5)
  • Number of single visits per year on AETD-related CSA web-pages
  • Average number of visits per hours per day on AETD-related CSA web-pages
  • Number of AETD-related press announcements
  • Number of events for AETD-related solutions per year by category (categories: visits to CSA, public events, trade shows, media demos, and miscellaneous)
  • Archival data review
  • Communications
Has the economic viability of the Canadian space exploration sector been maintained or increased? (Oc6)
  • Total revenue and FTEs of Canadian Space Exploration Sector 2006-2012
  • Total revenues and FTEs for AETD contract recipients
  • Number and value of sub-contracts awarded to SMEs, large enterprises, and academia
  • Number and value of suppliers
  • Number of CSA and non-CSA follow-on contracts (by SMEs and by large enterprises)
  • Opinions about the extent to which the AETD program has influenced whether organizations will pursue future space exploration endeavours
  • Archival data review
  • Survey instrument for AETD contract recipients
  • Policy and External Relations
  • Evaluation
Has Canadian leadership in science and signature technologies been maintained or increased? (Oc7)
  • Number of signature technologies stemming from AETD funding
  • Opinions of Canada's leadership in science solutions
  • Opinions of Canada's leadership and signature technologies
  • Survey instrument for AETD contract recipients
  • Survey instrument for AETD contract recipients, CSA senior executives, AETD managers
  • Evaluation
  • Evaluation
Has Canadian participation in international space exploration missions been maintained or increased? (Oc8)
  • Number of space exploration missions in which Canada participates and the percent that use AETD solutions
  • Number of up-coming international space exploration opportunities for Canada and number and percent of these resulting from AETD space exploration solutions
  • Number of international technical-level collaborations per year
  • Number of coordinated technology developments
  • Number of joint deployments
  • Archival data review
  • Survey instrument for AETD contract recipients
  • Survey instrument for AETD contract recipients
  • AETD archival data
  • AETD program
  • Evaluation
  • Evaluation
  • AETD program
Have space technology transfers to new applications been maintained or increased? (Oc9)
  • Number of new and/or improved (incremental innovation) product/process developments for space applications and non-space applications as a result of AETD contract
  • Survey instrument for AETD contract recipients
  • Evaluation
Has space R&D and Canadian participation in space exploration missions led to socio-economic benefits? (Oc10)
  • Number of non-space companies joining the space exploration sector
  • Dollar value of applications commercialized by Canadian space exploration organizations resulting from AETD-based solutions
  • Extent of space exploration commercial portfolio expansion among private enterprises that were awarded AETD contracts
  • Stimulus Initiative only: Percent increase in space exploration R&D expenditures as a direct result of Stimulus funding
  • Opinions and examples of increased access to new markets following AETD contracts
  • Archival data review
  • Survey instrument for AETD contract recipients
  • Survey instrument for AETD contract recipients and AETD managers
  • Policy and External Relations
  • Evaluation
  • Evaluation
Is Canada's participation in space exploration missions successful? (Oc11)
  • Number and percent of Canadian solutions (AETD-based and not AETD-based) that have performed according to established requirements during space missions
  • Survey instrument for AETD contract recipients
  • Evaluation
Has Canada maintained or increased its influence in international space exploration decision-making? (Oc12)
  • Number of space exploration advisory/ consultation /working groups participated in by the CSA and by AETD contract recipients
  • Nature of participation in these groups (e.g., member, chair, observer, other)

Note: the CSA's PAA identifies the following indicator: "Number of CSA's sponsored HQP nominated in International Space Exploration Decision bodies" (PMF 1.2.2 R-2). However, data for this indicator had not yet been collected at the time of the evaluation.

  • Archival data review
  • Survey instrument for AETD contract recipients
  • AETD program
  • Evaluation
Performance: Efficiency and Economy
Efficiency: To what extent is the program delivering outputs and achieving outcomes in the most efficient manner? (Ee1)
  • Alternatives that were explored to realize the outputs of this program at a lower cost
  • Opinions on whether the program was delivered efficiently with respect to resource utilization
  • Opinions on the impact of external factors on program delivery
  • Opinions on whether good value is being obtained with respect to the use of public funds
  • Opinions about overall quality of the program compared with other space agencies' programs
  • Opinions on the appropriateness of the breadth of the types of solutions funded
  • Survey instrument for CSA senior executives, AETD managers, and AETD staff
  • Survey instrument for AETD contract recipients, CSA senior executives, and AETD managers
  • Survey instrument for AETD contract recipients, CSA senior executives, AETD managers, and AETD staff
  • Evaluation
  • Evaluation
  • Evaluation
Economy: To what extent has resource use been minimized in the implementation and delivery of the program? (Ee2)
  • Planned-to-actual resource use (budget vs. expenditures)
  • Actual O&M costs in relation to total program spending
  • Cost of AETD contracts (not including contracts for program administration) in relation to total program costs, and salary and O&M costs
  • Dollar amount spent on salaries for every $1 spent on the AETD contracts (not including contracts for program administration)
  • Actual program operational costs in relation to the production of outputs
  • Opinions on appropriateness of program inputs
  • Opinions about redundancies with other programs
  • Comparison of program costs with those of other similar programs
  • Archival data review
  • Survey instrument for CSA senior executives, AETD managers, and AETD staff
  • Document review
  • Survey instrument for CSA senior executives, AETD managers, and AETD staff
  • AETD Program
  • Finance
  • Policy and External Relations
  • Evaluation
  • Evaluation

Appendix B: Interview Guide for AETD Contract Recipients

For any of the questions below, please note that "I don't know" is also a valid response.

  1. To what extent are you familiar with the AETD program?
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

Scientific, Technological, and Operational Capability

  1. To what extent has this program contributed to increases in your organization's scientific capability with regard to space exploration (that is, your organization's capability to enhance scientific expertise enabled by technologies, which can include scientific publications, reports, demonstrations, etc.)? (Oc2)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  2. To what extent has this program contributed to increases in your organization's technological capability with regard to space exploration (that is, your organization's capability to produce a technology that performs in space to achieve space science and human spaceflight objectives)? (Oc2)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  3. To what extent has this program contributed to increases in your organization's operational capability with regard to space exploration (that is, your organization's capability to maintain, inspect and operate tasks of complex space systems in preparation for future missions)? (Oc2)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  4. To what extent has this program had a beneficial impact on your organization's Highly Qualified Personnel capacity (HQP; defined as individuals with university degrees at the bachelors' level and above)? (Oc1)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  5. [Recipients of Stimulus Initiative contracts only] Had the Government of Canada's Economic Action Plan (that is, "Stimulus" funding) not been available, how likely is it that your organization would have had to undergo workforce downsizing (or, as the case may be, further workforce downsizing)? (Oc1)
    1. Not at all likely
    2. Slightly likely
    3. Moderately likely
    4. Very likely
    5. Extremely likely

    Please explain your answer.

  6. To what extent has this program had a beneficial impact on your organization's infrastructure capacity (for example, analogue sites, control centres, or integration facilities used for developing and testing space exploration scientific, technological, and operational solutions)? (Oc1)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  7. Overall, to what extent has this program effectively enabled your organization to increase its intellectual property? (Oc2)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

Commercialization and Future Space Exploration Endeavours

  1. [Private enterprises only] To what extent has your organization expanded its commercial product portfolio in space exploration as a result of having been awarded an AETD contract(s)? (Oc6)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  2. How likely is it that your organization will engage in future space exploration endeavours? (Oc6)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  3. To what extent has this program influenced your organization's likelihood of engaging in future space exploration endeavours? (Oc6)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    If there was an influence, in what way?

Positioning Canadian Space Exploration within the International Context

  1. To what extent has this program been instrumental in ensuring that Canada is well-positioned within the international space exploration context? (Oc3)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  2. To what extent has Canada's leadership in science solutions for space exploration (that is, its contribution to space science expertise and instruments) been increased as a result of this program? (Oc7)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  3. This question pertains to signature technologies, defined as well-established or emerging Canadian products or product lines for which Canada is or has the potential to become a world leader and that are usable for multiple space missions. Examples of signature technologies include optics, robotic servicing, spectrometers, rovers, planetary drilling and extractions, advanced crew medical systems, etc.).

    To what extent has Canada's leadership in signature technologies been increased as a result of this program? (Oc7)

    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  4. Compared with other space agencies' advanced space exploration technology development programs, how would you rate the overall quality of this program? (Ee1)
    1. Much worse
    2. Worse
    3. About the same
    4. Better
    5. Much better

    Please explain your answer.

  5. This question pertains to the CSA's space exploration roadmaps and plans. These roadmaps and plans include CSA's strategic and technical space exploration documents, used to decide upon exploration funding commitments and to position Canadian space exploration in the international scene.

    Was your organization consulted during the development of CSA's space exploration roadmaps or plans? (The types of consultation can include, but are not limited to CSEW6, CSEW9, the Exploration Signature Technology Consultation Committee, individual consultations, etc...) (Op4)

    • [] Yes
    • [] No
  6. If Yes, to what extent were you consulted during the development of the CSA's space exploration roadmaps and plans (Op4)
    1. Far too narrow
    2. Too narrow
    3. Appropriate breadth
    4. Too broad
    5. Far too broad
  7. If Yes, to what extent has your organization's input had an impact on CSA's plans or roadmaps, following this consultation? (Op4)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    For responses 1-3 only, please explain your answer.

Relevance of the Program

  1. To what extent is there a continued need for this program? (R4)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    For responses 1-3 only, please explain your answer.

  2. What Canadian needs are addressed by this program? (R4)
  3. What gaps would exist in the absence of this program? (R4)
  4. Overall, to what extent has this program produced good value with respect to the use of Canadian public funds? Please explain your answer (Ee1)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

Global Perceptions of the AETD Program

  1. To your knowledge, have there been any unintended outcomes (either positive or negative) of this program? (Un0)
    • [] Yes
    • [] No

    If yes, please explain your answer.

  2. Overall, how satisfied are you with this program?
    1. Very unsatisfied
    2. Unsatisfied
    3. Neither satisfied nor dissatisfied
    4. Satisfied
    5. Very satisfied

    Please explain your answer.

  3. What aspect(s) of this program do you value the most?
  4. What aspect(s) of this program do you value the least?
  5. Do you have any other comments you'd like to add about the AETD program that have not already been addressed in this interview?

Appendix C: Online Questionnaire for AETD Contract Recipients

According to our records, your organization has received the following prime contracts from the CSA's AETD program since 2008:

Contract #: Name of Contract:

Is this contract information accurate?

  • [] Yes
  • [] No

If no, please explain the inaccuracy:

Section 1. The following questionnaire items pertain to your organization as a whole.

Publications and Conference Presentations

  1. Since 2008, approximately how many publically available publications stemming from AETD contracts has your organization published? (Oc2) #
  2. Of these, approximately how many were published in a peer-reviewed journal? #
  3. Since 2008, approximately how many conference/convention presentations stemming from AETD contracts has your organization delivered? (Oc2) #

Participation in International Space Exploration Bodies

  1. In how many international space exploration advisory, consultation or working groups does your organization currently participate? (Oc12) #
  2. In how many of these groups does your organization fill the following roles?
    • Member: (#)
    • Chair: (#)
    • Observer: (#)
    • Other: (#)

    If other, please describe your organization's other role(s).

Infrastructure, R&D, and Follow-on Contracts

  1. Taking into account all AETD direct contracts that your organization has received since 2008, approximately what percentage of their total monetary value did your organization allocate to infrastructure (that is, to analogue sites, control centres or integration facilities used for developing and testing space exploration scientific, technological, and operational solutions)? % (Oc1)
  2. [Recipients of Stimulus Initiative contracts only] Overall, have your organization's space exploration research and development (R&D) expenditures increased as a direct result of the Government of Canada's Economic Action Plan (Stimulus) funding? (Oc10)
    • [] Yes
    • [] No

    If yes, by approximately what percentage? %

  3. Please identify below the approximate number and total dollar value ($) of follow-on contracts issued by both CSA and by other organizations that your organization has been awarded as a consequence of the prime AETD contract(s) that your organization has received. (Oc6)

    Approximate number of follow-on contracts from the CSA (not including AETD contracts): #

    Approximate total value of CSA follow-on contracts (not including AETD contracts):

    • [] Less than $500,000
    • [] 500,000 to $1 million
    • [] $1-5 million
    • [] Over $5 million

    Approximate number of follow-on contracts from other organizations: #

    Approximate total value of follow-on contracts from other organizations:

    • [] Less than $500,000
    • [] 500,000 to $1 million
    • [] $1-5 million
    • [] Over $5 million

Signature Technologies and Space Exploration Solutions

  1. This question pertains to signature technologies, defined as well-established or emerging Canadian products or product lines for which Canada is or has the potential to become a world leader and that are usable for multiple space missions. Examples of signature technologies include optics, robotic servicing, spectrometers, rovers, planetary drilling and extractions, advanced crew medical systems, etc.

    Since 2008, how many Canadian signature technologies stemming from AETD contracts has your organization developed? # (Oc7)

  2. This next question pertains to the development of space exploration solutions which include, but are not limited to, designs, reports, software, or construction, utilization or operation of scientific or technological instruments and tools needed to work in space. Solutions may refer to a complete set of instructions and protocols that are required for the successful use of a prototype. A solution can still be at a preliminary phase of development, but with sufficient information available to reach a decision concerning the furthering of its development.

    Since 2008, has your organization developed a solution(s) in preparation for participation in a specific space exploration mission(s)? Note: A specific space exploration mission is defined here as a venture of discovery that has or will take place in Earth orbit, on the International Space Station, on or near the Moon, on a planet or an asteroid, or in deep space. (Oc8)

    • [] Yes
    • [] No [If No, Skip to questionnaire item #10]

    If yes, how many solutions have been developed in preparation for a specific space exploration mission? #

    In the table below, please identify the name and the type of each solution, as well as whether it stemmed from an AETD contract, and whether it has already been used on a space exploration mission. (Oc8)

Name of solution Type of Solution Did the solution stem from an AETD contract? Has the solution already been used on a space exploration mission?
1.
  • []  Scientific
  • [] Operational
  • [] Technological
  • [] Yes
  • [] No
  • [] Yes
  • [] No
2.
  • []  Scientific
  • [] Operational
  • [] Technological
  • [] Yes
  • [] No
  • [] Yes
  • [] No
3.
  • []  Scientific
  • [] Operational
  • [] Technological
  • [] Yes
  • [] No
  • [] Yes
  • [] No
4.
  • []  Scientific
  • [] Operational
  • [] Technological
  • [] Yes
  • [] No
  • [] Yes
  • [] No

For each solution that has already been used on a space exploration mission, please provide the name of the mission and indicate the extent to which the solution performed successfully according to mission objectives. (Oc11)

Name of the Mission To what extent to which the solution performed successfully according to mission objectives (i.e., perform according to mission objectives). Please explain your answer
1.
  • [] (1) Not at all successful
  • [] (2) Slightly successful performance
  • [] (3) Moderately successful performance
  • [] (4) Very successful performance
  • [] (5) Extremely successful performance
 
2.
  • [] (1) Not at all successful
  • [] (2) Slightly successful performance
  • [] (3) Moderately successful performance
  • [] (4) Very successful performance
  • [] (5) Extremely successful performance
 
3.
  • [] (1) Not at all successful
  • [] (2) Slightly successful performance
  • [] (3) Moderately successful performance
  • [] (4) Very successful performance
  • [] (5) Extremely successful performance
 
4.
  • [] (1) Not at all successful
  • [] (2) Slightly successful performance
  • [] (3) Moderately successful performance
  • [] (4) Very successful performance
  • [] (5) Extremely successful performance
 

International space exploration opportunities

  1. Does your organization have any up-coming international space exploration opportunities (e.g., international invitations, requests to participate, requests for collaborations, etc.? (Oc8)
    • [] Yes
    • [] No

    If yes, how many up-coming international space exploration opportunities? #

    Among these opportunities, how many have stemmed from an AETD contract(s)? #

Applications, Commercialization, and New Markets

  1. As a result of AETD contracts since 2008, how many solutions has your organizations transferred to new and/or improved applications (Oc9)? #

    Among these applications, how many were:

    • Space-related: (#)
    • Non-space related: (#)
  2. Please indicate the approximate revenue that your organization has generated from the commercialization of these new applications? $ (Oc10)
  3. Has your organization gained increased access to new markets as a result of having been awarded an AETD contract(s)? (Oc10)
    • [] Yes
    • [] No

    If yes, to which market(s) has your organization gained increased access:

    • [] Health
    • [] Mining
    • [] Transportation
    • [] Space tourism
    • [] No new markets have been accessed to date
    • [] Other: (please indicate which other market(s)

Section 2: The Following questionnaire items pertain to each of the individual AETD contracts that your organization has received directly from the CSA.

  1. [Recipients of Stimulus Initiative contracts only] Approximately how many full-time equivalents (FTEs) worked on this contract? Note: 1 FTE is equivalent to 1 staff working full time or 2 or more staff working part-time to total the hours equivalent to 1 full-time position. (Oc1)

    For your organization: #

    For all other organizations that your organization subcontracted for this contract: #

  2. Approximately how many Highly Qualified Personnel (HQP) on this contract? HPQ is defined here as individuals with university degrees at the bachelor's level and above. (Oc1)

    For your organization: #

    For all other organizations that your organization subcontracted for this contract: #

Among these HQP, please identify approximately how many were (a) hired at the beginning of the contract and (b) let go following completion of the contract. (Oc1)

  By your organization By subcontracted organization(s)
Hired at the beginning of the contract Let go following completion of the contract Hired at the beginning of the contract Let go following completion of the contract
Approximate number of HQP:        
  1. Approximately how many university students worked on this contract? (Oc1)

    For your organization: #

    For all other organizations that your organization subcontracted for this contract: #

  1. What types of organizations were subcontracted to complete this contract? (Oc1)
    • [] Academia

      How many universities? #

      What was the approximate dollar value of the work subcontracted to complete the contract? $

    • [] Small and medium-sized enterprises (SMEs; defined as enterprises with fewer than 500 employees)

      How many SMEs? #

      What was the approximate dollar value of the work subcontracted to complete the contract? $

    • [] Large enterprise (defined as enterprises with 500 or more employees)

      How many large enterprises? #

      What was the approximate dollar value of the work subcontracted to complete the contract? $

  2. Approximately how many suppliers (i.e., companies that supplied materials) were used to complete this contract? (Oc1)

    Number of Canadian suppliers: #

    Number of foreign suppliers (including foreign suppliers who distribute in Canada): #

  3. What was the approximate total dollar value of the work contracted to suppliers for this contract? $ (Oc1)

ADDITIONAL COMMENTS

  1. Do you have any comments you'd like to add about the AETD program?

Appendix D: Interview Guide for AETD Program Managers

For any of the questions below, please note that "I don't know" is also a valid response.

  1. To what extent are you familiar with the AETD program?
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

Canadian Space Exploration Scientific, Technological, and Operational Capability

  1. To what extent has this program contributed to increases in Canadian Space Exploration scientific capability (that is, the capability to enhance scientific expertise enabled by technologies, which can include scientific publications, reports, demonstrations, etc.)? (Oc2)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  2. To what extent has this program contributed to increases in Canadian Space Exploration technological capability (that is, the capability to produce a technology that performs in space to achieve space science and human spaceflight objectives)? (Oc2)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  3. To what extent has this program contributed to increases in Canadian Space Exploration operational capability (that is, the capability to maintain, inspect and operate tasks of complex space systems in preparation for future missions)? (Oc2)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  4. Overall, how satisfied are you with the quality of the solutions produced by the AETD contracts? (Oc2)
    1. Very unsatisfied
    2. Unsatisfied
    3. Neither satisfied nor dissatisfied
    4. Satisfied
    5. Very satisfied

    Please explain your answer.

  5. How would you describe the appropriateness of the breadth of the types of solutions funded by the program? (Ee1)
    1. Far too narrow
    2. Too narrow
    3. Appropriate breadth
    4. Too broad
    5. Far too broad

    Please explain your answer.

Positioning Canadian Space Exploration within the International Context

  1. As a result of the suite of solutions that have stemmed from the program to date, do you anticipate that the extent of Canada's participation in international space missions will: (Oc8)
    1. Decrease a lot
    2. Decrease a little
    3. remain the same
    4. increase a little
    5. increase a lot

    Please explain your answer.

  2. To what extent has this program been instrumental in ensuring that Canada is well-positioned within the international space exploration context? (Oc3)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  3. To what extent has Canada's leadership in science solutions for space exploration (that is, its contribution to space science expertise and instruments) been increased as a result of this program? (Oc7)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  4. This question pertains to signature technologies, defined as well-established or emerging Canadian products or product lines for which Canada is or has the potential to become a world leader and that are usable for multiple space missions. Examples of signature technologies include optics, robotic servicing, spectrometers, rovers, planetary drilling and extractions, advanced crew medical systems, etc).

    To what extent has Canada's leadership in signature technologies been increased as a result of this program? (Oc7)

    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  5. Compared with other space agencies' advanced space exploration technology development programs, how would you rate the overall quality of this program? (Ee1)
    1. Much worse
    2. Worse
    3. About the same
    4. Better
    5. Much better

    Please explain your answer.

  6. In your opinion, has the Canadian space exploration sector gained increased access to new markets as a result of the program? (Oc10)
    • [] Yes
    • [] No

    If yes, to which market(s) has the Canadian space exploration sector gained increased access:

    • Health
    • Mining
    • Transportation
    • Space tourism
    • Other:

    Please provide an example(s):

CSA's Space Exploration Roadmaps and Plans

  1. To what extent have the CSA's space exploration roadmaps and plans been useful for decision-making with respect to Canadian Space Exploration? (Oc4)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    For responses 1-3 only, please explain your answer.

Relevance of the Program

  1. To what extent is there a continued need for this program? (R4)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    For responses 1-3 only, please explain your answer.

  2. What Canadian needs are addressed by this program? (R4)
  3. What gaps would exist in the absence of this program? (R4)
  4. To what extent do the needs addressed by this program overlap with needs addressed by other Canadian programs? (Ee2)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    For responses 3-5 only, please indicate which other program(s) and explain your answer

Efficiency and Economy of the Program

  1. Are you aware of any alternatives that were explored to realize the outputs of this program at a lower cost? (Ee1)
    • [] Yes
    • [] No

    If yes, please describe these alternatives.
    What was the rationale for selecting the approach used?

  2. To what extent has this program delivered efficiently with respect to resource utilization? (Ee1)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    For responses 1-3 only, please explain your answer.

  3. What can be done to improve the efficiency of AETD program delivery? (Ee1)
  4. What external factors (such as contextual issues, risks, or other assumptions) impacted the delivery of this program (either favourably or unfavourably)? (Ee1)

    Please explain in what way (e.g., cost, timing, quality, quantity or appropriateness of the AETD program).

  5. Would the development of exploration-enabling medical technologies be more efficiently carried out by the Operational Space Medicine program (SSP 1.2.3.2) or by the AETD program? (Ee1)
    • [] Operational Space Medicine program (OSM)
    • [] Advanced Exploration Development program (AETD)
    • [] Other (please specify):

    Please explain your answer.

  6. How would you describe the appropriateness of the program's inputs (such as human resources or spending costs) for achieving program delivery? (Ee2)
    1. Far too little
    2. Too little
    3. About right
    4. Too much
    5. Far too much

    Please explain your answer.

  7. Overall, to what extent has this program produced good value with respect to the use of Canadian public funds? (Ee1)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  8. Are you aware of any other programs, internationally, that have similar objectives to those of the AETD program? (Ee2)
    • [] Yes
    • [] No

    If Yes, which one(s)?

    How do the costs associated with this (or these) program(s) compare with those of the AETD program?

Global Perceptions of the AETD Program

  1. To your knowledge, have there been any unintended outcomes (either positive or negative) of this program? (Un0)
    • [] Yes
    • [] No

    If yes, please explain your answer.

  2. Overall, how satisfied are you with this program?
    1. Very unsatisfied
    2. Unsatisfied
    3. Neither satisfied nor dissatisfied
    4. Satisfied
    5. Very satisfied

    Please explain your answer.

  3. What aspect(s) of this program do you value the most?
  4. What aspect(s) of this program do you value the least?
  5. Do you have any other comments you'd like to add about the AETD program that have not already been addressed in this interview?

Appendix E: Interview Guide for CSA Senior Executives

For any of the questions below, please note that "I don't know" is a valid response.

  1. To what extent are you familiar with the AETD program?
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

Canadian Space Exploration Scientific, Technological, and Operational Capability

  1. To what extent has this program contributed to increases in Canadian Space Exploration scientific capability (that is, the capability to enhance scientific expertise enabled by technologies, which can include scientific publications, reports, demonstrations, etc.)? (Oc2)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  2. To what extent has this program contributed to increases in Canadian Space Exploration technological capability (that is, the capability to produce a technology that performs in space to achieve space science and human spaceflight objectives)? (Oc2)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  3. To what extent has this program contributed to increases in Canadian Space Exploration operational capability (that is, the capability to maintain, inspect and operate tasks of complex space systems in preparation for future missions)? (Oc2)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  4. How would you describe the appropriateness of the breadth of the types of solutions funded by the program? (Ee1)
    1. Far too narrow
    2. Too narrow
    3. Appropriate breadth
    4. Too broad
    5. Far too broad

    Please explain your answer.

Positioning Canadian Space Exploration within the International Context

  1. As a result of the suite of solutions that have stemmed from the program to date, do you anticipate that extent of Canada's participation in international space missions will: (Oc8)
    1. Decrease a lot
    2. Decrease a little
    3. Remain the same
    4. Increase a little
    5. Increase a lot

    Please explain your answer.

  2. To what extent has this program been instrumental in ensuring that Canada is well-positioned within the international space exploration context? (Oc3)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  3. To what extent has Canada's leadership in science solutions for space exploration (that is, its contribution to space science expertise and instruments) been increased as a result of this program? (Oc7)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  4. This question pertains to signature technologies, defined as well-established or emerging Canadian products or product lines for which Canada is or has the potential to become a world leader and that are usable for multiple space missions. Examples of signature technologies include optics, robotic servicing, spectrometers, rovers, planetary drilling and extractions, advanced crew medical systems, etc.).

    To what extent has Canada's leadership in signature technologies been increased as a result of this program? (Oc7)

    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  5. Compared with other space agencies' advanced space exploration technology development programs, how would you rate the overall quality of this program? (Ee1)
    1. Much worse
    2. Worse
    3. About the same
    4. Better
    5. Much better

    Please explain your answer.

CSA's Space Exploration Roadmaps and Plans

  1. To what extent have the CSA's space exploration roadmaps and plans been useful for decision-making with respect to Canadian Space Exploration? (Oc4)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    For responses 1-3 only, please explain your answer.

  2. Now more specifically about the AETD program, to what extent have the advice and analysis provided by AETD been useful to CSA's decision-making with respect to Canadian Space Exploration? (Oc4)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    For responses 1-3 only, please explain your answer.

Relevance of the Program

  1. To what extent is there a continued need for this program? (R4)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    For responses 1-3 only, please explain your answer.

  2. What Canadian needs are addressed by this program? (R4)
  3. What gaps would exist in the absence of this program? (R4)
  4. To what extent do the needs addressed by this program overlap with needs addressed by other Canadian programs? (Ee2)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    For responses 3-5 only, please indicate which other program(s) and explain your answer.

  5. To what extent are this program's objectives aligned with the federal government's priorities? (R1)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  6. To what extent are this program's objectives aligned with the CSA's strategic outcome (which is, Canada's exploration of space, provision of space services and development of its space capacity meet the nation's needs for scientific knowledge, innovation and information)? (R2)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  7. To what extent do you agree that this program should fall within the jurisdiction of the CSA, rather than a different organization? (R3)
    1. Strongly disagree
    2. Disagree
    3. Neither agree or disagree
    4. Agree
    5. Strongly agree

    For responses 1-3 only, please explain your answer and indicate where the jurisdiction for this type of program should lie (e.g., other federal government department, industry, provinces or other).

Efficiency and Economy of the Program

  1. Are you aware of any alternatives that were explored to realize the outputs of this program at a lower cost? (Ee1)
    • [] No
    • [] Yes

    If yes, please describe these alternatives.
    What was the rationale for selecting the approach used?

  2. To what extent has this program delivered efficiently with respect to resource utilization? (Ee1)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    For responses 1-3 only, please explain your answer.

  3. What can be done to improve the efficiency of AETD program delivery? (Ee1)
  4. What external factors (such as contextual issues, risks, or other assumptions) impacted the delivery of this program (either favourably or unfavourably)? (Ee1)

    Please explain in what way (e.g., cost, timing, quality, quantity or appropriateness of the AETD program).

  5. Would the development of exploration-enabling medical technologies be more efficiently carried out by the Operational Space Medicine program (SSP 1.2.3.2) or by the AETD program? (Ee1)
    • [] Operational Space Medicine program (OSM)
    • [] Advanced Exploration Development program (AETD)
    • [] Other (please specify):

    Please explain your answer.

  6. How would you describe the appropriateness of the program's inputs (such as human resources or spending costs) for achieving program delivery? (Ee2)
    1. Far too little
    2. Too little
    3. About right
    4. Too much
    5. Far too much

    Please explain your answer.

  7. Overall, to what extent has this program produced good value with respect to the use of Canadian public funds? (Ee1)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    Please explain your answer.

  8. Are you aware of any other programs, internationally, that have similar objectives to those of the AETD program? (Ee2)
    • [] No
    • [] Yes

    If Yes, which one(s)?
    How do the costs associated with this (or these) program(s) compare with those of the AETD program?

Global Perceptions of the AETD Program

  1. To your knowledge, have there been any unintended outcomes (either positive or negative) of this program? (Un0)
    • [] Yes
    • [] No

    If yes, please explain your answer.

  2. Overall, how satisfied are you with this program?
    1. Very unsatisfied
    2. Unsatisfied
    3. Neither satisfied nor dissatisfied
    4. Satisfied
    5. Very satisfied

    Please explain your answer.

  3. What aspect(s) of this program do you value the most?
  4. What aspect(s) of this program do you value the least?
  5. Do you have any other comments you'd like to add about the AETD program that have not already been addressed in this interview?

Appendix F: Online Questionnaire for AETD Program Staff

  1. To what extent are you familiar with the AETD program?
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

Efficiency and Economy of the Program

  1. To what extent has this program delivered efficiently with respect to resource utilization? (Ee1)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    For responses 1-3 only, please explain your answer.

  2. How would you describe the appropriateness of the program's inputs (such as human resources or spending costs) for achieving program delivery? (Ee2)
    1. Far too little
    2. Too little
    3. About right
    4. Too much
    5. Far too much

    Please explain your answer.

  3. To what extent were inputs made available to ensure timely completion of program activities? (Ee2)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    For responses 1-3 only, please explain your answer.

Availability of Infrastructure

  1. To what extent was CSA infrastructure available when you needed it to work on the AETD program? Note: Infrastructure refers to analogue sites, control centers or integration facilities used for developing and testing space exploration scientific, technological, and operational solutions (Oc1)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely
    6. Not applicable

    For responses 1-3 only, please explain your answer.

Global Perceptions of the AETD Program

  1. Overall, to what extent has this program produced good value with respect to the use of Canadian public funds? (Ee1)
    1. Not at all
    2. Slightly
    3. Moderately
    4. Very
    5. Extremely

    For responses 1-3 only, please explain your answer.

  2. Overall, how satisfied are you with this program?
    1. Very unsatisfied
    2. Unsatisfied
    3. Neither satisfied nor dissatisfied
    4. Satisfied
    5. Very satisfied

    Please explain your answer.

  3. What aspect(s) of this program do you value the most?
  4. What aspect(s) of this program do you value the least?
  5. Do you have any other comments you'd like to add about the AETD program that have not already been addressed in this interview?

Appendix G: Focus Group Guide for AETD Program Staff

  1. To your knowledge, are there other programs, internationally, that have similar objectives to those of the AETD program? (Ee2)

    If Yes, which one(s)?

    How do the costs associated with this (or these) program(s) compare with those of the AETD program?

  2. What external factors (such as contextual issues, risks, or other assumptions) impacted the delivery of this program (either favourably or unfavourably)? (Ee1)

    Please explain in what way (e.g., cost, timing, quality, quantity or appropriateness of the AETD program).

  3. Are you aware of any alternatives that were explored to realize the outputs of this program at a lower cost? (Ee1)

    If yes, Please describe these alternatives. What was the rationale for selecting the approach used.

  4. What can be done to improve the efficiency of AETD program delivery [excluding Stimulus]? (Ee1)
  5. To your knowledge, have there been any unintended outcomes (either positive or negative) of this program? (Un0)

    If yes, please explain your answer.

  6. Are you aware of any AETD solutions that have been transferred to new and/or improved applications? (Oc9)

    If yes, which ones?

  7. Do you have any other comments you'd like to add about the AETD program that have not already been addressed in this interview?

Footnotes

Footnote 1

Materiality risks were identified as low in the CSA's 2012-2013 to 2016-2017 Departmental Evaluation Plan (2012b) because they were based on the program's forecasted budget over the next five years. Had these risks been determined based on retrospective program spending over the evaluation period, they would have been identified as high due to the added funding provided by the Stimulus Initiative.

Back to footnote 1 referrer

Footnote 2

The current President of the CSA was appointed at the time that interviews were conducted, but after the end of the evaluation period. The former President held this position throughout the majority of the evaluation period.

Back to footnote 2 referrer

Footnote 3

There was a net average of -0.2 (SD = 1.2) HQP from the start to the completion of AETD contracts, with a net range of -4 to +8 HQP per contract. Furthermore, the net change in HQP from contract start to contract completion dates did not differ from zero in 98% of reported cases.

Back to footnote 3 referrer

Footnote 4

These data were extracted from the CSA's library databases. Therefore, they likely represent an under-estimation of the total number of presentations and publications.

Back to footnote 4 referrer

Footnote 5

One of the TriDAR space missions and four of the Q6 missions took place shortly after the end of the evaluation period.

Back to footnote 5 referrer