Evaluation of the Class Grant and Contribution Program

For the period from to

Project # 14/15 02-04

Prepared by the Audit and Evaluation Directorate

Table of Contents

  1. Introduction
  2. Evaluation Approach and Methods
  3. Results
  4. Conclusions and Recommendations
  5. References

List of Tables

List of Figures

Acronyms and Definitions Used in the Report

Acronyms

  • AO: Announcement of opportunity
  • CSA: Canadian Space Agency
  • CoE: Centre of Expertise
  • DG: Director general
  • FTE: Full time equivalent
  • G&C: Grant and contribution
  • HQP: Highly qualified personnel
  • NRC: National Research Council
  • NRCan: National Resources Canada
  • NSERC: Natural Sciences Research and Engineering Council of Canada
  • OGD: Other government departments
  • O&M: Operations and maintenance
  • PAA: Program alignment architecture
  • PM: Performance measurement
  • R&D: Research and development
  • SD: Standard deviation
  • S&T: Science and technology
  • TB: Treasury Board of Canada
  • TBS: Treasury Board of Canada Secretariat

Definitions

Announcement of opportunity:

A time-bound, competitive process for soliciting and selecting proposals that addresses certain themes and/or sectors of intervention and/or certain types of recipients.

Funded projects:

Recipients' projects funded through the CSA's Class Grant and Contribution Program.

Funding mechanisms:

The processes through which solicited and unsolicited proposals are submitted and selected.

Highly qualified personnel:

Individuals with university degrees at the bachelors' level and above.

Research team:

The highly qualified personnel involved directly and indirectly in funded initiatives and projects, including faculty, students (undergraduates, graduates, post-doctoral level), and other personnel.

Solicited proposals:

A proposal submitted in response to a CSA, time-bound, Announcement of Opportunity to address certain themes and/or sectors of intervention and/or certain types of recipients. Projects are evaluated against the Program's and Announcement of Opportunity's assessment criteria and in comparison with each other. Funds are set aside for this purpose.

Standard deviation:

The extent of deviation or dispersion from the average. A low standard deviation indicates that the data points tend to be very close to the mean, whereas a high standard deviation indicates that the data points are spread out over a large range of values.

Unsolicited proposals:

A funding proposal submitted at any time, typically outside of the solicited proposal AO timeframe. Projects are evaluated individually against the CSA's priorities and the Program's assessment criteria, and may be approved based on merit and availability of funds.

Executive Summary

The Class Grant and Contribution (G&C) Program is designed to support the overarching objectives of the Canadian Space Agency (CSA) by allowing the CSA to work closely with Canadian universities and industries in order to advance space knowledge, develop and demonstrate new technologies, and help train the skilled workforce needed by Canada's universities and high tech sectors in partnership with the Canadian granting councils. As per the program's Terms and Conditions approved in , the program includes two main components, the Research component and the Awareness and Learning component. However, the Class G&C Program has focused primarily on funding initiatives under the Research component since . Through the Research component, the Class G&C Program aims to support targeted knowledge development and innovation that will sustain and enhance the Canadian capacity to use space in addressing national needs and priorities.

This report contains the evaluation of the Class G&C Program's relevance and performance over the course of the evaluation period (- to -). Emphasis was placed on evaluating the program's Research component because it represented $36M of the $40M in total program spending during the evaluation period ($4M was spent on the Awareness and Learning component). In carrying out this evaluation, a participatory and utilization-focus approach was employed that entailed the use of mixed quantitative and qualitative methods. Specifically, document and archival data reviews were performed, online questionnaires were administered to 124 program stakeholders (including funding recipients, non-recipients, and CSA employees), and telephone interviews were conducted with representatives from five other government departments (OGDs).

As a whole, the Class G&C Program remains relevant and continues to meet the needs of the Canadian space sector. It is a vital program to many organizations throughout Canada, particularly academia. Its performance has been effective with respect to achievement of the program's expected outcomes. It has successfully funded 195 projects over the course of the evaluation period, leading to important achievements for space-related knowledge and capacity. The program is also a good use of public funds as it has made a positive impact on Canada's reputation for space-related research and development (R&D) with modest investments.

Program Relevance

In terms of relevance, the Class G&C Program aligns with federal priorities and is consistent with federal roles and responsibilities. Alignment with departmental priorities was also demonstrated for the Research component, but was not evaluated for the Awareness and Learning component. The Research component responds to the need for a federal program that contributes to the attainment of the CSA's mandate and priorities by building Canadian capacity for space-related activities and fostering collaboration among national and international partners. To further respond to the needs of the Canadian space sector, key informants suggested continuing program funding in domains targeted by prior Announcements of Opportunity (AOs) and using the Class G&C Program more extensively across the CSA's branches in order to support R&D in a broader variety of CSA priorities (e.g., science support to space missions).

Effectiveness of Program Performance

Achievement of Expected Outputs

The Program performed well in producing all its intended outputs, such as launching AOs, producing ranked lists of eligible applicants and awarding G&Cs. According to most key informants, effective funding application mechanisms were in place, the AO selection process was fair, and the reporting requirements were similar to those of OGDs. However, information regarding both the nature of the AO proposal selection process and delays in the AO selection process was not provided to applicants consistently across AOs, and constructive feedback on proposals was also provided inconsistently. To address these shortcomings and in keeping with TBS recommendations for transfer payment programs, the CSA should standardize the application, selection, and feedback processes for both solicited and unsolicited proposals, and clearly communicate these processes to the Canadian space community.

Although the Class G&C Program's Performance Measurement (PM) Strategy identifies performance indicators, baseline data, and targets for the program's expected outcomes, there is a need to update the program's PM Strategy to ensure that it contains output-related indicators and baseline data and targets. Also, some of the G&C data stored in the CSA's databases were inaccurate, indicating a need to improve the data entry process in order to ensure archival data validity.

Achievement of Expected Outcomes

The Class G&C Program successfully achieved the expected outcomes of the program and in many cases exceeded the program's targets and key informants' expectations. The extent of knowledge gained by funded projects was considerable, especially with respect to technological or scientific breakthroughs and also with respect to developing new ideas for integration in future space missions, use of satellite data, the development of applications and algorithms, and the creation of new R&D projects. According to key informants, the extent of new knowledge resulting from funded projects either met or surpassed their initial expectations. Also, a considerable number of presentations and publications were produced based on the knowledge gained from the majority of funded projects.

Although there was very little funding awarded to industry and not-for-profit organizations, the program played an important role in supporting a space focus in academia. Half of the funded projects brought new players into space-related research fields during the course of the evaluation period. Furthermore, the number of organizations participating in funded projects' research teams increased considerably and included national and international representation from academia, industry, and other types of organizations. Projects funded by the program also led to notable increases in highly qualified personnel and space-related expertise capacity, as well as to new and maintained national and international collaborations. For example, funding recipients partnered with 225 different organizations in order to carry out their funded projects. The majority of these partnering organizations were from Europe, the United States, and other countries (e.g., Japan, Australia). In addition, multidisciplinary collaborations and achievements surpassed the program's target in which 50% of funded projects resulted in at least one multidisciplinary achievement.

Program Efficiency and Economy

With respect to efficiency and economy, the Class G&C Program is unique in its targeted support of Canadian space-related science and technology development. The program produces good value with respect to use of public funds by achieving its expected outcomes and bolstering Canada's reputation for space research and development through modest and flexible investments. Though salary dollars allocated by the CSA's main programs for managing G&C initiatives were not tracked, available data show alignment between forecasted and actual spending on G&C agreements. In addition, as G&C agreement expenditures increased over time, costs associated with the program's Center of Expertise (CoE) decreased.

Based on evaluation findings and conclusions, the CSA's Evaluation function recommends the following:

  1. The Class G&C Program's Terms and Conditions should be reviewed to determine whether the Awareness and Learning component remains aligned with the CSA's priorities.
  2. For both solicited and unsolicited proposals, the application, selection, and feedback processes should be standardized and clearly communicate to the Canadian space community.
  3. The program's PM Strategy, as well as data entry, collection, and storage processes, should be revised to ensure performance data availability, validity, and accessibility. In addition, the process for identifying ranked lists of funding priorities applicable to all G&C initiatives should be standardized across the CSA's main branches.

1. Introduction

This document constitutes the evaluation of the Canadian Space Agency (CSA) Class Grant and Contribution (G&C) Program to Support Awareness, Research and Training in Space Science and Technology, hereafter referred to as the Class G&C Program. The evaluation was conducted during the - and - fiscal years by the CSA's Audit and Evaluation Directorate (specifically, the CSA's Evaluation function) in response to the Treasury Board of Canada Secretariat's (TBS) Policy on Evaluation (2009a) and to the Policy on Transfer Payments (), which require that all ongoing federal G&Cs programs be evaluated every five years. The evaluation covers the period from to .

1.1 Program History and Description

The Class G&C Program is designed to support the overriding objectives of the CSA by allowing the CSA to work closely with Canadian granting councils, universities, and industries in order to advance space knowledge, develop and demonstrate new technologies, and help train the skilled workforce needed by Canada's universities and high tech sectors.

The Class G&C Program has been in place since -. Originally, the program was comprised of 11 distinct components and an additional component was subsequently added in . However, following recommendations stemming from a summative program evaluation (CSA, ), new Terms and Conditions were approved by the Treasury Board of Canada (TB) which streamlined the program into two main components:

  1. Awareness and Learning and
  2. Research.

The purpose of the Awareness and Learning component is to support initiatives targeting Canadian youth, students, physicians, and educators focusing on increasing their awareness, knowledge development, and participation in space-related disciplines/activities and advanced educational programs. However, following a review of the CSA's programs in , it was decided that initiatives under the Awareness and Learning component aimed at elementary and secondary school students would no longer be financially supported. Therefore, with the exception of a few initiatives for post-secondary students, the Class G&C Program has focused primarily on funding initiatives under the Research component since . Over the course of the evaluation period, the Awareness and Learning component represented only $4M of the $40M spent on the Class G&C Program.

The purpose of the Research component is to promote space-related research and development (R&D) in CSA priority areas by providing financial support to organizations. Through the Research component, the Class G&C Program aims to support targeted knowledge development and innovation that will sustain and enhance the Canadian capacity to use space in addressing national needs and priorities. Over the course of the evaluation period, the Research component represented $36M of the $40M spent on the Class G&C Program.

Given that the Class G&C Program has focused primarily on funding initiatives under the Research component since , its revised Performance Measurement (PM) Strategy covers only the program's Research component (CSA, 2013a). According to this PM Strategy, the target population for the Research component includes:

  • Canadian universities and post-secondary institutions (i.e., academia);
  • for-profit organizations (i.e., industry) established and operating in Canada;
  • not-for-profit organizations established and operating in Canada;
  • not-for-profit international research organizations; and
  • the scientific and technological community at large, as a result of access to a larger pool of knowledge and information for future work.

The program stakeholders include:

  • other government departments (OGDs) (e.g., Canadian granting councils);
  • academia across Canada;
  • industry across Canada;
  • not-for-profit organizations; and
  • the international space community.

The Class G&C Program's funding is allocated either through a competitive process (referred to as solicited proposals) in response to time-bound Announcements of Opportunity (AOs) or through unsolicited proposals.

1.2 Governance and Roles and Responsibilities

The Class G&C Program's governance includes a permanent G&C Committee that is supported by a G&C Advisory Committee and a G&C Centre of Expertise (CoE). During the course of the evaluation period, the CoE's functions were part of the CSA's Future Canadian Space Capacity program (program 1.3 in the CSA's Program Alignment Architecture; PAA) and, since , these functions falls under Internal Services (PAA program 1.4). However, the Class G&C Program is not a distinct program in the CSA's PAA. Rather, it supports G&C initiatives implemented across all four of the main PAA programs – namely, Space Data, Information and Services (PAA 1.1), Space Exploration (PAA 1.2), Future Canadian Space Capacity (PAA 1.3), and Internal Services (PAA 1.4).

Strategic direction for the Class G&C Program, including risk tolerance, is determined through the G&C Committee. The Executive Committee acts as the G&C Committee and is chaired by the President of the CSA. According to the G&C Committee Terms of Reference (CSA, 2011a), the mandate of the G&C Committee is to serve as a forum for senior officials to provide corporate and strategic direction in the selection, design, implementation, and establishment of departmental policies and transfer payment frameworks. The main responsibilities of the G&C Committee are to:

  • Provide direction and approval for new, renewed or modified transfer payment programs, and formulate recommendations on such initiatives to the approval authority;
  • Provide direction and approval on departmental policies and frameworks such as, but not limited to, service standards, monitoring and reporting, and recipient audit plans to ensure that administrative requirements imposed on recipients are proportionate to the risk level; and
  • Approve all high risk project proposals, as recommended by the G&C Advisory Committee before being signed by the approval authority.

As per the G&C Advisory Committee Terms of Reference (CSA, 2011b), the mandate of the G&C Advisory Committee is to serve as a focal point for the review and analysis of all activities and documents associated with the management of grants and contributions prior to making recommendations to the G&C Committee. The G&C Advisory Committee is chaired by a member on a rotational basis for a two-year mandate and is comprised of a director or equivalent representative of each of the following sectors and branches:

  • Vice-President's Office, Space Exploration, Space Utilization, Space Science & Technology;
  • Finance, Corporate Services, and Legal Services; and
  • the G&C CoE.

The main responsibilities of the G&C Advisory Committee are to:

  • identify opportunities for improved services, streamline, standardize, and harmonize the application process, introduce risk management practices, and enhance stakeholder engagement;
  • provide guidance for the development or renewal of program terms and conditions, departmental policies, procedures and tools, and other activities related to the management of G&Cs, as well as make recommendations on such initiatives to the G&C Committee for approval; and
  • review all high risk project proposals and make recommendations to the G&C Committee for approval.

Harmonization and standardization of program delivery and reporting is undertaken by the G&C CoE, which is comprised of permanent staff who act as coordinators, facilitators, and resource persons for all G&C initiatives managed by the CSA. As per the CoE Terms of Reference (2010a), the mandate of the CoE is to provide G&C expertise, services in G&C management, program oversight, and performance for all G&C programs of the CSA. The CoE is also responsible for supporting the activities of the G&C Advisory Committee. The key responsibilities are grouped under four main roles:

  • G&C expertise
    • Serves as the centre of knowledge in the area of G&Cs;
    • Provides guidance and direction on the implementation of the Government of Canada G&C Policy requirements;
    • Provides expertise in the design, implementation, and maintenance of CSA-wide governance framework, infrastructure, and training in G&C; and
    • Promotes sharing of best practices.
  • Portfolio services in G&C management
    • Provides operational expertise, guidance, and services in G&C management through matrix support to other branches in the development and delivery of their programs and initiatives; and
    • Provides support for specific initiatives and/or CSA-wide initiatives involved with granting councils and foundations through a specific understanding with the sponsoring branch (e.g., NSERC Industrial Research Chair and Partnerships Support Program).
  • Program oversight and performance
    • Performs an oversight function on all G&C programs;
    • Fosters collaboration within and among CSA, as well as with other OGDs;
    • Develops and produces the annual Recipient Audit Plan; and
    • Develops and produces the annual G&C Performance Report that is focused on outcomes.
  • Support of G&C Advisory Committee activities
    • Coordinates the activities of the G&C Advisory Committee; and
    • Provides secretariat services to the G&C Advisory Committee.

At the implementation level, working groups are established to work on a thematic basis across the CSA to identify the activities needed to deliver program outputs and attain the specified outcomes (CSA, 2013a). The key roles and responsibilities are to:

  • identify opportunities for improved services;
  • streamline, standardize, and harmonize the application process;
  • introduce risk management practices; and
  • enhance stakeholder engagement.

Branches are responsible for identifying which initiatives will be supported and managing the budgets for initiatives in order to attain the specified outcomes of the program. The key roles and responsibilities are to:

  • ensure that branches' staff accurately monitor and assess risk using the Risk Management Tools;
  • engage in direct contact with recipients and monitor initiatives' advancement, feasibility, and ongoing eligibility;
  • monitor risk throughout the course of initiatives; and
  • report initiative risks and concerns to the supervisor and to the G&C Advisory Committee through the CoE.

1.3 Resource Allocation

In this section, financial data pertaining to the Class G&C Program's resource allocation are provided for the program as a whole, as well as for the Research component, the Awareness and Learning component, and the CoE.

Table 1 shows the financial resources spent on grants and contributions awarded via the CSA's Class G&C Program over the course of the evaluation period. Total financial resources are provided, as are sub-totals for grants and contributions, and for each of the CSA's four main Programs as per the CSA's PAA. The table also includes forecasted budgets as identified in annual work plans.

Table 1: Amount forecasted and spent on Class G&C agreements over the course of the evaluation period.
- - - - - Total
Forecasted Budget ($)Table note 1
Type of Funding
Contribution 250 2,390 4,301 1,587 2,050 10,578 (29%)
Grant 3,624 4,764 5,227 6,044 6,395 26,054 (71%)
PAA Program
1.1 Space Data, Information and Services 0 0 500 1,000 3,275 4,775 (13%)
1.2 Space Exploration 2,010 0 721 1,169 824 4,724 (13%)
1.3 Future Canadian Space Capacity 1,864 7,004 8,307 5,462 4,346 26,983 (74%)
1.4 Internal Services 0 150 0 0 0 150 (<1%)
Program Total 3,874 7,154 9,528 7,631 8,445 36,632
Actual Spending ($)
Type of Funding
Contribution 250 1,254 3,337 1,015 1,520 7,376 (18%)
Grant 5,700 5,963 8,319 6,223 6,293 32, 498 (82%)
PAA Program
1.1 Space Data, Information and Services 0 99 681 2,035 2,735 5,550 (14%)
1.2 Space Exploration 0 142 1,222 796 852 3,012 (8%)
1.3 Future Canadian Space Capacity 5,950 6,976 9,750 4,407 4,226 31,309 (79%)
1.4 Internal Services 0 0 3 0 0 3 (<1%)
Program Total 5,950 7,217 11,656 7,238 7,813 39,874

Source: The CSA's Finance Directorate

Note: Values are represented in thousands of dollars.

Table notes

Table note 1

As approved in annual work plans.

Return to table note 1 referrer

Table 2 presents the forecasted budgets for transfer payments through the CSA's Class G&C Program. A comparison of Tables 1 and 2 indicates that the forecasted budget for each of the next four fiscal years is consistently higher than was the actual annual spending over each of the last five fiscal years.

Table 2: Planned spending for Class G&C agreements over the four fiscal years following the evaluation period.
- - - - Total
Contribution 5,440 11,685 9,800 9,200 36,125 (56%)
Grant 7,008 7,456 7,237 6,792 28,493 (44%)
Total 12,448 19,141 17,037 15,992 64,618

Source: - Report on Plans and Priorities

Note: Values are represented in thousands of dollars.

Tables 3 and 4 present detailed financial information regarding the amounts spent on grants and contributions awarded via each of the two components of the Class G&C Program: Research and Awareness and Learning. As described above, a decision was made in to no longer support initiatives under the Awareness and Learning component aimed at elementary and secondary school students. Therefore, the Research component accounted for 98% of the Class G&C Program's spending since the CSA's program review in and 91% of spending over the course of the evaluation period.

Table 3: Amount spent on Research component G&C agreements over the course of the evaluation period.
- - - - - Total
Type of Funding
Solicited
Contribution 0 0 0 90 978 1,068 (5%)
Grant 2,921 3,769 5,258 3,360 3,414 18,722 (95%)
Sub-total 2,921 3,769 5,258 3,450 4,392 19,790 (55%)
Unsolicited
Contribution 0 875 2,614 646 542 4,677Table note 2 (28%)
Grant 2,065 1,431 2,561 2,822 2,879 11,758Table note 3 (78%)
Sub-total 2,065 2,306 5,175 3,468 3,421 16,435 (45%)
PAA Program
1 .1 Space Data, Information and Services 0 99 681 2,035 2,735 5,550 (15%)
1.2 Space Exploration 0 142 1,222 796 852 3,012 (8%)
1.3 Future Canadian Space Capacity 4,986 5,834 8,527 4,087 4,226 27,660 (76%)
1.4 Internal Services 0 0 3 0 0 3 (<1%)
Research Component Total 4,986 6,075 10,433 6,918 7,813 36,225

Source: The CSA's Finance Directorate

Note: Values are represented in thousands of dollars.

Table notes

Table note 2

Includes a $3,6M project initially proposed in response to a request for proposals for contracts and assessed both in accordance to the procurement rules and to those of G&Cs.

Return to table note 2 referrer

Table note 3

Includes two projects totaling $1.5M, initially proposed in response to a request for proposals for contracts and assessed both in accordance to the procurement rules and to those of G&Cs.

Return to table note 3 referrer

Table 4: Amount spent on Awareness and Learning component G&C agreements over the course of the evaluation period.
- - - - - Total
Type of Funding
Contribution 250 379 723 279 0 1,631 (45%)
Grant 714 763 500 41 0 2,018 (55%)
PAA Program
1.1 Space Data, Information and Services 0 0 0 0 0 0 (0%)
1.2 Space Exploration 0 0 0 0 0 0 (0%)
1.3 Future Canadian Space Capacity 964 1,142 1,223 320 0 3,649 (100%)
1.4 Internal Services 0 0 0 0 0 0 (0%)
Awareness and Learning Component Total 964 1,142 1,223 320 0 3,649

Source: The CSA's Finance Directorate

Note: Values are represented in thousands of dollars.

Resources allocated to the CoE are presented in Table 5. Note that the mandate and structure of the CoE, as described in the section above on governance and roles and responsibilities, was established in . Therefore, CoE resources for - are not provided in the table.

Additional costs incurred by the CSA in managing and administering the Class G&C Program are funded from existing reference levels across the CSA and are not systematically tracked. Therefore, data pertaining to the resources allocated to program management and administration (other than CoE resources) were not available.

Table 5: Resources allocated to the CoE for the Class G&C Program over the course of the evaluation period.
Type of Resource - - - - -
FTEsTable note 4
Forecasted - 4.67 4.64 4.85 5.00
Actual - 4.66 4.56 4.81 3.47
Forecasted Budget ($)Table note 5
SalaryTable note 6 - 420 420 401 478
Operations and maintenance (O&M) - 340 277 100 75
Capital - 0 0 0 0
Program Total - 760 697 501 553
        4-year total  = 2,511
Actual Spending ($)
Salary - 445 436 465 349Table note 7
O&M - 279 209 58 29
Capital - 0 0 0 0
Program Total - 724 645 523 378
        4-year total = 2,270

Source: The CSA's Finance Directorate

Table notes

Table note 4

FTEs are full time equivalents.

Return to table note 4 referrer

Table note 5

As approved in annual work plans.

Return to table note 5 referrer

Table note 6

Excludes employee benefit plan.

Return to table note 6 referrer

Table note 7

This amount does not include the overtime in - because it was compensated by leave with pay.

Return to table note 7 referrer

Note: Values are represented in thousands of dollars.

1.4 Program theory

The logic model in Figure 1 depicts the theory behind the Class G&C Program's Research component by providing a visual representation of how the resources allocated to the program are used to achieve expected outcomes. The narrative that follows the logic model draws from the Class G&C Program's PM Strategy (CSA, 2013a) and information provided by the CoE to explain in detail the various elements of the program theory.

The Class G&C Program's PM Strategy (CSA, 2013a) does not include a logic model for the Awareness and Learning component of the program because of the decision to no longer support initiatives aimed at elementary and secondary school students.

Figure 1: Logic model for the Research component of the Class G&C Program
Figure 1. Text version below:

Inputs: Inputs are the financial and non-financial resources used to deliver activities, produce outputs, and accomplish outcomes (TBS, ). The Research component of the Class G&C Program's inputs include both financial and human resources (i.e., FTEs).

Activities and Outputs: Activities are the actions that a departmental organization undertakes to produce one or more outputs under the program (TBS, ). Outputs are the direct products or services generated from program activities; they are usually within the control of the program. There are five Class G&C Program activities linked to five outputs for the Research component, including:

  1. Prioritization (A1): This activity is performed inside each branch and not at the CSA level. Each branch, according to its priorities and level of funding, decides what type of activities should be implemented using the Class G&C Program. The prioritization process could be different from one branch to the other, but it usually involves an assessment of the needs of target groups, the governance processes of the branch, discussions at the G&C Advisory Committee, and endorsement by the CSA's Executive Committee or the G&C Committee as part of the annual work plan process.
    • The output of this activity is a ranked list of funding priorities for each branch (Op1).
  2. Promotion and solicitation (A2): This activity is performed at two levels: the promotion of the program itself and the solicitation for specific proposals using Announcement of Opportunities (AO). Unsolicited proposals can also be considered at the own discretion of the CSA and this possibility is mentioned as part of the promotion of the program on the CSA website. Because much of the funding is allocated through competitive process (solicited proposals), the solicitation activity takes on an important role. Based on an AO template developed by the CSA's G&C CoE, each branch is in charge of developing the necessary material for the announcement, although a review is performed by the CoE to ensure standardization. AOs are posted on the CSA website, which provide specific guidelines and forms required for the application.
    • The outputs of the promotion and solicitation activity are the actual AOs, including the proposals received in response to the AOs (Op2).
  3. Evaluation (A3): This activity is performed for all proposals received under an AO or for each unsolicited proposal received at CSA. First, the proposals are screened for eligibility, which usually take the form of yes/no mandatory criteria. Then, each proposal is assessed for compatibility with areas of priority (which is also called the programmatic criteria evaluation). Usually done in parallel to the programmatic evaluation, a scientific and technical point-rated evaluation is performed, either internally at the CSA or using an external peer-review committee.
    • The output of this activity is a ranked list of eligible proposals (Op3). For AOs, this ranked list is included in an evaluation final report that also describes the evaluation process. For unsolicited proposals, a form is used to document the evaluation process and the decision on eligibility of the unsolicited proposal.
  4. Funding (A4): This activity is performed for all proposals that receive a positive decision for funding. Upon a positive decision for funding, a grant or contribution funding agreement is negotiated and signed with the recipient. For grants, disbursement is done at predetermined dates, with one or multiple instalments. For contributions, the disbursement is performed following the reception of a complete claim package. Also, the continued eligibility of recipients is verified annually.
    • The outputs of the funding activity are the funding agreements (for either grants or contributions) and the associated financial assistance (Op4).
  5. Administration (A5): This activity is performed for each funding agreement. The CSA manager receives progress reports, the final report, performance reports and completed surveys. These reports are assessed against the requirements of the funding agreement. Also, this activity includes all other due diligence and administrative processes, such as the periodic review of risks and the assessment that all milestones are met and results are achieved.
    • The output of this activity is the assessment of the progress of the recipient project, including the assessment of the progress and final reports, and of the associated financial assistance (Op5).

Immediate outcomes: Immediate outcomes are those that are directly attributable to the outputs delivered (TBS, ). In terms of timeframe, these are short-term outcomes. The Research component of the Class G&C Program has three immediate outcomes, including:

  1. Increased knowledge from research projects in priority space science and technology (S&T) areas (Oc1): Research activities of interest to Canada related to space science and technology disciplines and their applications will, by their very nature, lead to increased knowledge in those areas. Results of projects will add to the pool of knowledge related to space science, technology, and applications.
  2. Maintained and/or increased space focus in universities, post-secondary institutions, and not-for profit organizations (Oc2): The research conducted in universities, post-secondary institutions, and not-for profit organizations educates students in advanced fields of knowledge. The researchers of tomorrow will most likely gain their first research experience in this setting. Higher education research in the area of space S&T is relatively new and therefore support is expected to help maintain and/or lead to an increase in space focus in these institutions.
  3. Partnerships (Oc3)
    1. Partnerships established and/or sustained: Space science, technology, and applications are most typically undertaken by individual organizations. The Research component places particular emphasis on establishing multidisciplinary or institutional collaboration to undertake space related research and activities.
    2. Partner contribution leveraged: True partnerships are characterized by active participation and/or support (financial, human resource to in-kind). It is expected that recipients will be in a better position to leverage their contribution through their involvement and participation in the Research component.
    3. Access to international collaboration for Canadian organizations: It is expected that Canadian organizations, through their involvement and participation in the Research component as recipients, will gain access to international collaboration that would otherwise not occur.

Intermediate Outcomes: Intermediate outcomes are those that are logically expected to occur once one or more immediate outcomes have been achieved (TBS, ). The Research component of the Class G&C Program has three intermediate outcomes, including:

  1. Increased availability of space-related knowledge and information in priority areas (Oc4): The results of the research projects funded by the program are not in and of themselves end products, but are meant to contribute to and be used for further advancements of knowledge. In order for this to occur, the results need to be made available and disseminated through various means, such as peer-reviewed publications, conference presentations, etc.
  2. Increased space-related S&T capacity in targeted areas (Oc5): All of the immediate outcomes, as described above, will contribute to this intermediate outcome. Not only are the results of individual projects important because they contribute to the pool of space-related knowledge, but the experience, knowledge, and skills gained by conducting the research will help sustain space R&D capabilities. The emphasis on new partnerships and the advent of space-focus in institutions will all contribute to developing and sustaining a critical mass of highly qualified personnel who will be available to respond to future Canadian space-related R&D needs.
  3. Increased multidisciplinary and/or institutional collaboration (Oc6): Beyond the collaborative work that was required to the conduct research or other space-related activities funded by the program, it is expected these new partnerships will lead to linkages and collaborations beyond the scope of the projects funded by the program.

Ultimate outcome: Ultimate outcomes are the highest-level outcomes that can be reasonably and causally attributed to a program as a consequence of one or more intermediate outcomes having been achieved (TBS, ). The Research component of the Class G&C Program has one ultimate outcome, namely:

  1. Canadian space-related R&D responds to national needs and priorities (Oc7): All of the intermediate outcomes are expected to contribute to the same outcome at this level. In the long-term, it is expected that Canada will have the required capacity to conduct space-related R&D, and that there will be sufficiently advanced space-related knowledge and information to respond to national needs and priorities.

1.5 Prior Evaluation of the Program and Performance Measurement

This report represents the first evaluation of the Class G&C Program since its Terms and Conditions were approved by the Treasury Board in . Prior to this date, however, a summative evaluation of the program had been completed in . The evaluation findings were favourable and confirmed the program's relevance in terms of its alignment with Government of Canada priorities and with the CSA's mandate and objectives. The findings also clearly highlighted the need to continue supporting innovation and strengthening human resources in the Canadian space sector (CSA, ). The evaluation report included the following recommendations:

  1. Streamline the twelve components of the Program into three primary components that focus on increasing awareness, human resources, and research capacity.
  2. Broaden the scope of human resources beyond doctoral and post-doctoral researchers, and develop human resources capacity in other functional areas, such as engineering and technology.
  3. Centralize certain administrative aspects of service delivery, such as calls for proposals, records management, and performance reporting.
  4. Improve the PM Strategy.

A management action plan follow-up by the Evaluation function showed that all of these recommendations were fully implemented (CSA, 2010b). Specifically, the Class G&C Program was streamlined into two main components, Research and Awareness and Learning. The Research component was broadened to include all levels of students in various fields of space science and a program implementation framework was developed that included the creation of the G&C CoE. In addition, the program's PM Strategy was developed.

The Class G&C Program's PM Strategy, developed by the CoE in collaboration with program managers, was approved in . The goals of the strategy are to:

  • oversee and measure the results of the program systematically;
  • produce useful, relevant reports for decision-making purposes; and
  • gather credible, reliable information to effectively support program evaluation (CSA, 2013a).

This PM Strategy was revised in . The revised strategy pertains solely to the Research component of the Class G&C Program, pursuant to the CSA decision (made in ) to primarily support funding initiatives under this component.

In , CSA's Audit and Evaluation Directorate audited the Class G&C Program. The audit concluded that the program's PM strategy "was not fully implemented, as some indicators posed a challenge in terms of data collection methods" (CSA, 2013b). This finding was based on CSA's - Annual Report on the State of Performance Measurement (CSA, ). Once a year, in accordance with the Directive on the Evaluation Function (TBS, 2009b), the Evaluation function reviews the state of performance measurement at the CSA and evaluates every PM Strategy to measure its ability to effectively support evaluation.

According to the - annual report on the State of PM strategies, the Class G&C Program's PM strategy is well developed overall, but requires a few modifications (such as adding indicators for outputs). There are some major issues, however, with the PM Strategy's implementation and, more specifically, the lack of accessibility to data collected for performance indicators. In cases where these data are accessible, data cleaning and validation are required before any analysis can be performed (CSA, 2015a).

2. Evaluation Approach and Methods

2.1 Purpose, Evaluation Questions, and Scope

In accordance with the TB Policy on Evaluation (2009a), the purpose of this evaluation is to provide an evidence-based, neutral assessment of the Class G&C Program's value for money, with respect to both the program's continued relevance and performance. Evaluation findings and recommendations aim to support accountability to Parliament and Canadians and to support decision-making regarding managing for results, program improvements, and resource allocation. The intended users of this evaluation include the CSA's directors, managers, and staff who make use of the Program, the CoE, the CSA President and Executive Committee, the Minister of Innovation, Science and Economic Development, Parliamentarians, the Canadian space sector, and the Canadian public.

In keeping with the purpose of the evaluation, the approach adopted in its development was utilization-focused and participatory. Thus, a consultative group comprised of key stakeholder representatives was formed at the onset of the evaluation to offer insight into stakeholders' information needs and to provide guidance and feedback throughout the evaluation process. The consultative group was comprised of 12 manager- and director-level employees working for programs that implement Class G&C initiatives, three CoE personnel (a manager and two staff), one senior executive, one manager from the CSA's Finance Directorate, and one external expert advisor from NSERC, for a total of 18 members. The information needs of the CSA's Vice-President and DGs were also solicited, resulting in evaluation questions that were then grouped according to the five core evaluation issues stipulated by the TB's Directive on the Evaluation Function (2009b):

  1. Continued relevance
    1. Continued need for the program
    2. Alignment with federal and departmental government priorities
    3. Alignment with federal roles and responsibilities
  2. Performance
    1. Achievement of expected outcomes
    2. Demonstration of efficiency and economy

The resources invested in the Class G&C Program, the activities carried out, and the results obtained between and comprise the scope of the evaluation. In keeping with the Class G&C Program's revised PM Strategy (CSA, 2013a), the main focus of the evaluation is on the program's Research component, which represents 98% of program spending since the CSA's program review in and 91% of spending over the course of the evaluation period (see the Resource Allocation section for details). Furthermore, in accordance with the PM Strategy, the evaluation focuses on the Research component of the Class G&C Program as a whole; the relevance and performance of specific funding initiatives are beyond the scope of this evaluation and will be assessed via future evaluations of the CSA programs that issued them, as per the CSA's Departmental Evaluation Plan (2015b).

2.2 Methods

The evaluation design was calibrated to align with the risks associated with the Class G&C Program, which were identified as medium based on an assessment of materialityFootnote 8 (CSA, 2014a). A mixed-methods approach that combines qualitative and quantitative analyses was employed to produce multiple lines of evidence upon which to base evaluation findings and recommendations. The evaluation strategy framework outlines the indicators, methods and data sources, and the responsibility for data collection for each evaluation question, grouped by core evaluation issues. In addition, performance measurement indicators used to for annual reporting in the CSA's Departmental Performance Reports are flagged by those indicators for which baseline data and target yearsFootnote 9 were established.

Specifically, the methods employed included document reviews, archival data reviews, key informant online questionnaires, and telephone interviews with representatives of OGDs.

Document Review

Documents produced both internally (i.e., by the CSA) and externally (e.g., by Parliament and OGDs) were reviewed primarily to evaluate the program's continued relevance, though they also informed the evaluation of program performance in some cases. For a list of documents reviewed, refer to the References section of this report. In addition, a search of the Internet was conducted in order to identify programs similar to the Class G&C Program, with the aim of informing the assessment of the program's performance.

Archival Data Review

Archival data provided by various units within the CSA (including the CoE, Financial Management, Information Technology, Management and Oversight, and program authorities across the CSA who implement Class G&C initiatives) were reviewed to evaluate the Class G&C Program's performance. Because complete G&Cs data were not readily accessible at the onset of the evaluation (see the Limitations section below), they were compiled from various archival sources, including CoE program files, files from CSA programs that have used the Class G&C Program, SAP (the CSA's financial tracking system), and Unitas (a CSA database used in part to track Class G&C performance data and funding recipient contact information). This compilation exercise resulted in the following types of data about the G&Cs funded during the evaluation period:

  • Financial data, including:
    • the timing and financing of G&C agreements,
    • the nature of the agreements (i.e., Research or Awareness or Learning component, grant or contribution, solicited or unsolicited project, links with AOs),
    • the source of funding for agreements (i.e., links with the CSA's PAA), and
    • funding recipient demographics (e.g., type of organization, contact information); and
  • Performance data for projects funded through the Research component, including data pertaining to the achievement of the program's expected outcomes.

As per the Class G&C Program's PM Strategy and G&Cs funding agreements, performance data for funded projects are provided by funding recipients via progress and final reports. These reports are typically requested annually throughout the duration of a funded project and are submitted electronically using a standardized online questionnaireFootnote 10. As shown in Table 6, the report completion rates during the evaluation period were high when calculated in terms of both the percent of projects for which reports were completed (94%) and the percent of reports completed (94%).

Table 6: Number of projects funded via the Research component for which reports were requested, number of reports requested, and completion rates.
Requested Completed
Funded projects 162
83% of the 195Table note 11 funded projects
153 (94% of 162 projects)
Reports 317
average of 1,99 reports per project
297 (94% of 317 reports)
average of 1,94 reports per project

Table notes

Table note 11

In total, 195 projects were funded via the Research component of the Class G&C Program during the evaluation period. The most common reasons why reports were not requested for certain projects were that

  1. they began shortly before the end of the evaluation period or
  2. they began prior to the approval of the Class G&C Program's Terms and Conditions in , when reporting requirements had not yet been established.

Return to table note 11 referrer

As described below in the Limitations section of this report, progress and final report templates were modified over time and multi-year data were not accessible for some of the items contained in these reports.

Key Informant Online Questionnaires

Three groups of key informants were surveyed via online questionnaires in order to evaluate program relevance and performance:

  1. Funding recipients: Principal investigators from academia or senior executives of for-profit organizations that received Class G&C funding via the program's Research component during the course of the evaluation period.
  2. Non-recipients: Principal investigators who applied for Research component funding in response to AOs issued during the course of the evaluation period, but whose proposals were not funded.
  3. CSA employees, including:
    1. program directors and managers responsible for managing CSA programs that make use of the Class G&C Program,
    2. funded project staff (i.e., program, scientific, and technical authorities),
    3. staff and the manager of the CoEFootnote 12,
    4. Financial Management staff, directors, and the Chief Financial Officer, and
    5. CSA senior executives (i.e., DGs, the Vice-President, and a Special Advisor to the President).

In order to better contextualize the evaluation findings, all three groups of key informants were asked to describe any unintended outcomes of the program.

Table 7 presents the number of key informants surveyed and the response rate relative to those solicited for their participation, for each group of key informants. Of note, the samples of funding recipients and non-recipients surveyed via online questionnaires were selected such that:

  • funding recipients were awarded one or more funding agreements during the evaluation period, regardless of whether or not they had also submitted unsuccessful proposals; and
  • non-recipients had submitted one or more proposals that were not funded; they had not been awarded any funding agreements during the evaluation period.

Most frequently (60% of cases), funding recipients who responded to the online questionnaire had been awarded one funding agreement during the evaluation period. However, the number of agreements per funding recipient ranged from one to five, with an average of 1.74 (SD = 1.11) agreements. When completing the online questionnaire, funding recipient were instructed to take into consideration all of the funding they received when responding to the questionnaire items.

The majority (86%) of non-recipients who responded to the online questionnaire had submitted one unsuccessful proposal during the evaluation period and none had submitted more than two proposals, for an average of 1.14 (SD = .36) unsuccessful proposals per non-recipient. Non-recipients were instructed to take into consideration all of their proposals when responding to the questionnaire items.

Table 7: Number of key informants surveyed and response rates, by group of key informants
Group of Key Informants Number of Respondents Response Rate
Funding recipients 57Table note 13 46%
Non-recipients 14Table note 14 32%
CSA employeesTable note 15(total) 53 83%
Program directors, managers, and staff 38 78%
Class G&C Program's Center of Expertise (CoE) 5 100%
Financial Management 5 100%
Senior Executives 5 100%

Table notes

Table note 13

Ninety-five percent of the funding recipients who responded to the online questionnaire were from academic institutions and 5% were from industry, which is representative of the total population of Class G&C Program funding recipients over the course of the evaluation period.

Return to table note 13 referrer

Table note 14

Eighty-six percent of non-recipients were from academia and 14% were from industry.

Return to table note 14 referrer

Table note 15

CSA employees from the CoE and Financial Management were administered a subset of the online questionnaire items, focused on topics with which these employees are most familiar.

Return to table note 15 referrer

Telephone Interview with Representatives from Other Government Departments

An internet search and consultation with the evaluation's consultative group members resulted in the identification of four federal transfer payment programs that are similar to the CSA's Class G&C Program in design and/or focus. Senior staff responsible for these programs, as well as from the Financial Management Sector of TBS, were interviewed via telephone in order to provide points of comparison for the Class G&C program and identify good practices. The list of OGDs interviewed and, where applicable, the relevant transfer payment programs discussed included:

  1. Industry Canada (now referred to as Innovation, Science and Economic Development Canada): Technology Demonstration Program
  2. National Research Council (NRC): NRC Industrial Research Program
  3. Natural Sciences Research and Engineering Council of Canada (NSERC): Industrial Research Chairs program and Collaborative Research and Development program.
  4. Natural Resources Canada (NRCan): NRCan Class G&C Program
  5. TBS's Financial Management Sector, responsible for the Policy on Transfer Payments (TBS, )

Topics discussed during the telephone interviews were initially identified based on a preliminary analysis of emergent themes from the online questionnaires and were subsequently adapted to the context of the specific program being discussed in the interview. The telephone interview guide contained open-ended questions that were administered in a semi-structured interview format.

In several cases, interviewees provided additional documentation pertaining to their programs following the telephone interviews.

Factors to be Considered when Interpreting Results

The following factors pertaining to the analysis and treatment of online questionnaire data should be considered when interpreting the results presented below:

  • Percentages do not always add up to 100 due to number rounding;
  • The valid percent was reported when responses were not provided to online questionnaire items and when items were not included in the online questionnaires (i.e., for certain groups of CSA employees); and
  • Where applicable, quantitative data from online questionnaires were analyzed statistically in order to compare responses provided by funding recipients, non-recipients, and CSA employees, as well as sub-groups of CSA employees. In most cases, only statistically significant differences are reported.
  • Several CSA employees who responded to the online questionnaire indicated that it was too soon for them to provide ratings for some questionnaire items, typically because the Class G&C initiatives they were familiar with were still in early development stages or because they weren't familiar with aspects of the program outside of their responsibilities. In these cases, data pertaining only to those CSA employees who were able to provide ratings are described in the text below, and noted as such.

2.3 Limitations

PM Data Availability: Though the program's PM Strategy (CSA, 2013a) does not include indicators for monitoring program outputs, sufficient archival data were available to compensate for this missing information. Moreover, data were systematically collected for almost all of the indicators identified in the PM Strategy Framework. However, it was not possible to determine the ratio of administrative costs to program expenditures because, with the exception of costs associated with the CoE, financial data regarding the full administrative costs of administering the program across the CSA had not been tracked. TBS recommends that relevant and reliable cost information is used for financial planning, resource allocation, performance reporting, and decision making (TBS, 2009c).

PM Data Accuracy:The Class G&C funding agreement data that was retrieved from the CSA's SAP financial tracking system contained the following shortcomings:

  • Some of the funded projects that spanned more than one year were assigned a different agreement identification number every year, thereby erroneously identifying the same funded project as multiple projects.
  • Accurate start and end dates of funded projects were not available.
  • Funded projects linked to the program's Research component could not be distinguished from those linked to the Awareness and Learning component.

In addition, several inaccuracies were identified in the Unitas data with respect to:

  • the list of projects funded (due to inaccurate software programming that connects Unitas with SAP data, resulting in misidentification of funded projects' links with the CSA's PAA), and
  • the list of progress and final reports both requested and obtained from funding recipients.

For the purpose of this evaluation, these data issues were resolved by cross-referencing diverse sources of archival data and manually entering corrected information into new data files. The development of a more systematic data collection and storage approach is required in order to accurately track G&C data on an ongoing basis.

PM Data Accessibility: At the onset of the evaluation work, the performance measurement data provided by funding recipients in their progress and final reports could not be extracted from Unitas in a manner conducive to program evaluation data analysis. Specifically, data could only be provided for one year at a time and could not be linked to the same funded project's or performance indicator's data collected during other years in the evaluation period, thereby precluding an aggregated, multi-year analysis of the program's performance. Following considerable data extraction efforts, multi-year data were made available for approximately two-thirds of the relevant indicators. The missing data were supplemented by funding recipients' responses to the online questionnaires.

One of the factors that contributed to the challenge of extracting multi-year data was that the progress and final report templates were modified over time to better align with performance measurement needs. In total, seven progress and final report templates (the first of which was produced in Excel rather than in an online format) were used during the course of the evaluation period. Though differences in the template versions were fairly minor, some of the findings presented below are based only on data from recent years in cases where the items contained in the report templates varied over time.

PM Baseline Data and Targets: The program's PM Strategy Framework (CSA, 2013a) identified which fiscal year should be used to establish baseline data for each performance indicator and stipulated that these baselines should also serve as targets for annual performance measurement. In some cases, specific numbers were provided in addition to the fiscal year upon which baselines and targets should be established. However, for the purpose of this evaluation, these numbers were recalculated in order to

  1. account for the progress and final report data that had previously not been entered into Unitas,
  2. account for the inaccuracies contained in Unitas regarding the list of funded projects, and
  3. align with the multi-year manner in which the data drawn upon in this evaluation were analyzed.

Online Questionnaires: The resources required for data collection by the CSA's Evaluation function were calibrated in accordance with the Class G&C Program's medium risk rating for program materiality. Consequently, online questionnaires were conducted to solicit the perspectives of CSA employees, funding recipients, and non-recipients, rather than conducting interviews and onsite visits. This methodology allowed for information to be collected from a large number of key informants, but likely reduced the richness and quality of the information provided.

Interpretation of Findings: The scope of this evaluation pertains to the Class G&C Program as a whole and findings should be interpreted as such. Variances across AOs with respect to the achievement of both program outputs and outcomes are noted in the findings reported below, but an evaluation of the extent to which specific AOs contributed toward the achievement of the program's performance is beyond the scope of this evaluation.

Gender-Based Analysis: No demographic information about applicants and funding recipients was available, precluding a gender-based analysis (as per the Government of Canada's commitment).

Changes to the Program: Following the end of this evaluation's data collection period in , changes to the Class G&C Program were introduced, such as:

  • increasing the budget for funding agreements, the volume of funding agreements, and the proportion of contributions (versus grants),
  • modifying the emphasis placed on the program's target population to broaden the focus on Canadian industry, and
  • moving the location of the CoE's functions in the CSA's PAA from the Future Canadian Space Capacity program to Internal Services.

The impact on program relevance and performance of the changes will be assessed in the next evaluation of the Class G&C Program.

3. Results

This section presents the Class G&C Program evaluation findings, first with respect to program relevance (i.e., the extent to which the program is appropriate to the federal government and addresses the needs of Canadians) and second with respect to program performance (the extent to which effectiveness, efficiency, and economy are achieved by the program).

Codes that identify specific elements to the program logic model are included in the text boxes that introduce each evaluation question and finding below.

3.1 Relevance

The relevance of the Class G&C Program was evaluated with regard to

  1. the linkages between program objectives and federal government priorities,
  2. the linkages between program objectives and departmental strategic outcomes,
  3. the role and responsibilities for the federal government in delivering the program, and
  4. the extent to which the program continues to address a demonstrable need and is responsive to the needs of Canadians.

3.1.1 Alignment with Federal Government Priorities

Evaluation Question #1: Are the program's objectives aligned with federal government priorities? (R1)

Evaluation Finding #1: The objectives of the Class G&C Program are aligned with federal government priorities, including the Government of Canada's Whole-of-Government Framework, the principles and priorities set out in Canada's Science, Technology and Innovation Strategy, Canada's Space Policy Framework, the Economic Action Plan, and the Speech from the Throne and official statements of the Government elected in .

The space and aerospace sectors make critical contributions to Canada's prosperity (Industry Canada, ). According to the Aerospace Review Report commissioned by the federal government, "advancing the national interest through space-based activity and fostering a competitive Canadian space industry will require resolve, clear priorities that are set at the highest levels, and effective plans and programs to translate these priorities into practice" (Industry Canada, 2012a, p. 1). The same report concludes that a focus on developing technological capacity can go a long way toward keeping Canada among the global leaders in space (Industry Canada, 2012a). The Class G&C Program helps address these concerns by supporting targeted knowledge development and innovation to sustain and enhance the Canadian capacity to use space to address national needs and priorities in the future. The program objectives are aligned with federal government priorities as set out in the Government of Canada's Whole-of-Government Framework, the Science, Technology and Innovation Strategy, the Space Policy Framework, the Economic Action Plan, and in the Speech from the Throne and official statements made by the Government that was elected in .

Government of Canada's Whole-of-Government Framework: The Class G&C Program objectives are aligned with the Economic Affairs sector of the spending areas represented in the Whole-of-Government Framework (TBS, ). More specifically, the Class G&C Program is designed to contribute to the Government's expected outcomes to create an innovative and knowledge-based economy (TBS, ).

Canada's Science, Technology and Innovation Strategy: Science, technology, and innovation were the focus of a federal government strategy published in . In this strategy, the Government affirms its intention to partner "with industry and the Canadian space research community to leverage existing resources and encourage further technology development opportunities" (Industry Canada, , p. 51). To this end, the Government will "ensure that our policies and programs inspire and assist Canadians to perform at world-leading levels of scientific and technological excellence" (Industry Canada, , p. 13) and will "continue to support and deepen research across a broad spectrum of disciplines that include both discovery- and application-driven research" (Industry Canada, , p. 36). The Science, Technology and Innovation Strategy has identified five research priorities of strategic importance to Canada:

  1. environment and agriculture,
  2. health and life sciences,
  3. natural resources and energy,
  4. information and communication technologies, and
  5. advanced manufacturing—a priority in which aerospace has been identified as a focus area (Industry Canada, ).

For the Government, the five research priorities and focus areas "address the needs of Canada's key industrial sectors, such as space, robotics [and] aerospace" (Industry Canada, , p. 21).

In addition, the Class G&C Program objectives are aligned with the three pillars of the Government of Canada's Science, Technology and Innovation Strategy: People, Knowledge and Innovation. First and foremost, the program contributes to the strategy's People pillar, through which the Government seeks to develop, attract and retain highly qualified and skilled individuals, as well as top experts and leaders needed for Canada to thrive in the global knowledge economy (Industry Canada, ).

In addition, the program contributes to the Knowledge pillar by fostering the acquisition of specialized skills in areas of Canadian strength, such as advanced space optics, robotics, and the development of innovative instruments. Through the Knowledge pillar, the Government seeks to support "research and scientific capacity in universities, colleges and polytechnics" (Industry Canada, , p. 2). The Government aims to strengthen its "support for excellence across the spectrum of discovery-driven and applied activities by investing in research and infrastructure" (Industry Canada, , p. 14).

Lastly, the strategy's third pillar, Innovation, aims to "encourage greater partnerships among Canadian businesses, universities and colleges to drive innovation and encourage the adoption of new processes and technologies that help Canadian businesses prepare to compete and win in the global marketplace" (Industry Canada, , p. iii). The Class G&C Program is aligned with the Innovation pillar in that it supports the development or maintenance of partnerships between universities, government, and industry, and it increases international collaboration opportunities for Canadian scientists.

Canada's Space Policy Framework: Published in , Canada's Space Policy Framework is central to the federal government's approach to space. One of the guiding principles outlined in the framework is "Excellence in Key Capabilities," for which the Government highlights that it will "continue to support and advance proven Canadian competencies while keeping a close watch on new niches of technological accomplishment" (CSA, 2014b, p. 10). Within Canada's Space Policy Framework, the Government has also identified four avenues of strategic action, including R&D. To this end, the federal government is committed to working with industry and the Canadian space research community, and to encouraging further opportunities in R&D and innovation by "increasing support for technology development" (CSA, 2014b, p. 11). The Class G&C Program helps the federal government respond to its priority of encouraging further opportunities in R&D and innovation by offering organizations financial aid, which they can use to carry out space R&D activities in priority areas for the CSA.

Economic Action Plan: Further to the Aerospace Review, chaired by the Honourable David Emerson, the Government announced in Economic Action Plan and over subsequent months several measures to support the aerospace sector. The measures included launching the Consortium for Aerospace Research and Innovation in Canada to facilitate collaboration among aerospace stakeholders (industry, academia, and research centres) to foster the development of advanced technologies, support the training of highly qualified personnel, and support innovation outreach (Department of Finance Canada, ). In Economic Action Plan , the Government of Canada announced its intention to continue to support the pillars of a knowledge-based economy (Department of Finance Canada, ). According to the Government, "as the economic opportunities stemming from the application of new knowledge increase, it is crucial for Canada to possess a strong science and technology base, robust research infrastructure and a highly qualified workforce" (Department of Finance Canada, , p. 90). The Class G&C Program objectives of supporting the development of S&T relevant to CSA priorities and fostering the continuing development of a critical mass of researchers and highly qualified personnel (HQP) in areas relevant to CSA priorities contribute to the fulfillment of this Government priority.

Speech from the Throne and official statements of the Government elected in : After the data gathering period for the Class G&C Program evaluation, the Speech from the Throne was given by the newly elected government. The Throne Speech outlines the policies that the Government is seeking to implement in the months to follow. In the speech, the Government recognized that public investment is needed to create and support economic growth, job creation, and economic prosperity (Government of Canada, 2015a). In this vein, the Minister of Innovation, Science and Economic Development, the Minister of Science, and the Minister of Small Business and Tourism declared, in a joint message to CSA employees, that their overarching objectives will be to promote and support scientific research and the integration of scientific considerations in their investment and policy choices (Message from ministers to CSA employees, ). Finally, in the Minister of Science Mandate Letter, the Prime Minister affirms, "We believe that investments in scientific research, including an appropriate balance between fundamental research to support new discoveries and the commercialization of ideas, will lead to good jobs and sustainable economic growth" (Government of Canada, 2015b). The Class G&C Program contributes to these priorities by funding research organizations, for-profit organizations, and, to a lesser extent, non-profit organizations, with a view to developing knowledge in areas of S&T that are a priority for the CSA.

3.1.2 Alignment with Departmental Priorities

Evaluation Question #2: Are the program objectives aligned with departmental strategic outcomes? (R2)

Evaluation Finding #2:The Class G&C Program objectives are aligned with the CSA's priorities and strategic outcomes, particularly with respect to the program's Research component. Alignment with the program's Awareness and Learning component is less evident, given the decision made in to no longer fund initiatives aimed at elementary and secondary school students.

According to the CSA's - Report on Plans and Priorities, the Canadian space industry remains reliant on continued R&D investments to overcome its growth challenges (CSA, 2014c). In addition, "Canada's ability to address national interests through an increasing dependence on space requires a critical mass of HQP in academia, industry and government; therefore, it is vital to develop, mobilize, and retain experts in space and related fields" (CSA, 2014c, p. 10). The Class G&C Program addresses this concern of the CSA's by aiming to increase the number of HQP participating in space S&T initiatives and projects.

In addition, the Class G&C Program supports the CSA's strategic priorities of consolidating CSA's technology development and capability demonstration activities and continuing the implementation of Canada's Space Policy Framework (CSA, 2015c).

The program also contributes to the CSA strategic outcome, which states, "Canada's exploration of space, provision of space services and development of its space capacity meet the nation's needs for scientific knowledge, innovation and information" (CSA, 2015c, p. 5).

As described in the Evaluation Approach and Methods section above, the main focus of the current evaluation was on the program's Research component, which represents 91% of program spending over the course of the evaluation period and 98% of program spending since . As shown in the Resource Allocation section above, only $320K were spent on the Class G&C Program's Awareness and Learning component since , when a decision was made to no longer fund initiatives aimed at elementary and secondary school students. Therefore, the alignment between the Awareness and Learning component and the CSA's priorities is tenuous.

3.1.3 Alignment with Federal Roles and Responsibilities

Evaluation Question #3: Is the program consistent with federal roles and responsibilities? (R3)

Evaluation Finding #3: The federal government's participation in space research and innovation through the Class G&C Program is aligned with federal roles and responsibilities, as described in the Canadian Space Agency Act.

According to the Canadian Space Agency Act (the Act), the CSA mandate is to:

"...promote the peaceful use and development of space, to advance the knowledge of space through science and to ensure that space science and technology provide social and economic benefits for Canadians" (Canadian Space Agency Act, , s. 4).

The CSA is delivering on its mandate in collaboration with Canadian industry, academia, Government of Canada organizations, and other international space agencies and organizations (CSA, 2015c). The activities and objectives of the Class G&C Program are aligned with one of the CSA's main functions, which, as set out by the Act, is to "plan, direct, manage and implement programs and projects relating to scientific or industrial space research and development and the application of space technology" (Canadian Space Agency Act, , par. 5(2)(b)). The program is also aligned with paragraph 5(3)(c) of the Act, which states:

"In carrying out its objects, the Agency may...make grants and contributions in support of programs or projects relating to scientific or industrial space research and development and the application of space technology, including projects designed to develop, test, evaluate or apply new or improved processes, products, systems or information relating to space science and technology with a view to determining the commercial potential of that science and technology, but not including any programs or projects relating solely to the commercial exploitation of space science or technology."

According to this provision, the CSA has a major role to play in space R&D and is authorized to implement or support initiatives to collect data, conduct space research and studies, and maintain a critical mass of HQP in the field of space.

Furthermore, the Aerospace Review report highlighted that "space-related public investments are essential for the achievement of fundamental principles of...advancing the development of new technologies, and pushing the boundaries of knowledge" (Industry Canada, 2012a, p. 5). Numerous studies have also demonstrated how important it is for governments to be involved in the initial funding of space R&D (Industry Canada, 2012a; Industry Canada, 2012b; Library of Parliament, ; Organization for Economic Cooperation and Development [OECD], ; OECD, ). In a paper on the federal government's role in the Canadian aerospace industry, the Library of Parliament concluded that "the very long research and product development cycles, combined with the incremental nature of innovation, make early stage funding for R&D—provided in part through government assistance—particularly important" (, p. 3). From this perspective, the financial assistance provided to space organizations through the Class G&C Program is aligned with federal roles and responsibilities.

3.1.4 Continued Need for the Program

Evaluation Question #4: Does the program continue to address a demonstrable need and is it responsive to the needs of Canadians? (R4)

Evaluation Finding #4: The Class G&C Program addresses three main needs:

  1. the need for capacity development in the Canadian space sector,
  2. the need to support collaboration between industry, government, academia, and international partners, and
  3. the need to support funding recipients' activities that contribute to the fulfillment of the CSA's mandate and priorities.

However, some funding recipients identified the need for CSA to ensure continuity in domains targeted by AOs, as well as the need for continuity in research funds to develop projects to a higher level of maturity.

Several data sources were drawn upon, including information gathered through a document review and an online questionnaire for key informants (CSA employees, as well as recipients and non-recipients of Class G&C Program funding), to determine whether the Class G&C Program continues to address a demonstrable need and whether it is responsive to the needs of Canadians. As shown in Figure 2, the majority of key informants (98% of funding recipients, 71% of non-recipients, and 89% of CSA employees) felt that the Class G&C Program continues to address a demonstrable need.

When asked to explain their answers, more than half of funding recipients as well as some non-recipients indicated that the Class G&C Program is the only federal program entirely dedicated to the development of the space sector. Although there are other programs that offer G&Cs to the space and aerospace sectors, and there may be some overlap, none of those other programs designates space research as a priority. Similarly, the evaluation of the Class G&C Program also noted that "[t]he CSA is the only organization that is poised to offer focused support to stakeholders in achieving [their] objectives. The Class G&C Program provides the financial support in areas in which other sources of funding do not exist" (CSA, , p. 9). This point will be further detailed in the performance (efficiency and economy) section of this report.

Figure 2: Key informants' opinions regarding the extent to which there is a continued need for the Class G and C Program (%).
Recipients Non-Recipients CSA Employees
don't know   14% 6%
extremely 82% 64% 56%
very 16% 7% 33%
moderately 2% 7% 2%
slightly   7% 2%
not at all      

More specifically, when key informants were asked which needs the Class G&C Program addresses, they identified three main needs, which are also supported by the information that was gathered in the document review:

  1. the need for capacity development in the Canadian space sector;
  2. the need to support collaboration between industry, government, academia, and international partners; and
  3. the need to support funding recipients' activities that contribute to the fulfillment of the CSA's mandate and priorities.

The need for capacity development in the Canadian space sector: The first Summative Evaluation of the Class Grant and Contribution Program (CSA, ) identified the need for capacity development in the Canadian space sector. The evaluation concluded that "because of the small size of the Canadian space sector, the critical mass to achieve broadly based innovative capacity is difficult to attain within academia and industry without the explicit support and assistance of the government" (p. 9). According to the report, the program played a role in addressing this need by coordinating between different actors and through funding. In addition, according to a OECD report, "[t]he acquisition and development of space capabilities remains a highly attractive strategic goal, and the number of countries and companies investing in space systems and their downstream applications continues to grow" (OECD, , p. 9).

Of the three needs identified by key informants, the need for capacity development in the Canadian space sector was the most frequently cited. This need can be divided into three sub-categories:

  1. The number of HQP, defined as individuals with university degrees at the bachelor's level and above;
  2. The extent of space-related expertise; and
  3. The availability of useful materials and equipment (e.g., laboratory equipment) for conducting space-related research.

When asked what gaps would exist in the absence of the Class G&C Program, most funding recipients, half of CSA employees, and a few non-recipients stated that there would be a decrease in Canadian capacity and skills in the space sector. They said that, without the Class G&C Program, there would be a lack of resources to support the development of new technologies or to maintain a highly qualified and skilled workforce.

Several non-recipients, some funding recipients, and some CSA employees also mentioned the need for the Class G&C Program to be more attentive to the needs and realities of the Canadian and international space community. These key informants explained that because of the changing global context, when determining project funding priorities, the CSA should give greater consideration to opportunities available to academia and industry, and the challenges these sectors face.

More specifically, funding recipients offered two suggestions to be more responsive to the needs of Canadians. First, the CSA should ensure continuity in the domains targeted by Class G&C Program's AOs, rather than shifting priorities in the projects it decides to fund. In their view, some AOs should be renewed to afford recipients greater opportunities to continue their research activities. Second, funding recipients suggested that the CSA continue funding research projects to develop higher levels of technological maturity. They explained that funding is usually set aside for early-stage R&D projects rather than for the development of more mature technologies.

Lastly, some CSA employees stated that, over the years, the Future Canadian Space Capacity program (PAA 1.3) has benefited greatly from the Class G&C Program, while other CSA programs (Space Data, Information and Services [PAA 1.1] and Space Exploration [PAA 1.2]) have not made sufficient use of it. They explained that the Class G&C Program was designed to support the achievement of priorities across the CSA's programs and that, to be more responsive to the Canadian space community's needs, it should be utilized more extensively by all CSA programs. According to key informants, this approach would support R&D and knowledge gain in a broader variety of space-related domains aligned with CSA priorities.

The need to support collaboration between industry, government, academia, and international partners: In the Space Policy Framework, the Government of Canada recognizes that space is a competitive domain and an expensive undertaking. The Government therefore affirms that it will look to continue partnerships to share the expenses and rewards of major space initiatives (CSA, 2014b, p. 10). Access to multidisciplinary and international collaboration opportunities for Canadian organizations is one of the Class G&C Program's expected outcomes. Several funding recipients as well as some CSA employees and non-recipients indicated that the Class G&C Program addresses the need to support collaboration between industry, government, academia, and international partners.

Some funding recipients and some non-recipients also said that the absence of the Class G&C Program would have repercussions on the capacity of Canadian industry and academia to create national and international collaboration opportunities. They explained that the program allows Canadian researchers to participate in space-related activities and disciplines as well as international research programs.

The need to support funding recipients' activities that contribute to the fulfillment of the CSA's mandate and priorities: This need was primarily cited by nearly a third of CSA employees, who said that G&Cs broaden the range of available funding tools. These CSA employees, as well as some funding recipients stated that the Class G&C Program addresses the need to support activities that contribute to the fulfillment of the CSA's mandate and priorities. Some employees explained that the CSA's financial support for R&D work is critical to the achievement of CSA objectives.

Similarly, some CSA employees specified that a number of pertinent projects funded for the fulfillment of the CSA's mandate and priorities could not be sustained without the Class G&C Program. They said that contracts are not a suitable alternative for funding initiatives geared toward knowledge and skills development.

3.2 Performance

The Class G&C Program's performance was evaluated with regard to

  1. its progress toward producing outputs and achieving expected outcomes, and
  2. its resource utilization in relation to the production of outputs and progress toward expected outcomes.

The evaluation of these core performance issues are conducted sequentially below.

3.2.1 Achievement of Expected Outputs and Outcomes

The extent to which the Class G&C Program has achieved each of the outputs and outcomes identified in the logic model is evaluated in this section of the report, which is divided into output, immediate outcome, intermediate outcome, and ultimate outcome sub-sections.

3.2.1.1 Outputs

Evaluation Question #5: Have ranked lists of funding priorities been produced? (Op1)

Evaluation Finding #5: Ranked lists of funding priorities for each branch have not been produced and some key informants described, more generally, a lack of clearly articulated, CSA-wide priorities. Evidence suggests that alignment with CSA funding priorities is more pronounced for AO than for unsolicited funding mechanisms.

Information provided by this evaluation's consultative group indicated that ranked lists of funding priorities have not been produced by the branches. Therefore, it was not possible to definitively assess whether the funding mechanisms in place are aligned with each branch's priorities. However, evidence suggests that the AO funding mechanism is better aligned with CSA priorities than is the unsolicited funding mechanism.

Figure 3: CSA employees' opinions regarding the extent to which the selection mechanisms in place ensure that funded projects are aligned with the CSA's priorities (%).
Solicited Projects Unsolicited Projects
don't know 8% 27%
extremely 31% 19%
considerably 44% 19%
moderately 17% 23%
slightly   13%
not at all    

By its very nature, the AO funding mechanism is designed to solicit proposals in DG-approved priority areas. Furthermore, alignment between CSA priorities and solicited proposals is one of the AO proposal selection criteria. As shown in Figure 3, the majority of CSA employees (75%) reported considerable alignment between the AO selection mechanism and the CSA's priorities.

However, others expressed less confidence in this alignment, primarily due to unclear and/or shifting CSA priorities. These employees highlighted the need for CSA-wide strategic planning that takes into account the Canadian space community's needs and is manifested in clear and stable R&D funding priorities.

CSA employees were significantly more likely to report alignment with CSA priorities for AO than for unsolicited funding mechanismsFootnote 16. Most CSA employees either reported slight-to-moderate alignment between the unsolicited funding mechanism and CSA priorities (35%) or indicated that they don't know whether there is alignment (27%; Figure 3), suggesting that alignment is more pronounced for AO than for unsolicited funding mechanisms. Of note, there were no standardized selection criteria in place for unsolicited proposal during the evaluation period.

Evaluation Question #6: Have announcements been made available? (Op2)

Evaluation Finding #6: Ten AOs were made available over the course of the evaluation period, the majority of which were launched by the CSA's Future Canadian Space Capacity program (PAA 1.3). Though most CSA employees reported that effective funding application mechanisms are in place, CSA employees, funding recipients, and non-recipients alike reported challenges stemming from the infrequent and unpredictable timing of AOs. In addition, several funding recipients and non-recipients indicated that the funding application process is arduous and complex, and that there were lengthy delays in announcing the selection results with little communication from the CSA. Despite these concerns, a sizable minority of funding applicants rated the application process favourably. Discrepant perspectives are likely due to variances in the application processes across the ten AOs.

A total of ten AOs were launched over the course of the evaluation period. Among these AOs, five pertained to the CSA's Future Canadian Space Capacity program (PAA 1.3), three pertained to Space Exploration (PAA 1.2), and two pertained to Space Data, Information and Services (PAA 1.1).

In order to gauge the overall quality of the funding mechanisms in place, key informants were asked to compare the Class G&C Program's application process with OGDs' funding application processes. As shown in Figure 4, perspectives varied considerably across the three main key informant groups. Though most CSA employees didn't know enough about OGDs' funding mechanisms to comment, those who did most frequently reported that the application process is comparable to others for both solicited (65%) and unsolicited proposals (58%). In contrast, non-recipients most frequently rated the Class G&C Program's application process as inferior to OGDs' programs (71%). Funding recipients' ratings were more favourable than those of non-recipientsFootnote 17, but fairly evenly split with 25% rating the Class G&C Program's application process as superior, 38% rating it as about the same, and 32% rating it as inferior to OGDs' application process.

Figure 4: Key informants' ratings of the Class G and C Program's funding application process relative to those other federal departments (%).
Recipients Non-Recipients CSA employees
CSA employees -
Solicited proposals
CSA employees -
Unsolicited proposals
don't know 5% 7% 58% 75%
much better 5%   2%  
better 20%   4% 2%
about the same 38% 21% 27% 15%
worse 27% 36% 6% 2%
much worse 5% 36% 2% 6%

To gain greater insight into the quality of the Class G&C Program's funding application process, key informants were asked to provide feedback on the AO solicitation process, as well as on the proposal submission processes for both solicited and unsolicited proposals. Though variances across AOs and unsolicited proposals likely contributed to discrepancies in key informants' perspectives, the group data reported below nonetheless provide an overview of the program's application process strengths and weaknesses.

AO Solicitation Process

According to most CSA employees (73%), effective mechanisms are in place for obtaining AO funding applications. However, several CSA employees pointed out that the infrequency and unpredictability of AOs pose challenges for the Canadian space community in terms of planning for and maintaining HQP and infrastructure capabilities. This sentiment was echoed by both funding recipients and non-recipients, several of whom recommended replicating the approach used by other departments of establishing fixed, annual solicitation dates. According to some funding recipients, introducing regular and predictable timing for AOs would not only facilitate and improve the effectiveness of their internal planning, but would also reduce the "time crunch" that some experienced in the past due to short timelines between the launch of an AO and the application deadline.

Though it is not mandatory to standardize the timing of solicitation processes, interviews conducted with OGDs suggest that such standardization can be beneficial both in terms of providing applicants with predictability and in terms of managing forecasted spending within the granting department. However, such standardization may not be aligned with the needs of the CSA if, for example, funding priorities and/or budgets vary considerably from one year to the next. If it is not possible to establish fixed AO solicitation dates then key informants from TBS recommended informing applicants of why a non-standardized approach has been selected.

Because the Class G&C Program does not have standardized solicitation dates, a need for broader dissemination was identified. Given that departments are required to engage their potential applicants, key informants from TBS suggested strategies like email lists, twitter and social media, and notices in journals for engendering greater awareness of funding opportunities. In addition, some OGD interviewees use more personal approaches, like promoting funding opportunities at sector-specific events, society meetings, and trade shows.

Figure 5. Text version below:

Figure 5: Number of calendar days between AO closing date and agreement start date.

Figure 5 - Text version

Histogram, a graph depicted by bars, displaying the frequency distribution of the calendar days that elapsed from the AO closing date to the agreement start date for 63 agreements. The distribution of the bars varies with three well-defined peaks displayed at the interval 75-100 days, 175-200 days, and 275-300 days. A bell curved line passes through the bars, and peaks at the point where the mean (192.33 days) occurs.

In conjunction with the absence of standardized solicitation dates, the service delivery timelines of AO administration varied considerably among the AOs launched during the evaluation period. For example, the number of days between the AO application due date and the funding agreement start date ranged from 33 to 468 days, as shown in Figure 5.

AO funding recipients perceived that the CSA responded to their funding application in a timelier manner than did non-recipientsFootnote 18 (see frequencies in Figure 6) and some funding recipients expressed appreciation for the speed with which funding decisions were made by the CSA. However, some respondents from both groups explained that the length of time between the submission deadline and the announcement of the results was too lengthy (more than one year in some cases).

In addition to lengthy delays, some recipients and non-recipients indicated that they did not receive communications from the CSA during the waiting period, which created considerable planning challenges for applicants. Furthermore, two of the 14 non-recipient respondents (who had applied to different AOs) reported that they never received notification from the CSA that their proposal had been rejected.

Figure 6: Opinions of funding recipients who submitted solicited proposals and of non-recipients regarding the extent to which the CSA responded to their application in a timely manner (%).
Recipients Non-Recipients
don't know 5% 8%
extremely 12%  
considerably 35% 8%
moderately 30% 8%
slightly 14% 31%
not at all 5% 46%

Proposal Submission Process

Only a minority of both funding recipients and non-recipients indicated that it was easy to find information about the Class G&C Program on the CSA website (39% of recipients and 21% of non-recipients) and that this information was clear (43% of recipients and 14% of non-recipients), though recipients indicated significantly greater satisfaction in both casesFootnote 19. In addition, some funding recipients (30%) and non-recipients (21%) reported that the level of effort required to complete the funding application was reasonable (Figure 7). However, these data also indicate that most recipients and non-recipients thought the application process could be improved upon.

Figure 7: Funding recipients and non-recipients' opinions regarding the extent to which the level of effort required to complete the funding application was reasonable (%).
Recipients Non-Recipients
don't know   7%
extremely 4% 14%
very 26% 7%
moderately 54% 29%
slightly 9% 14%
not at all 7% 29%

In explaining their responses, almost all of the funding recipients and non-recipients who rated the application process less favourably wrote that the process is arduous and complex – a concern also raised by a few CSA employees. Some respondents explained that the level of effort required to complete the proposal is too high, especially for small G&Cs. However, respondents most frequently explained that there is considerable redundancy within the application process, whereby the same information is requested in different formats across the various sections of the proposal. Other frequently mentioned concerns regarding the cumbersomeness of the application process included that the requested information and the format of the application are inconsistent from one AO to the next, and that the proposal and verification signatures cannot be submitted electronically. In order to lighten the application process, both CSA employees and funding recipients recommended that the CSA introduce a streamlined online application form that does not vary extensively across funding opportunities. CSA employees added that electronic submissions would also improve the efficiency of the proposal selection process.

Though guidelines have been developed regarding the Class G&C Program's application process for solicited and unsolicited proposals, they have not been applied systematically. According to the OGD interviews carried out for this evaluation, TBS recommends an "enterprise-wide" approach whereby both the application process and the application guidelines are standardized (as well as harmonization across government departments, if possible). In line with this recommendation, most of the interviewed OGDs who provide grants or contributions indicated that they have standardized application guidelines, and all have standardized proposal templates for solicited and/or unsolicited proposals.

With respect to the proposal submission format, TBS recommends automation not only for electronic submissions but also for tracking funding recipients across applications. NRC's "Client-Relationship Management" system is a laudable example of such automation, in that their secure log-in, electronic system entails a centralized database that generates and tracks all aspects of the funding process, such as online proposal submissions, approvals, agreements, claims, support documents, reporting and performance reporting, and direct deposit of funding payments.

Evaluation Question #7: Have ranked lists of eligible applicants been produced? (Op3)

Evaluation Finding #7: Ranked lists of eligible applicants were produced for each AO. The selection mechanisms used to create these lists were deemed fair by most CSA employees and funding recipients, but not by non-recipients. All three groups of key informants rated the AO selection process as significantly less transparent than fair. In particular, several funding recipients and non-recipients reported that they did not receive constructive feedback about their proposals. With respect to unsolicited proposals, numerous CSA employees reported concerns with both the fairness and the transparency of the selection process.

A document review identified ranked lists of eligible applicants for each of the ten AOs launched during the evaluation period.

To partially assess quality, CSA employees were asked to compare the Class G&C Program's selection process with that of OGDsFootnote 20. However, many CSA employees reported not knowing enough about OGD selection processes to comment, especially with respect to the selection of unsolicited proposals. For solicited proposals, most (76%) CSA employees who provided ratings reported that the Class G&C Program's selection process is either equivalent or superior to that of OGDs.

Quality was also examined with respect to key informants' perceptions of the fairness and transparency of the selection process. According to CSA employees, the process for selecting both solicited and unsolicited proposals is significantly more fair than it is transparentFootnote 21.

Despite this discrepancy, the majority of CSA employees rated the solicited proposals selection process as both fair (88%) and transparent (77%; Figure 8). In their open-ended responses, several employees commented on the high quality of the AO selection process and attributed its success to the guidance of the CoE. The only suggestion for improvement provided was to use peer or expert-reviews in the selection processes more extensively.

Figure 8: CSA employees' opinions regarding the extent of fairness and transparency of the selection process (%).
Solicited proposals Unsolicited proposals
Solicited proposals -
Fair
Solicited proposals -
Transparent
Unsolicited proposals -
Fair
Unsolicited proposals -
Transparent
don't know 4% 4% 27% 27%
extremely 46% 40% 10% 10%
very 42% 38% 13% 6%
moderately 6% 17% 29% 23%
slightly 2% 2% 10% 17%
not at all     10% 17%

In sharp contrast to the solicited proposal selection process, only a small portion of CSA employees rated the unsolicited proposal selection process as fair (23%) and transparent (17%; Figure 8). Many CSA employees explained that the selection criteria are not well developed and documented. In order to improve the fairness and transparency of unsolicited proposals, CSA employees suggested:

  • Decreasing the proportion of G&C funding spent on unsolicited proposals by increasing the frequency of AOs and introducing AOs with broad domains and fixed application dates;
  • Introducing standardized application guidelines and selection criteria for unsolicited proposals; and
  • Disseminating information on the CSA website about the unsolicited proposal selection process.

Funding recipients rated the CSA's process for selecting funding recipients as fairer and more transparent than did non-recipientsFootnote 22. Also, like CSA employees, both funding recipients and non-recipients indicated that the selection process is significantly more fair than it is transparentFootnote 23 (see frequencies in Figure 9).

Figure 9: Funding recipients and non-recipients' opinions regarding the extent of fairness and transparency of the selection process (%).
Recipients Non-Recipients
Recipients -
Fair
Recipients -
Transparent
Non-Recipients -
Fair
Non-Recipients -
Transparent
don't know 14% 7% 29% 8%
extremely 13% 9%    
very 45% 31% 14%  
moderately 27% 31% 7% 23%
slightly 2% 9% 21% 15%
not at all   13% 29% 54%

In terms of fairness, a few funding recipients and non-recipients reported that some AOs appeared to be geared towards the selection of large organizations with ample prior experience. In terms of transparency, numerous recipients and non-recipients expressed concerns about the lack of information provided. Specifically, these respondents explained that

  1. the selection process was not clearly articulated,
  2. they received little-to-no feedback on their proposals, and
  3. the feedback they did receive was not constructive.

Though the type and extent of feedback about unsuccessful proposals varied across AOs, key informants from TBS recommended establishing service standards for feedback whereby unsuccessful applicants are informed of the reasons why their proposals were not accepted. For example, Industry Canada and NRC provide both written feedback and feedback communicated via telephone conversations that include recommendations for strengthening unsuccessful proposals.

Evaluation Question #8: Have G&C agreements been produced and has financial assistance been provided? (Op4)

Evaluation Finding #8: $36M were awarded across 195 G&C agreements pertaining to the Research component of the Class G&C Program during the evaluation period. Half of this funding was awarded to solicited proposals and the other half was awarded to unsolicited proposals, primarily by the CSA's Future Canadian Space Capacity program (PAA 1.3). The vast majority of Research component funding was awarded to universities.

As shown in the Resource Allocation section of this report, $40M in G&Cs were awarded by the Class G&C Program during the course of the evaluation period. The vast majority (91%, $36M) of this funding pertained to the program's Research component, which constitutes the main focus of this evaluation, and 76% of the Research component funds were awarded by the CSA's Future Canadian Space Capacity program (PAA 1.3; Figure 10).

Figure 10: Research component spending by CSA Program, not including smaller than 1% awarded by Internal Services.
Space Data, Information and Services Space Exploration Future Canadian Space Capacity
15% 8% 76%
Figure 11: Percent of Research component funding spent on solicited and unsolicited projects
Solicited projects Unsolicited projects
55% 45%

As per Figure 11, approximately half (55%; $20M) of the Research component's funding was awarded to solicited projects via ten AOsFootnote 24 (eight AOs launched during the evaluation period and two AOs launched prior to its commencement). Like the Research component as a whole, most funding for both solicited (95%) and unsolicited projects (54%) stemmed from the CSA's Future Canadian Space Capacity program (PAA 1.3).

The $36M spent on the Research component was used to fund 195 projects (19% of which had start or end dates extending beyond the evaluation period). Of these 195 funded projects:

Figure 12: Research component G and C agreements by type of recipient organization.
Universities Private companies Not-for-profit organizations
95% 4% 1%
  • 185 (95%) were awarded to universities (Figure 12), representing 98% of the Research component funding spent during the evaluation period;
  • 117 (60%) were solicited, awarded an average of $182K per project; and
  • 78 (40%) were unsolicited, awarded an average of $228K per project (or an average of 185K per project, if the $3.6M contribution mentioned in Footnote #14 is treated as a data outlier and omitted from the calculation).

Though the range of funding awarded per project was wide, it exceeded $400K for only 14% of solicited projects (Figure 13) and 13% of unsolicited projects (Figure 14).

Figure 13. Text version below:

Figure 13: Total funding awarded ($) per solicited projects

Figure 13 - Text version

Histogram, a graph depicted by bars, displaying the frequency distribution of the awarded amount per project for 117 solicited projects. The distribution of the bars is concentrated on the left side of the graph with two well-defined peaks at the interval $0-50K and $175-200K, and with one low frequency outlier (small bar at the complete right (the tail end of the histogram at interval $750K-$800K). A bell curved line passes through the bars, and peaks at the point where the mean ($181,729) occurs.

Figure 14. Text version below:

Figure 14: Total funding awarded ($) per unsolicited project

Figure 14 - Text version

Histogram, a graph depicted by bars, displaying the frequency distribution of the awarded amount per project for 78 unsolicited projects. The distribution of three bars in descending order of height is concentrated on the left side of the graph with one distinct peak occurring at $0K-33K interval, and with one very small outlier bar at the complete right (the tail end of the histogram at interval $333K-366K). A bell curved line passes through the bars, and peaks at the point where the mean ($228,403) occurs.

As described in the Methods section above, most of the funding recipients who responded to this evaluation's online questionnaire (60%) received funding for one project during the evaluation period, though the range of funded projects per recipient was from one to five (Figure 15). In total, 51% of funding recipients received funding for solicited proposals, 25% received funding for unsolicited proposals, and the remaining 25% received funding for a combination of solicited and unsolicited proposals.

Figure 15. Text version below:

Figure 15: Number of funded projects per recipients

Figure 15 - Text version

Histogram, a graph depicted by bars, displaying the frequency distribution of the number of funded projects awarded per recipient for 57 recipients. There are five bars descending in order of height and a bell curved line passes through the bars, and peaks at the point where the mean (1.74 projects) occurs.

Evaluation Question #9: Have assessments of progress and final reports been conducted? (Op5)

Evaluation Finding #9: The vast majority of requested progress and final reports were completed. According to key informants, the reporting requirements are similar to those of OGDs, though over half of funding recipients indicated that the level of effort required to complete progress and final reports was too high.

As described in the Methods section of this report, 297 progress and final reports were completed for projects funded via the Research component, representing 94% of requested reports and 94% of funded projects for which reports were requested. On average, funding recipients completed 1.94 reports per funded project.

When asked how the Class G&C Program's reporting requirements compare with those of OGDs, funding recipients and CSA employees who provided ratings most commonly indicated that the reporting requirements are fairly similar to others (Figure 16). However, most CSA employees did not know how the reporting requirements compare with those of OGDs and therefore did not provide a rating.

Figure 16: Funding recipients and CSA employees' ratings of the Class G and C Program's reporting requirements, relative to those of other federal departments (%).
Recipients CSA employees
CSA employees -
Solicited projects
CSA employees -
Unsolicited projects
don't know 5% 64% 75%
much better 5% 4% 4%
better 12% 4% 2%
about the same 47% 23% 15%
worse 25% 4% 2%
much worse 5%   2%

Funding recipients' perspectives regarding the level of effort required to complete the reporting requirements were divided, with 42% reporting that the level of effort required is very reasonable and the remaining 58% reporting that it is only slightly or moderately reasonable. In explaining their answers, funding recipients voiced discordant perspectives regarding whether or not the degree of reporting is extensive, the information requested is relevant, and the online reporting format is user-friendly.

Though CSA employees were not explicitly asked to comment on the content of the reporting requirements, a few spontaneously mentioned that some of the indicators assessed in progress and final reports (e.g., space S&T capacity building and knowledge dissemination) are not relevant to funding recipients from industry. Similar concerns were not mentioned by funding recipients, though very few recipients were from industry during the evaluation period.

To improve the reporting process, a few funding recipients suggested replacing hard-copy verification signatures with electronic ones. In addition, it was suggested that a follow-up reporting requirement be introduced to capture outcomes that only occur after funded projects are completed. "The metrics that are asked for (papers, talks, etc.) are all things that tend to come to fruition at the very end (or even after) the projects are complete," explained one funding recipients. Though the utility of follow-up reporting would likely be influenced by size and scope of funded projects, it is noteworthy that both NRC and Industry Canada request follow-up reports for up to five years after funded projects are completed.

3.2.1.2 Immediate Outcomes

Evaluation Question #10: Has knowledge been increased from research projects in priority space S&T areas? (Oc1)

Evaluation Finding #10: During the evaluation period, ten AOs were launched and 195 projects were funded by the Research component, with the aim of increasing space S&T knowledge. The extent of knowledge gained by funded projects was considerable, especially with respect to technological or scientific breakthroughs but also with respect to developing new ideas for integration in future space missions, use of satellite data, the development of applications and algorithms, and the creation of new R&D projects. According to most CSA employees and funding recipients, the extent of new knowledge resulting from funded projects either met or surpassed their initial expectations. Conversely, a detrimental impact on knowledge gain was evidenced for proposed projects that didn't receive Class G&C funding. However, some key informants indicated that the potential for knowledge gain was thwarted by both the discontinuation of AOs in targeted space S&T areas and the tendency for only one of the CSA's main programs to extensively use the Class G&C Program. In addition, CSA employees reported closer alignment between the knowledge produced and the CSA's priorities for solicited projects than for unsolicited projects.

Archival data indicates that, over the course of the evaluation period, competitive processes (AOs) were launched and G&C agreements were awarded via the Research component of the Class G&C Program, with the aim of increasing knowledge in priority space S&T areas. As described above, a total of 195 projects were funded and 117 (60%) of these projects were funded as a result of a competitive solicitation process. These solicited projects stemmed from ten AOs, eight of which were launched after the start of the evaluation period. By the end of the evaluation period, two more AOs had been launched (in and of ), though G&C agreements had not yet been awarded for them.

In the Class G&C Program's PM Strategy (CSA, 2013a), baseline data and targets for the number of AOs launched and the number of new and ongoing projects funded were established based on the - fiscal year. As shown in Table 8, the number of AOs with proposal due dates set for the two years following - were inconsistent (first lower then higher than in -), as were the number of new and ongoing projects funded.

The program's PM Strategy (CSA, 2013a) established the baseline data and target for the number of funded projects completed per year based on the - fiscal year, which was the last year of the evaluation period. At this time, 138 (71%) of the 195 funded Research projects had been completed (Table 8). The remaining 57 funded projects (29%) were ongoing, with the latest funded project end dates planned for .

Table 8: Number of AOs launched, new and ongoing projects funded, and funded projects completed per fiscal year
- - - - -
Number of AOs launched
(based on proposal due dates)
0 2 3 1 4
Number of new and ongoing projects funded
Solicited 52 51 58 30 49
Unsolicited 29 19 27 30 37
Total 81 70 85 60 86
Number of funded projects completedTable note 25
Solicited 13 14 34 0 15
Unsolicited 17 9 9 6 21
Total 30 23 43 6 36

Table note

Table note 25

A funded project is considered "complete" when the full amount of awarded funding has been given to the funding recipient.

Return to table note 25 referrer

Extent of Knowledge Increase

Information provided by funding recipients in their progress and final reports, as well as opinions shared by key informants via their responses to the online questionnaires, suggest that, overall, the extent of knowledge increase generated by the funded projects was considerable. However, in some cases, continued project funding and funding awarded by a wider representation of CSA programs would be required in order to achieve a sufficient amount of knowledge gain.

Table 9 shows the number and percent of funded projects for which at least one success was achieved in each of six key indicators of increased knowledge. As explained in the Limitations section of this report, usable data for these indicators was available only for - through - because of variations in the progress and final report templates over time. However, it is important to note that the source of these data was funding recipients' annual progress and final reports regarding their funded projects, many (85) of which were ongoing at the end of the evaluation period. Therefore, the findings reported in Table 9 likely underestimate the extent of knowledge increase that will be achieved by the time all of the funded projects have come to fruition.

As per the findings in Table 9, technological or scientific breakthroughs were the most common types of knowledge increases reported by funding recipients, with two-thirds (66%) of funded projects producing at least one breakthrough by the end of the evaluation period. In addition, about half of funded projects led to the development of new ideas that might be integrated into future space mission (56%), the use of satellite data (51%), and the development of applications and algorithms (43%). Though increased levels of technology and commercial successes were seldom reported (23% and 5%, respectively), the latter finding is not surprising given that, during the evaluation period, the bulk of G&C funding was awarded to academic organizations. Similarly, the limited achievements with respect to increased levels of technology likely reflect the CSA's decisions regarding which projects to fund; as reported in the Relevance section above, funding recipients suggested that, to better address the needs of Canadians, the Class G&C Program should resume discontinued AOs and award more G&Cs for developing higher technology readiness levels of existing technologies.

Table 9: Number and percent of funded projects between - and - (n=121) for which increased knowledge was reported in progress and final reports
Indicator of Increased Knowledge Number of
Funded Projects (%)
Technology or scientific breakthrough 81 (66%)
Development of new ideas that might be integrated in future space missions 69 (56%)
Satellite data used (Canadian or foreign) 63 (51%)
Applications and algorithms development (software readiness level) 53 (43%)
Technology led to a higher level of development (technology readiness level) 28 (23%)
Commercial success 6 (5%)

Of note, though only a little over half of funded projects (56%) had developed new ideas that might be integrated in future space missions by the end of the evaluation period, the potential for Class G&C funding to increase Canadian participation in space missions was spontaneously mentioned by many key informants in their online questionnaire responses, as illustrated by following quotes:

I believe that G&C is the single most important mechanism I have had in the past 5 years for conducting international-level planetary research. It has been crucial in enabling us to participate in the wide range of international planetary missions in which we are now engaged. (written by a funding recipient)

The collaborations formalized from CSA's G&C Program have led to consistent flight opportunities for Canadian scientists who tie in to new or existing national and international mission teams. (written by a CSA employee)

The impact [of not having received funding] is significant. We cannot carry forward our research and mission ideas. They are carried forward by colleagues in other countries. (written by a non-recipient)

Key informants' online questionnaire responses provided several other indications that the extent of knowledge engendered by Class G&C Program funding was considerable. For example, over half (54%) of funding recipients reported that the results of the Class G&C funding they received generated new projects funded through means other than the program (often via international partners); another 39% reported that it was too soon to tell if new projects would be generated.

When specifically asked to identify the extent of knowledge generated by funded projects, most funding recipients (85%) and CSA employees (58%) reported that Class G&C funding contributed either considerably or extremely to increasing knowledge (Figure 17). Those key informants who reported less knowledge gain explained that, though the funded projects were informative, continued funding would have been required to obtain a sufficient amount of knowledge. Incidentally, most of those respondents who did not provide ratings explained that the funded projects with which they were involved were still in the early stages of development; however, a few CSA employees explained that they would require more specific and targeted performance data in order to meaningfully rate the extent of knowledge generated by funded projects.

Figure 17: Funding recipients and CSA employees' opinions regarding the extent to which G and C funding increased space S and T knowledge (%).
Recipients CSA employees
don't know 5% 17%
extremely 32% 17%
very 53% 41%
moderately 11% 21%
slightly   5%
not at all    

Key informants were also asked via online questionnaires to indicate how closely the extent of increased knowledge matched their initial expectations. The vast majority of funding recipients (93%) and over half of CSA employees (56%) reported that the extent of knowledge generated by Class G&C funding either matched or surpassed their initial expectations. However, according to 14% of CSA employees (or 20% of those who provided a rating), the extent of knowledge gain had been less than they expected. These employees explained that, to match their initial expectations, the Class G&C Program would need to be used more extensively across the CSA's programs, rather than primarily by the CSA's Future Canadian Space Capacity program (PAA 1.3). As described earlier in this report, this theme also emerged from CSA employees' responses pertaining to the relevance of the program, in which they explained that more extensive use of the Class G&C Program across the CSA would be required in order to better meet the needs of Canadians.

Another means of assessing the extent of knowledge increase is to examine the impact on proposed projects that were not funded by the program. According to non-recipients' online questionnaire responses, 21% were able to carry out their proposed projects without Class G&C funding. However, these non-recipients explained that the funding they received for their proposals from other sources was a stop-gap measure (i.e., insufficient in the long-run). Furthermore, they reported that receiving Class G&C funding would have had an important positive impact on their projects in terms of timelines, scope, and quality. These responses suggest that the absence of Class G&C funding can have a detrimental impact on knowledge gain.

Regardless of whether or not non-recipients were able to carry out their proposed projects, most of them described a negative impact on the potential for knowledge gain resulting from lack of G&C funding. For example, one non-recipient explained that "It is difficult to estimate the damage of not having these results [from the proposed project], but I am convinced that it is influencing the competitiveness of our industry."

Alignment between Knowledge Increase and the CSA's Priorities

To fully achieve the expected outcomes of the Class G&C Program, the knowledge gained from funded projects should be aligned with the CSA's space S&T priorities – especially given that supporting activities that contribute to the CSA's mandate and priorities was identified by CSA employees as a key need that should be addressed by the program (see Relevance section). As reported in the Outputs section of this report, alignment with the CSA's priorities was found to be more pronounced for AO than for unsolicited funding mechanisms. Therefore, it is not surprising that a similar trend was evidenced regarding the alignment between knowledge increase in priority areas and the type of mechanism used to fund projects. As illustrated by the frequencies presented in Figure 18, those CSA employees who provided ratings were significantly more likely to report that the knowledge produced by solicited projects was aligned with the CSA's priorities than the knowledge produced by unsolicited projectsFootnote 26.

Figure 18: CSA employees' opinions regarding the extent to which the knowledge produced was aligned with the CSA's space S and T priorities (%).
Solicited projects Unsolicited projects
don't know 6% 26%
extremely 30% 11%
very 49% 34%
moderately 11% 28%
slightly 4% 2%
not at all    

Evaluation Question #11: Has space focus in universities, post-secondary institutions, and not-for-profits and for-profit organizations been maintained and/or increased? (Oc2)

Evaluation Finding #11: Half of the funded projects brought new players into space-related research fields during the course of the evaluation period. In addition, the number of organizations participating in funded projects' research teams increased considerably and included national and international representation from academia, industry, and other types of organizations. Further evidence of increased focus on space as a result of the Class G&C Program was provided by key informants, most of whom reported that the program has played an important role in supporting a space focus in academia. However, for the period of time under review, very little Class G&C funding was awarded to industry and not-for-profit organizations; therefore, the program's contribution to maintaining a space focus in these types of organizations was limited.

In their annual progress and final reports, funding recipients identified which other organizations were related to the research teams for their funded projects. Some of the annual data pertaining to the identification of these organizations were missing and reliable data were only available for the last three years of the evaluation period (- to -). However, based on available data for these three years, a total of 242 different organizations (other than the funding recipients' organizations) participated in at least one funded project's research team for at least one year. Given that several of these organizations were related to more than one funded project's research team, the average number of other organizations related to the research team per funded project was 4.48 (SD = 4.69), with a wide range of zero to 34.

Among the 242 other organizations that participated in at least one funded project's research team, 42% were Canadian (Figure 19). The remainder were organizations based in Europe, the United States, and other regions (e.g., Costa Rica, Australia, China).

Figure 19: Organizations that participated in research teams by geographic location.
Canadian American European Other
42% 25% 27% 6%

Keeping in mind that 95% of funding recipients during the evaluation period were from academic institutions, it follows that the most common type of organization participating in research teams was universities (representing 49% of organizations that participated in at least one funded project's research team). However, as shown in Figure 20, the remaining 51% of the research team organizations were private companies, research centres, national government entities, and other types of organizations (e.g., provincial entities, not-for-profit organizations, colleges).

Figure 20: Organizations that participated in research teams by type of organization
University Research center National/federal governement department or entity Private company Other
49% 14% 9% 18% 10%

As shown in Figure 21, the number of other organizations participating in research teams each year increased considerably in the last two years for which reliable data were available. In -, the annual number of research team organizations nearly doubled the baseline data and target established in - for this performance indicator, as per the Class G&C Program's PM Strategy (CSA, 2013a). Given that the number of funded projects per year did not increase as steeply, these data indicate that more organizations were participating in funded projects' research teams by the latter part of the evaluation period. Indeed, according to annual progress and final reports, half (51%) of the funded projects brought new players into space-related research fields.

Figure 21: Number of organizations participating in research teams by fiscal year
- - -
Frequency 92 162 170

When funding recipients were asked in their online questionnaires about the extent of space focus within their organizations, two thirds (68%) indicated that the Class G&C Program has contributed either considerably or extremely to maintaining a focus on space within their organizations. As one funding recipient explained, "This program is the most important source of funding for our space-related research. Without it, it is unlikely we would have near the space focus that we have." However, other funding recipients reported less impact on the space focus within their organizations because, as a couple of recipients explained, Class G&C funding cannot be relied upon as a source of ongoing support.

CSA employees expressed similar opinions to those of funding recipients regarding the Class G&C Program's contribution to maintaining a focus on space in academic institutions. Because academic institutions represented the vast majority of funding recipients during the course of the evaluation period, most CSA employees (58%) indicated that the program successfully maintains an academic space-focus. However, few CSA employees (21%) reported benefits of maintaining a space focus within industry and most (65%) were unsure about any such benefits for not-for-profit organizations (which have not been a main target population of the program).

Following the end of the evaluation period, the Class G&C Program's target population was broadened to focus more on industry (see Limitations section above), which will likely produce different findings regarding the program's contribution to maintaining a space focus in for-profit organizations at the time of the next program evaluation.

Evaluation Question #12: Have partnerships been established and/or sustained, has there been access to international collaborations for Canadian organizations, and have partner contributions been leveraged? (Oc3)

Evaluation Finding #12: The majority of funding recipients established and/or maintained partnerships in order to carry out their funded projects and over two-thirds of these collaborations were with organizations outside of Canada. Key informants indicated that Class G&C funding effectively increased beneficial collaborations with new and pre-existing partners, which contributed to the success of funded projects. In addition, about half of the funded projects leveraged partners' contributions. The estimated leverage ratio was $1.86 (cash or in-kind) for every $1 of Class G&C funding invested, but this estimated ratio should be interpreted with caution due to methodological limitations.

Partnerships Established and/or Sustained

According to findings stemming from the first evaluation of the Class G&C Program (CSA, ), the program played a critical role as an enabler of collaboration between stakeholders, which led to the success of numerous funded projects. As reported above in the Relevance section of this report, the key informants surveyed for the current evaluation identified an important continued need for the Class G&C Program to foster collaboration between national and international partners, in order for Canadians to participate in space-related activities.

According to annual progress and final reports (for those years during which reliable data were available, - to -), the majority (84%) of funding recipients maintained or established partnerships in order to carry out their funded projects. In total, partnerships were established with 225 different organizations during this period of time. However, partnering organizations were often implicated in more than one funded project, resulting in an average of 3.98 (SD = 4.16) partnerships per funded project and a range of zero to 17.

Figure 22: Partnering organizations by type of organization
University Research center National/federal governement department or entity Private company Other
49% 20% 10% 11% 10%

The composition of the 225 different partnering organizations resembled that of organizations participating in research teams (reported above) with respect to the types of organization. As shown in Figure 22, about half of partnering organizations were universities (49%) and the remainder were research centres, private companies, national government entities, and other types of organizations (e.g., space agencies, not-for-profit organizations).

As shown in Figure 23, the number of organizations per year that partnered with funding recipients on at least one funded project fluctuated over the three years of the evaluation period for which reliable data were available (- to -). The baseline data and target for this indicator were established in - (CSA, 2013a).

Figure 23: Number of partnering organizations by fiscal year.
- - -
Frequency 129 140 95

To gain a richer understanding of the quantitative findings reported above, key informants were asked about their perceptions of the partnerships formed as a result of Class G&C Program. When asked to compare the collaborations that resulted from Class G&C funding with their initial expectations, CSA employees and funding recipients alike indicated that their expectations were either met or surpassed with respect to the extent of collaboration with both new partners (74% of CSA employees and 91% of funding recipients) and partners that funding recipients had collaborated with in the past (70% of CSA employees and 94% of funding recipients; Figures 24 and 25). To further promote collaboration, it was suggested to add development of partnerships and collaborations to the selection criteria in future AOs. Of note, a similar practice is incorporated into Industry Canada's Technology Demonstration Program and NSERC's Industrial Research Chairs and Collaborative Research and Development programs, which focus on the development of partnerships and collaborations.

Figure 24: Extent to which collaboration with new and pre-existing partners matched CSA employees' initial expectations (%).
new partners pre-existing partners
too soon to know/ don't know 21% 26%
much more than expected 2%  
more than expected 30% 14%
same as expected 42% 56%
less than expected 5% 5%
much less than expected    
Figure 25: Extent to which collaboration with new and pre-existing partners matched recipients' initial expectations (%).
new partners pre-existing partners
too soon to know/ don't know 5% 6%
much more than expected 9% 6%
more than expected 40% 26%
same as expected 42% 62%
less than expected 4% 2%
much less than expected    

According to most funding recipients, the Class G&C funding they received considerably increased their collaboration with new partners (70%) and with partners they had collaborated with in the past (61%). Furthermore, most funding recipients reported that they have greatly benefited from the collaborations they developed with new partners (72%) and pre-existing partners (61%). Several funding recipients explained that the partnerships forged or strengthened as a result of Class G&C funding fostered continued collaboration following project completion. Others remarked that the success of their funded projects would have been diminished in the absence of collaboration from partners.

In a similar vein, most non-recipients indicated that, had they received the Class G&C funding, this funding would have generated considerable collaboration both with new partners (58%) and pre-existing partners (73%). In the absence of Class G&C funding, a couple of non-recipients explained that their opportunities to collaborate with partners have diminished.

Access to International Collaborations

As reported in the section directly above, funding recipients partnered with a total of 225 different organizations in order to carry out their funded projects. Among these partnering organizations, less than one-third (29%) were Canadian (Figure 26). The remainder were from Europe, the United States, and other geographic locations (e.g., Japan, Australia).

Figure 26: Partnering organizations by geographic location
Canada United States Europe Other Country
29% 25% 38% 8%

As shown in Figure 27, most CSA employees (65%) and funding recipients (86%) reported that the extent of international collaborations developed to carry out funded projects either met or exceeded their initial expectations. As a result of having received Class G&C funding, most funding recipients reported increased collaboration with international partners (64%) and over half of funding recipients indicated that they benefited greatly from these international collaborations (57%). As one funding recipient explained, "G&C funding has been key to enabling many of these collaborations."

Further evidence that Class G&C funding fosters international collaboration was provided by non-recipients, about half (55%) of whom indicated that their collaborations with international partners would have increased notably had they received Class G&C funding. Some described their lack of Class G&C funding as a missed opportunity for better positioning Canada on the international space scene.

Figure 27: Funding recipients and CSA employees' opinions regarding the extent to which collaboration with international partners matched their initial expectations (%).
Recipients CSA employees
too soon to know/ don't know 7% 26%
much more than expected 14%  
more than expected 35% 26%
same as expected 37% 40%
less than expected 4% 9%
much less than expected 4%  

Leveraged Partner Contributions

Unlike some federal transfer payment programs (e.g., NSERC's Collaborative Research and Development program, Industry Canada's Technology Demonstration Program), the CSA's Class G&C Program does not include the level of leveraged funds among its selection criteria. However, according to funding recipients' annual progress and final reports, partners' contributions were leveraged for 52% of the projects that received Class G&C funding between - and - (the years for which reliable data were available). As shown in Figure 28, the numbers of funded projects that leveraged partners' contributions in both - and - were higher than the baseline data and target established in -, as per the program's PM Strategy (CSA, 2013a).

Figure 28: Number of funded projects per year that leveraged partners' contributions
- - -
Frequency 18 41 38

Among those funding recipients who completed the online questionnaire, 79% reported having leveraged funds (cash and in-kind) for at least one project funded by the Class G&C ProgramFootnote 27 (which approximates the proportion of non-recipients who anticipated that they would have been able to leverage funds had they received Class G&C funding; 82%). Among those funding recipients who leveraged funds, 66% indicated that they had already benefited greatly from this leveraging. Furthermore, the majority of funding recipients (91%) and CSA employees (69%) indicated that the extent of contributions leveraged through the Class G&C Program either matched or exceeded their initial expectations.

Figure 29. Text version below:

Figure 29: Dollars ($) leveraged for every $1 of Class G&C funding received (Frequency)

Figure 29 - Text version

Histogram, a graph depicted by bars, displaying the frequency distribution of the dollar amount leveraged for $1 of Class G&C funding based on 40 recipients who indicated leveraging. The distribution of bars in descending order of height is concentrated on the left side of the graph, peaking at the very beginning. After a low plateau in the center-right of the graph, two small bars ascend at the right end of the graph. A bell curved line passes through the bars, and peaks at the point where the mean ($1.86) occurs.

According to estimates provided by funding recipients in their online questionnaires, an average of $1.86 was leveraged from partners or other sources for every $1 of Class G&C funding (Figure 29). This leverage ratio is comparable to that of NSERC's Collaborative Research and Development program ($1.56 per $1 invested) and lower than NRCan's Class G&C program, as reported in program evaluation reports ($3.26 per $1 invested; NSERC, ; NRCan, ). However, leverage ratio comparisons should be interpreted with caution because, though these programs share similarities with the CSA's Class G&C Program, their nature and design differ from the CSA's program. Furthermore, the validity of the leverage data is questionable because funding recipients provided a retrospective estimate that, in some cases, combined leveraged funds across multiple funded projects. Of note, leverage data were also collected via funding recipients' annual progress and final reports, but the validity of these data was also questionable and the data were not accessible in an aggregated form that is conducive to data analysis.

3.2.1.3 Intermediate Outcomes

Evaluation Question # 13: Has the availability of space-related knowledge and information in priority areas been increased? (Oc4)

Evaluation Finding #13: A considerable number of presentations and publications were produced based on the knowledge gained from the majority of funded projects. Knowledge was disseminated nationally and internationally, as well as within academic circles and for the general public. Furthermore, following project completion, most funding recipients continued to disseminate knowledge to communities outside of their organizations.

The first evaluation of the Class G&C Program (CSA, ) found that a broad dissemination of funded projects' research results had not yet been achieved due to the short history of the program at that time. In contrast, findings from this evaluation indicate space-related knowledge generated from funded projects is now widely disseminated. According to funding recipients' progress and final reports, most funded projects (69%) led to important achievements with respect to outreach and general scientific awareness (for the years for which reliable data were accessible, - through -).

Throughout the evaluation period, the majority of the projects funded produced presentations (83%) and publications (69%).

  • Most of the presentations were conferences, seminars, or workshops (75%), and the remainder were presentations to media or the general public (18%) and other types of presentations (8%).
  • Most of the publications were peer reviewed (73%), and the remainder were non-peer reviewed (25%) and books (2%).
  • Most of these publications (74%) contained an acknowledgement of the CSA's funding.
  • For fiscal years - and - (the years for which reliable data were available), 37% of the new and ongoing funded projects also produced other types of publications (e.g., materials published on the internet, technical reports, training manuals).

As shown in Figures 30 and 31, an average of 17.52 presentations and 13.71 publications were produced per funded project. However, the ranges were very wide and only a few funded projects resulted in 50 or more presentations (6%) or publications (5%). In particular, one funded project produced 539 publications, far above the number produced by other funded projects. When this outlier was removed from the calculation, the average number of publications per funded project was 10.52 (SD = 16.49).

Figure 30. Version textuelle ci-dessous:

Figure 30: Number of presentations delivered per funded projects.

Figure 30 - Text version

Histogram, a graph depicted by bars, displaying the frequency distribution of the number of presentations per funded project representing a total 153 projects. The distribution of bars in descending order of height is concentrated on the left side of the graph, peaking at the very beginning. The center of the graph displays a plateau of very low frequency bars with one low frequency outlier bar at interval 150-200 presentations. A bell curved line passes through the bars, and peaks at the point where the mean (17.52 presentations) occurs.

Figure 31. Version textuelle ci-dessous:

Figure 31: Number of publications (peer reviewed, non-peer reviewed, and books) delivered per funded projects.

Figure 31 - Text version

Histogram, a graph depicted by bars, displaying the frequency distribution of the number of publications per funded project representing a total of 153 projects. The distribution of bars in descending order of height is concentrated on the left side of the graph, peaking with a high frequency bar at very beginning of the graph. One low frequency outlier bar is displayed at the right side graph at interval 500-600 publications. A bell curved line passes through the bars, and peaks at the point where the mean (13.71 publications) occurs.

As per Figure 32, the numbers of presentations and publications per year increased steadily over the course of the evaluation period. The baseline data and target for these indicators were established in - (CSA, 2013a) and, in the two following years, the numbers of both presentations and publications exceeded the target.

Figure 32 : Number of presentations and publications per year (Frequency)
- - - - -
Presentations 184 254 537 868 838
Publications 63 546 367 636 485

Information provided by funding recipients in response to the online questionnaire for this evaluation suggests that the knowledge gained from funded projects continued to be widely disseminated following project completion. Specifically, 70% of funding recipients reported a considerable amount of continued knowledge dissemination to communities outside their organizations.

When asked how closely the extent of knowledge dissemination matched their initial expectations, the majority of both funding recipients (89%) and CSA employees (83% of those who provided ratings) indicated that their initial expectations were either matched or exceeded. They explained that the knowledge produced by Class G&C-funded projects was widely disseminated both nationally and internationally, as well as both within academic circles and for the general public. However, some respondents suggested that the CSA could facilitate an even wider dissemination of knowledge by making publications more accessible both within the CSA and to the general public. In addition, some respondents noted that publications and presentations are less pertinent measures of performance for funding recipients from industry than from academia.

Evaluation Question # 14: Has space-related S&T capacity in target areas been increased? (Oc5)

Evaluation Finding #14: Space-related S&T capacity has been increased as a result of the Class G&C Program, particularly with respect to HQP and space-related expertise capacity, but also with respect to the availability of materials and equipment for conducting space-related research. During the evaluation period, capacity-building was most evident within academic settings for carrying out space science research, as evidence by findings that most members of funded projects' research teams (HQP) were students or faculty. However, the discontinuation of AOs in previously targeted space S&T areas threatens the maintenance of some of the capacities developed to date.

Overall Impact on Capacity

As reported in the Relevance and Immediate Outcomes section of this report, when asked what needs are addressed by the Class G&C Program, key informants most frequently identified the need to build capacity in the Canadian space sector. During the course of the evaluation period, the Class G&C Program's Research component funded 195 space S&T projects, with the goal of sustaining and enhancing the Canadian capacity to use space in addressing national needs and priorities. These funded projects resulted in an increase in space S&T knowledge and, according to most funding recipients and CSA employees, Class G&C funding also contributed to maintaining a space focus in Canada.

Figure 33: Funding recipients and CSA employees' opinions regarding the extent to which the program's impact on capacity matched their initial expectations (%).
Recipients CSA employees
too soon to know/ don't know 9% 33%
much more than expected 7% 2%
more than expected 37% 16%
same as expected 46% 37%
less than expected 2% 12%
much less than expected    

Against this backdrop, it is not surprising that most funding recipients (77%) reported a strong beneficial impact of the Class G&C Program on their organizations' overall capacity for conducting space-related S&T research. As illustrated by the following quote, some funding recipients explained that their organizations' capacity would have diminished dramatically without Class G&C funding: "Without it, we'd have next to nothing; it has been absolutely critical." Similarly, most CSA employees (69% of those who provided ratings) reported a beneficial impact on Canada's overall capacity for conducting space-related S&T research.

In addition, the majority of both funding recipients (89%) and CSA employees (68%) indicated that the extent of the program's impact on capacity either matched or surpassed their initial expectations (Figure 33).

As mentioned earlier in this report (see Immediate Outcomes section), only 20% of non-recipients who completed the online questionnaire carried out their proposed projects and, in the absence of Class G&C funding, the timelines, scope, and quality of these projects were hindered. When asked what effects receiving Class G&C funding would have had, 62% indicated that this funding would have considerably increased their organizations' overall capacity for conducting space-related S&T research. In their open-ended responses, several non-recipients reported challenges maintaining their organizations' capacity without Class G&C funding.

Though the data reported above demonstrate that Class G&C funding effectively increases space-related S&T capacity, non-recipients and funding recipients did not differ significantly with respect to their ratings of their organizations' current capacity for conducting space-related S&T research. A partial explanation for this lack of statistical difference was offered in the open-ended questionnaire responses provided by funding recipients and CSA employees. Several respondents from both groups explained that they have recently witnessed a detrimental effect on capacity, brought about by the discontinuation of AOs in previously targeted space S&T areas. In particular, they pointed to challenges with maintaining HQP and expertise in space science domains. A recipient explained that "the real question is what happens from here, now that these programs are done. How do we maintain our expertise in space-related research? We risk losing it."

Types of Capacity-Building

In their online questionnaires, funding recipients, non-recipients, and CSA employees were asked about the extent to which the Class G&C Program beneficially impacts three specific types of capacity building:

  • HQPFootnote 28,
  • space-related expertise, and
  • availability of useful materials and equipment (e.g., laboratory equipment) for conducting space-related research. 

The responses to these online questions aligned with the findings reported in the Relevance section concerning the capacity needs addressed by the Class G&C Program.

The Class G&C Program's Research component was not explicitly designed to increase the availability of materials and equipment for conducting space-related research. Nonetheless, as shown in Figures 34 through 36 below, several key informants pointed to the effectiveness of the program in increasing materials and equipment capacity. Specifically, half of funding recipients (51%) reported a considerable beneficial impact on the materials and equipment available within their own organizations and 26% reported beneficial impacts that extended to organizations outside of their own. Similarly, a notable minority of non-recipients anticipated that Class G&C funding would have greatly benefited their organizations' (42%) and other organizations' (42%) materials and equipment capacity. Though several CSA employees were not able to provide ratings, 48% of those who did so reported considerable materials and equipment benefits for Canadians.

Though the Class G&C Program appears to have an unintended positive impact on materials and equipment capacity, all three groups of key informants indicated that the main capacity-building benefits of the program pertain to HQP and space-related expertise (Figures 34, 35, and 36). According to most recipients, the Class G&C funding they received beneficially impacted both HQP (74%) and space-related expertise (77%) capacity within their organizations. In addition, a minority of funding recipients reported beneficial impacts on HQP (36%) and expertise (43%) in other organizations. In somewhat similar proportions, non-recipients indicated that, had they received Class G&C funding, their organizations and others would have experienced beneficial impacts on both HQP (67% and 17%, respectively) and expertise capacity (67% and 58%, respectively). Likewise, most of the CSA employees who provided ratings reported a beneficial impact in Canada on HQP (64%) and space-related expertise (60%) resulting from the Class G&C Program. Key informants from all three groups of respondents explained that the program is particularly beneficial for creating new jobs, attracting new talent (both from within Canada and abroad), and training students and personnel.

Figure 34: Funding recipients' opinions regarding the extent of beneficial impact that the program had (within their own organization and in other organizations) on HQP, space-related expertise, and materials and equipment (%).
HQP Expertise Materials and equipment
HQP -
Within own
organizations
HQP -
Outside their
organizations
Expertise -
Within own
organizations
Expertise -
Outside their
organizations
Materials and
equipment -
Within own
organizations
Materials and
equipment -
Outside their
organizations
too soon to know/ don't know 2% 17% 2% 11% 4% 17%
extremely 32% 11% 32% 15% 21% 13%
considerably 42% 25% 46% 28% 30% 13%
moderately 23% 23% 14% 25% 16% 17%
slightly 2% 9% 7% 13% 14% 13%
not at all   15%   8% 16% 26%
Figure 35: Non-recipients' opinions regarding the extent of beneficial impact that program would have had (within their organization and in other organizations) on HQP, space-related expertise, and materials and equipment (%).
HQP Expertise Materials and equipment
HQP -
Within own
organizations
HQP -
Outside their
organizations
Expertise -
Within own
organizations
Expertise -
Outside their
organizations
Materials and
equipment -
Within own
organizations
Materials and
equipment -
Outside their
organizations
too soon to know/ don't know 25% 25% 17% 17% 25% 25%
extremely 25%   17% 8% 25% 17%
considerably 42% 17% 50% 50% 17% 25%
moderately 8% 42% 17% 8% 8% 8%
slightly       8% 17% 8%
not at all   17%   8% 8% 17%
Figure 36: CSA employees' opinions regarding the extent of beneficial impact in Canada that the program has on HQP, space-related expertise, and materials and equipment (%).
HQP Expertise Materials and equipment
too soon to know/ don't know 23% 19% 41%
extremely 14% 9% 5%
considerably 35% 44% 7%
moderately 19% 14% 29%
slightly 9% 14% 10%
not at all     10%

Funding recipients' annual progress and final reports corroborate the finding that the Class G&C Program effectively supports HQP capacity. As shown in Table 10, the number of HQP working on funded projects' research teams increased over the course of the evaluation period and, in the last two years, it approximately doubled the baseline data and target established in - for this indicator (CSA, 2013a). However, when interpreting this data, it should be noted that some HQP likely worked on more than one funded project's research team in the same fiscal year and, therefore, were counted more than once. Furthermore, the data reported in Table 10 do not take into account the amount of time that each HQP dedicated to working on a funded project. Data pertaining to full-time equivalents (FTEs) were collected toward the end of the evaluation period, but these data were not accessible in a format conducive to analysis for this evaluation.

Table 10: Type, number, and percent of HQP involved in funded projects' research teams by fiscal year
Type of HQP - - - - -
Faculty 41 (21%) 84 (32%) 191 (38%) 374 (34%) 324 (34%)
Students & Post-Doctoral Fellows 128 (66%) 159 (61%) 256 (51%) 584 (53%) 435 (46%)
Other 26 (13%) 20 (8%) 53 (11%) 140 (13%) 198 (21%)
Total 195 263 500 1098 957

On average, 10.21 (SD = 11.66) HQP per fiscal year worked on each funded project's research team, with a range of 1 to 103 HQP per research team per year. In terms of composition, about half (52%) of funded projects' research members were students and post-doctoral fellows, 34% were faculty, and 15% were other types of HQP (e.g., scientists, engineers, technicians from industry).

The large proportion of students and faculty on research teams is a result of most Class G&C funding having been awarded to academic institutions. In their open-ended responses, some CSA employees questioned the relevance of HQP as a performance indicator for G&Cs awarded to industry, noting that maintaining HQP capacity is less of a concern for industry than it is for academic funding recipients. As the Class G&C Program evolves to include more funding recipients from industry, the pertinence of retaining HQP capacity as an indicator for these recipients should be examined.

Evaluation Question #15: Has multi-disciplinary and/or institutional collaboration been increased? (Oc6)

Evaluation Finding #15: Multi-institutional and multidisciplinary collaborations were forged by funding recipients, with half of funding recipients reporting a considerable increase in their multidisciplinary collaborations as a result of Class G&C funding. The extent to which multidisciplinary collaborations took place either matched or exceeded most funding recipients and CSA employees' initial expectations. Furthermore, half of the funded projects resulted in at least one achievement related to multidisciplinary research.

Data from progress and final reports pertaining to the number of multidisciplinary collaborations established by funding recipients were not accessible in a format conducive to analysis for this evaluation. However, as reported in the Immediate Outcomes section, the vast majority of funding recipients were from academic institutions and about half of the organizations that they worked with in their research teams and that they collaborated with through partnerships were from private companies, research centres, national government entities, and other types of non-academic organizations. These findings indicate that multi-institutional collaborations were established by funding recipients in order to carry out their funded projects.

According to most funding recipients (90%) and CSA employees (67%), the extent to which funded projects led to multidisciplinary collaborations either matched or exceeded their initial expectations. About half of funding recipients reported that their multidisciplinary collaborations increased considerably as a result of Class G&C funding (51%) and that they benefited greatly from these collaborations (46%). As one funding recipient explained, "For the first time… we have created a multidisciplinary team. This has been a fantastic move and has greatly improved both the training opportunities and the effectiveness of the project – much more than I expected." In a similar vein, most non-recipients (64%) indicated that Class G&C funding would have increased their collaborations with multidisciplinary teams, had they received funding for the proposed projects.

According to funding recipients' progress and final reports, several of the multidisciplinary collaborations were particularly fruitful. Specifically, 50% of the funded projects resulted in one or more achievement related directly to multidisciplinary research (during the years for which reliable data were available, - through -).

Figure 37: Number of multidisciplinary achievements produced by funded projects per year.
- - -
Frequency 25 32 30

As shown in Figure 37, in both - and -, the number of multidisciplinary achievements produced by funded projects each fiscal year was slightly higher than the target for this indicator, which was established based on data from - (CSA, 2013a).

3.2.1.4 Ultimate Outcomes

Evaluation Question #16: Does Canadian space-related R&D respond to the national needs and priorities? (Oc7)

Evaluation Finding #16: Overall, the Class G&C Program aligns with departmental and federal priorities, and effectively responds to the space-related needs identified by funding recipients, non-recipients, and CSA employees.

As per the findings reported in the Relevance section of this report, the Class G&C Program's objectives are aligned with both federal priorities and the CSA's strategic outcomes. Furthermore, according to key informants, the program addresses the following Canadian space-sector needs:

  • capacity building for space-related activities,
  • collaboration between industry, government, academia, and international partners, and
  • support for funding recipient activities that contribute to the attainment of the CSA's mandate and priorities.

Findings pertaining to the achievement of program outputs and outcomes demonstrate that, as a whole, the Class G&C Program effectively responds to these needs. As reported above, the program has increased space-related capacity (particularly with regard to HQP and expertise), though some evidence points to challenges maintaining the capacities developed. In addition, the program has facilitated partnerships and multidisciplinary collaborations both nationally and internationally. Furthermore, strong alignment was evidenced between the CSA's priorities and the projects funded through the AO solicitation process, both in terms of the funding mechanism employed in the selection of solicited projects and the knowledge produced by solicited projects. However, according to some CSA employees, alignment with the CSA's priorities is less clear for unsolicited projects.

3.2.2 Demonstration of Efficiency and Economy

Evaluation Question # 17: Efficiency: To what extent is the program delivering outputs and achieving outcomes in the most efficient manner? (Ee1)

Evaluation Finding #17: The Class G&C Program produced good value with respect to use of public funds by achieving its expected outcomes and bolstering Canada's reputation for space R&D through modest and flexible investments. However, some key informants suggested that value for money could be further improved by providing more frequent and predictable CSA support (including to space-related domains targeted by prior AOs), by better aligning unsolicited projects with CSA priorities, and by increasing the transparency, speed, and amount of feedback of the selection process. In addition, divergent perspectives were voiced by CSA employees with respect to the appropriateness of the governance structure and the efficiency of resource utilization. In particular, CSA employees disagreed about whether the amount of CoE resources dedicated to the oversight of funding initiatives impedes or adds value to their development and, more generally, about whether the CSA should treat G&Cs as components of a broader program with its own specific objectives, processes, and expected outcomes. Keeping in mind the Policy on Transfer Payments (TBS, ) and Canada's Space Policy Framework (CSA, 2014b), a review of the Class G&C Program's Terms and Conditions, including the accountabilities and roles and responsibilities, could prove beneficial for improving program efficiency.

In order to evaluate the Class G&C Program's efficiency, CSA employees' experiences and opinions regarding the program's overall efficiency and the efficiency of its governance and delivery structure were solicited. Additional indicators of program efficiency were provided by key informants' opinions of the program's value with respect to the use of public funds, including perspectives regarding the overall quality of the program.

Efficiency of Resource Utilization and Governance Structure

Overall Efficiency of Resource Utilization: Divergent perspectives were voiced by CSA employees regarding the extent to which the Class G&C Program has been delivered efficiently with respect to resource utilization.

According to several key informants (32%), the program has been delivered in a very efficient manner (Figure 38). These CSA employees explained that laudable efforts have been made by the CoE to optimize resource utilization (e.g., through the development of templates and a checklist). Furthermore, they commended CoE personnel for the efficient and helpful manner in which they support funding initiatives.

In sharp contrast, other CSA employees (43%) described a less efficient program delivery, burdened with heavy bureaucratic processes and too much oversight from the CoE (Figure 38). Several of these CSA employees expressed concern that the burdensome bureaucratic process leads to excessive delays in the launch of AOs, resulting in a rush to award funding agreements at the end of the fiscal year and, occasionally, in lapsed program funds.

Figure 38: CSA employees' opinions regarding the extent to which the program has been delivered efficiently with respect to resource utilization (%).
Senior
executives
CoE Financial
Management
Directors Managers
and staff
Total
don't know 20%   80% 25% 21% 25%
extremely 20%       9% 8%
considerable 40%     25% 29% 25%
moderately 20% 60% 20% 50% 35% 36%
slightly   20%     6% 6%
not at all   20%       2%

As the volume of funding agreements increased and the number of CoE personnel decreased over time, CoE personnel indicated that a large portion of their time was dedicated to oversight of funding initiatives, leaving little time for carrying out the remainder of their functions. Though a direct comparison with OGD transfer payment programs was not possible due to fundamental differences between programs, it is interesting to note that the oversight function of G&C CoEs differs across federal programs. For example, the CoE of NRC's Industrial Research Assistance Program conducts little oversight of specific funding initiatives because many of their processes are automated (e.g., online funding application tools). To determine if guidelines and templates are followed uniformly, NRC relies upon a quality assurance assessment conducted by their Finance directorate, as well as internal audits and evaluations. In contrast, the CoE of NRCan's Class G&C Program includes an Agreement Review Group that conducts oversight of all high dollar value and medium-to-high risk funding agreements, as well as any changes to approved templates and a 10-15% sampling of other agreements. NRCan's CoE then delivers a quarterly report to senior management.

Governance Structure: As with the divergent opinions expressed regarding the overall efficiency of the Class G&C Program, contradictory views were also expressed with respect to the program's governance structure. As shown in Figure 39, when asked to rate the overall appropriateness of the program's governance structure, 27% of CSA employees indicated that it is very or extremely appropriate, whereas 48% indicated that it is either not at all, slightly, or moderately appropriate.

Figure 39: CSA employees' opinions regarding the extent to which the governance structure is appropriate (%).
Senior
executives
CoE Financial
Management
Directors Managers
and staff
Total
don't know 20%   60% 25% 24% 25%
extremely         3% 2%
considerable 40%     25% 30% 25%
moderately 40% 60%   50% 30% 33%
slightly     20%   12% 10%
not at all   40% 20%     6%

In response to more specific questions about the program's governance structure, a sizable minority of CSA employees (25 to 40%) reported that the CoE's accountabilities and roles and responsibilities are both clearly defined and appropriate. Conversely, between 50 and 60% of CSA employees indicated that this is not the case and that better communication and/or revision of the CoE's accountabilities and roles and responsibilities are required. In similar proportions, divergent views were expressed regarding the extent to which accountabilities and roles and responsibilities are clearly defined and appropriate for employees who work on G&C initiatives (e.g., directors, managers, staff).

Across the sub-groups of CSA employees surveyed, those who expressed that accountabilities and roles and responsibilities are appropriate and clearly defined tended to be program directors, managers, and staff who make use of the Class G&C program. However, other directors, managers and staff disagreed, as did most CoE personnel, Financial Management personnel, and senior executives.

Relationship between Perceived Purpose of G&Cs and Perceived Program Efficiency: In their open-ended responses to the online questionnaire, a few of the CSA employees who indicated dissatisfaction with the efficiency of the program's governance structure explained that it is outdated and should be revised to better align with both Canada's Space Policy Framework (CSA, 2014b) and the CSA's new project management approach.

However, much of the discord among CSA employees with respect to program efficiency and governance structure stemmed from the lack of a shared vision regarding the purpose of G&C funding agreements. Essentially, CSA employees disagree about whether:

  1. like contracts, G&Cs should be treated like a means of funding projects that are solely designed to support the objectives of other programs in the CSA's PAA, or whether
  2. G&Cs should be treated as components of a broader, CSA-wide program with its own specific objectives, processes, and expected outcomes.

Those CSA employees who viewed G&Cs merely as a means of funding projects in support of other CSA programs perceived the CoE's oversight efforts geared at ensuring consistencies in program delivery and results as inefficient. They explained that expertise regarding the Canadian space sector's and the CSA programs' needs lies with program directors, managers, and staff, rather than with the CoE. In order to improve efficiency, they recommended revising roles and responsibilities accordingly.

Conversely, those who view G&Cs as contributing to a CSA-wide program cautioned that G&Cs should not be confused with contracts. They expressed concern that there is a lack of harmonization across the CSA with respect to the AO development process and the selection, management, and reporting of funded projects, often because the guidelines and tools are not applied uniformly. Consequently, these key informants perceived large discrepancies in the development and administration of different AOs and in their resulting quality.

Several proponents of this view recommended further streamlining the Class G&C Program and better communicating the established accountabilities and roles and responsibilities, especially to newer users of the program who are accustomed to awarding contracts. However, a few CSA employees proposed allocating greater responsibilities and accountabilities to the CoE in order to better harmonize program delivery and results across the CSA.

Evidence collected for the purpose of the current evaluation supports the rationale for a CSA-wide G&C program with standardized objectives, processes, and expected outcomes that are uniformly applied across the CSA:

  • The Terms and Conditions of the CSA's Class G&C Program, as approved by TB in , demarcate a program designed to support the overriding objectives of the CSA, including a program implementation strategy and expected outcomes that are unique to this program and applicable to all initiatives funded through it.
  • As further delineated in the CoE's Terms of Reference (2010a), the CoE's primary role entails harmonization and standardization of program delivery and reporting, including providing G&C expertise, portfolio services in G&C management, program oversight and performance, and support of G&C Advisory Committee activities.
  • Though there are opportunities for improvement, as a whole, the Class G&C Program in its current form effectively responded to the space-related needs identified by key program stakeholders over the course of the evaluation period (as described in the Ultimate Outcomes section of this report).
  • As described in the Outputs section of this report, TBS recommends standardization of processes, systems, and procedures for the management and delivery of transfer payments within departments and harmonization across federal departments.
    • As per TBS's Policy on Transfer Payments (), TBS, ministers, and deputy heads each have responsibilities associated with the achievement of the policy's expected result that "collaboration exists within and among departments to harmonize transfer payment programs and standardize their administration, when appropriate."

However, the perception held by some CSA employees that the program's current oversight and standardization methods are inefficient cannot be ignored. A review of the Class G&C Program's Terms and Conditions, as well as of the Terms of Reference related to the program, could prove beneficial in order to ensure that the program is structured in the most efficient way to meet the needs of the Canadian space sector and CSA program priorities while, simultaneously, adhering to the TBS Policy on Transfer Payments. The results of this review and any subsequent amendments to the program's governance structure should be clearly communicated across the CSA to ensure buy-in and alignment.

Figure 40: Key informants' opinions regarding the extent to which the program produced good value with respect to the use of Canadian public funds (%).
Recipients Non-Recipients CSA employees
don't know 4% 57% 15%
extremely 61% 14% 25%
considerable 26% 21% 42%
moderately 7%   15%
slightly 2% 7% 4%
not at all      

Value with Respect to the Use of Public Funds

As shown in Figure 40, the majority of CSA employees (66%) and funding recipients (88%) reported that, overall, the Class G&C Program has produced good value with respect to the use of Canadian public funds. As one funding recipients wrote, "Canadians would be proud of the way we have spent each cent from this program!" Though most non-recipients indicated that they couldn't provide a rating, those who did so also reported good use of public funds (83%).

In explaining their answers, respondents from all three groups of key informants pointed to the program's success in meeting its expected outcomes, including knowledge generation and dissemination, capacity building and maintaining a focus on space, and facilitation of important partnerships and collaborations both nationally and internationally. According to one non-recipient, "Without [the program], Canada would quickly disappear from being essential participants [in international space programs] to followers and observers at most." In addition, several key informants described the Class G&C Program as providing a very good return on modest investments. For example, one funding recipient described "The 'bang for buck' in terms of science, publications, HQP, collaborations, etc..." In a similar vein, several CSA employees and a few funding recipients expressed particular satisfaction with the flexibility of G&Cs relative to contracts. They explained that G&Cs afford freedom to define project objectives, timelines, and budget without deliverables and in an efficient manner that is managed entirely by the CSA. "For certain activities, this program is an excellent way of financing university or industry teams, in comparison with contracts that can be much more demanding in terms of documentation," wrote one CSA employee. Similarly, a funding recipients wrote that "All of the funds contributed go directly to supporting research; the efficiency of other programs… has decreased dramatically to the point that many researchers struggle to find value in contract research."

Another means by which the Class G&C Program produced good value with respect to use of public funds was by raising the caliber of Canada's reputation for space-related R&D. Though only funding recipients explicitly identified this benefit to Canadians in their explanations of why the program produces value for money, key informants from all three groups of respondents spontaneously mentioned the program's positive impact on Canada's reputation at various points in their open-ended responses to the online questionnaire. The following quotes illustrate the credibility built by the Class G&C Program:

[The funded project] promises to put Canada firmly on the map as a solid and respected location for this type of research. (written by a funding recipient)

We are seen as credible players with good expertise and facilities, enabling us to play with the "big boys." (written by a funding recipient)

[Funding recipients] publish and present their results internationally, which makes Canada shine and attracts international collaborations. (written by a CSA employee)

Although difficult to specify the exact impact the funding would have had, it is clear that it would have represented a significant capacity-building to the benefit of Canada on the international scene. (written by a non-recipient)

Despite general satisfaction with the value produced by the Class G&C program, several key informants offered suggestions for improvement. According to some CSA employees, value for money could be improved upon by better aligning the funding spent on unsolicited proposals with the CSA's priorities (as described in the Outputs section of this report) and by reducing the level of oversight performed by the CoE to improve efficiency (as described in the sub-section directly above).

According to some funding recipients, ongoing CSA support to bring projects funded by past AOs to higher levels of maturity is the primary means by which better value for money could be achieved. As one funding recipient explained, without continued CSA funding, "… we cannot translate or bring the CSA's G&C to higher value for Canadians at this point." Writing from a broader perspective, another funding recipients explained that the better alignment of the CSA's future road map with the direction of funded projects is needed, stating that, "I believe transparency and clear definition of CSA direction are needed to improve the use of public funds." The needs for continuity of funding in priority domains, funding to bring projects to higher levels of maturity, and, more broadly, clearly articulated, CSA-wide priorities have emerged as findings of this evaluation at several points in both the Relevance section and the Achievement of Expected Outputs and Outcomes section of this report.

Figure 41 : Key informants' overall satisfaction with the Class G and C Program (%).
Recipients Non-Recipients CSA employees
don't know     6%
very satisfied 60%   19%
satisfied 14% 14% 49%
neither satisfied nor unsatisfied 16% 29% 17%
unsatisfied 9% 14% 8%
very unsatisfied 2% 43% 2%

Overall Program Quality: In their online questionnaires, key informants were asked to rate their level of overall satisfaction with the Class G&C Program, as well as to indicate how the program compares with other federal G&C programs. With respect to overall satisfaction, most funding recipients (74%) and CSA employees (68%) indicated that they are generally satisfied (Figure 41). However, non-recipients expressed significantly less overall satisfaction, with 57% indicating that they are dissatisfied with the programFootnote 29.

Similarly, most funding recipients (84%) and CSA employees (27% or 64% of those who provided ratings) indicated that the program is about the same or better than others, whereas most non-recipients (72%) indicated that the program is worse than others (Figure 42). In terms of between-group differences, funding recipients and CSA employees were significantly more likely than non-recipients to rate Class G&C Program favourably compared to others. Also, funding recipients were significantly more likely than CSA employees to rate the program favourably compared to others.

Figure 42: Key informants' ratings of the Class G and C Program compared with G and C programs from other federal departments (%).
Recipients Non-Recipients CSA employees
don't know 4% 14% 59%
much better 11%   2%
better 44%   6%
about the same 31% 14% 19%
worse 7% 36% 13%
much worse 4% 36% 2%

To better contextualize these ratings of the program's overall quality, key informants were asked to identify what they value most and what they value least about the Class G&C Program. Not surprisingly, the themes that emerged from the information provided paralleled many of the key findings reported upon in the Relevance and Performance sections above. Specifically, the most valued aspects of the program cited frequently by key informants were that:

  1. The program funds Canadian space-related activities (cited by several funding recipients and non-recipients, as well as some CSA employees);
  2. G&Cs offer more flexibility than do contracts in terms of project objectives, budget, and timelines (cited by several funding recipients and CSA employees, as well as some non-recipients);
  3. The program is instrumental in building Canadian space-related capacity (cited by several funding recipients and non-recipients); and
  4. The CoE provides useful support and expertise (cited by several CSA employees).

The least valued aspects of the program cited frequently by key informants were that:

  1. The timing of funding opportunities (AOs) is both infrequent and unpredictable, and the space-related priorities targeted by AOs also shift unpredictably (cited by several funding recipients and non-recipients, as well as some CSA employees);
  2. There is insufficient transparency and feedback in the selection process for solicited proposals, accompanied by lengthy delays between the submission due date and the announcement of selection decisions (cited by several funding recipients and non-recipients, as well as some CSA employees);
  3. The processes in place for launching AOs and awarding funding are overly bureaucratic and onerous (cited by several CSA employees).

Evaluation Question #18: Economy: To what extent has resource use been minimized in the implementation and delivery of the program? (Ee2)

Evaluation Finding #18: Class G&C agreement expenditures approximated the forecasted budget and increased by 24% over the course of the evaluation period. Though much of the data needed to evaluate program economy had not been tracked, available data showed that the costs associated with the CoE decreased by 48% over the same period of time, largely as a result of decreased spending on O&M in the latter years when the program's implementation was better established. In addition, the number of FTEs assigned to the CoE decreased in the last year of the evaluation period, resulting in a reduction of about 22% in salary dollars in -. Though fewer FTEs represented a cost saving, several CSA employees reported that this level of program inputs for the CoE was insufficient for achieving program delivery in a timely manner. In addition several CSA employees, funding recipients, and non-recipients expressed concerns that the funds awarded to academia were becoming insufficient as a result of discontinued AOs. The Class G&C Program (particularly its Research component) is unique in its targeted support of Canadian space-related S&T development, though opportunities may exist for greater collaboration with granting councils.

Three facets of economy were examined for the purposes of this evaluation:

  1. economy in program spending,
  2. the appropriateness of program inputs for supporting expected results, and
  3. redundancy or overlap with other programs.

Economy in Program Spending

Forecasted versus Actual Dollar Amount Awarded to G&C Agreements: Based on the financial data presented in Table 1 (in the Resource Allocation section), the actual amount awarded to funded projects via the Class G&C Program exceeded the forecasted budget by 8% (or $3,242K) over the combined five years of the evaluation period. The increased spending on G&C agreements relative to the program's forecasted budget took place in the earlier years of the evaluation period, when the program's current Terms and Conditions were initially being implemented. Conversely, during the last two years of the evaluation period (- and -), the forecasted budget exceeded actual spending by 5 to 8%.

An examination of forecasted and actual spending on G&C agreements by type of funding showed that, over the course of the evaluation period, the surplus in actual spending was primarily awarded via grants (Table 11). In terms of the CSA's four main programs, the Future Canadian Space Capacity program and, to a lesser extent, the Space Data, Information and Services program spent more than their forecasted budgets on G&C agreements. In contrast, the Space Exploration program spent less than its forecasted budget, as did the Internal Services program (though the budget for Internal Services was relatively small). Of note, it was not possible to examine forecasted and actual spending with respect to solicited and unsolicited projects, as well as with respect to the program's Research and Awareness and Learning components, because budget forecasts were not apportioned in this manner.

Table 11: Percent and dollar value of forecasted budget actually spent on awarded G&C agreements during the evaluation period (- to -), by CSA program and type of funding
CSA Program Type of Funding
Space Data, Information and Services (PAA 1.1) Space Exploration
(PAA 1.2)
Future Canadian Space Capacity (PAA 1.3) Internal Services
(PAA 1.4)
Grants Contributions
+13%
(775K)
-36%
($1,712K)
+14%
(4,326K)
-98%
(147K)
+20%
($6,444K)
-30%
($3,202K)

Allocation of Resources: Financial data regarding spending allocated specifically to each of the five activities or to each of the five outputs identified in the Class G&C Program's logic model were not available. In addition, the ratio of administrative costs to program expenditures could not be evaluated because, with the exception of costs associated with the CoE, financial data regarding the cost of managing and administering the Class G&C Program across the CSA had not been tracked.

The costs associated with the CoE are presented in Table 5 of the Resource Allocation section. In the years following the creation of the CoE, the forecasted FTEs and salary dollars remained fairly stable, but the forecasted budget for O&M decreased, resulting in a 27% reduction in the CoE's overall forecasted budget from - to -. In terms of actual spending, the CoE cost 32% less than its forecasted budget. A comparison of CoE costs in - with CoE costs in - shows that the actual cost of the CoE decreased by 48%, primarily due to considerably less O&M in the latter years when the program's implementation was better established. In addition to decreased spending on O&M, the number of FTEs allocated to the CoE in - decreased from about 5 to 3.5, representing a reduction of approximately 22% in salary dollars. Of note, a total of 3 CoE FTEs were budgeted for - (when the current report was written) and the actual FTEs for this year also approximated 3 FTEs.

In contrast to the decrease in CoE costs over the evaluation period, the amount spent on funded projects via the Class G&C Program was 24% higher (and 36% higher for the Research Component) in - than it had been five years earlier in - (see Tables 1 and 3 in the Resource Allocation section). Furthermore, the program's budget for all but the first of the following four fiscal years (- to -) was forecasted to more than double the actual spending on funded projects in - (Table 2). This projected budget increase is primarily accounted for by larger forecasted spending for projects funded via contributions, much of which are expected to be awarded to industry (as described in the Limitations section).

Across the evaluation period, an average of $0.07 was spent on the CoE for every dollar awarded to G&C agreements (Table 12). When tracked year-by-year, the cost of the CoE for every dollar awarded to funded projects decreased over time, from $0.10 in - (when the CoE was first created) to $0.05 in -.

Table 12: Dollar amount spent on the CoE for every $1 awarded to G&Cs funded by the Class G&C Program by fiscal yearTable note 30
- - - - Total
Amount awarded to funded projects $7,217K $11,656K $7,238K $7,813K $33,924K
Amount spent on the CoE $724K $645K $523K $378K $2,270K
Amount spent on the CoE per $1 awarded to funded projects $0.10  $0.06  $0.07 $0.05 $0.07

Table notes

Table note 30

The - fiscal year is not included in this table because the mandate and structure of the CoE was established in .

Return to table note 30 referrer

Appropriateness of Program Inputs

When CSA employees were asked about the appropriateness of the Class G&C Program's inputs, about one quarter of respondents (23%) reported that the inputs are appropriate and another 45% reported that they are insufficient for achieving program delivery (Figure 43).

Figure 43: CSA employees' opinions regarding the appropriateness of the Class G and C Program's inputs for achieving program delivery (%).
Senior
executives
CoE Financial
Management
Directors Managers
and staff
Total
don't know 20%   80% 25% 18% 23%
far too much         3% 2%
too much     20%   9% 8%
about right 40% 20%     27% 23%
too little 40% 60%   25% 41% 38%
far too little   20%   50% 3% 8%

The most commonly cited reason for insufficient program inputs was that there were too few human resources allocated to the CoE, causing delays in the administration of funding initiatives and limiting the CoE's ability to carry out its functions. In explaining their responses, almost half of CSA employees pointed to the recent increase in the use of the Class G&C Program (i.e., expanded program budget and volume of funding agreements) and the simultaneous decrease in FTEs allocated to the CoE, as described in the sub-section above on allocation of resources. In addition, according to a few CSA employees, the human resources allocated to the CoE should be expanded enough to not only cover their current roles and responsibilities, but also to take on additional functions, such as participating in the writing of AOs and answering funding applicants' questions.

In contrast, some of those CSA employees who indicated that the human resources allocated to the CoE are either adequate or excessive explained that the current CoE FTEs could be utilized more efficiently by reducing the amount of oversight performed by CoE personnel – a perspective described in detail in the Efficiency section of this report. Whether due to insufficient human resources allocated to the CoE or to inefficiency of the CoE in carrying out its oversight function, several CSA employees voiced concerns about the negative ramifications of lengthy delays between AO application due dates and the funding agreement start dates, as per the finding reported above in the Output section of this report.

The need to increase the level of G&C funding awarded to the Canadian space sector and especially to academia was also commonly cited by CSA employees who reported insufficient program inputs. With the discontinuation of funding for priorities targeted by prior AOs and increased funding currently being awarded to industry, concerns were expressed that the capacity and space focus in academia was diminishing.

Though funding recipients and non-recipients were not explicitly asked to comment on the appropriateness of the Class G&C Program's inputs, several respondents from both groups nonetheless identified concerns regarding insufficient G&C funding for academia in recent years and in the years to come. For example, one funding recipient wrote, "I hope that CSA will continue to value the partnerships with the academic community that have come through the G&C Program... and continue to support an active research community in all the areas of space science; this has been a very valuable tool for developing my research group and without the support from CSA I would not be at the point I am now." In addition, a few non-recipients commented that the extent of funding awarded by the CSA pales in comparison to that of other space-oriented countries.

Redundancy with Other Programs

An internet search did not reveal any comparable transfer payment programs aimed specifically at funding Canadian space-related S&T development. In addition, most CSA employees (57% or 77% of those who provided a rating) reported that the Class G&C Program is not redundant with other Canadian funding programs. As one CSA employee explained, "Although there are other funding sources available, few (if any) really cater to the space domain. No doubt that leveraging opportunities exists, but the reality remains that the CSA has a unique mandate and requirements that are best served by an internal program."

The absence of redundancy was further supported by the finding reported above that only 21% of non-recipients were able to carry out their proposed projects via other funding sources and, even then, the funding they received was insufficient. As one non-recipient (whose proposed project was never actualized) explained, "I very much appreciate having the opportunity to apply for funding from CSA, as it is essential for my research and not available elsewhere." Similarly, several funding recipients commented that there are only a few other funding opportunities for space-related S&T development and 88% of recipients reported that they most likely would not have been able to carry out their projects if Class G&C funding were unavailable.

Historically, the Class G&C program has collaborated on occasion with NSERC for specific funding initiatives. It was suggested that closer collaboration between the Class G&C Program and granting councils could prove beneficial for leveraging funds. To facilitate this collaboration, a few CSA employees suggested that the delivery mechanisms of the Class G&C Program (e.g., application and performance reporting processes) should be aligned with those of granting councils. Incidentally, efforts to harmonize the Class G&C Program with OGDs would be simplified by the fact that the federal granting councils have harmonized their Terms and Conditions and standardized their processes (according to interview with OGDs conducted for this evaluation).

4. Conclusions and Recommendations

Conclusions drawn from the relevance and performance findings reported above are presented in this section, as are evidence-based recommendations for program improvement. These recommendations appear in bold font in the text below and are summarized at the end of this section.

Program Relevance

In terms of relevance, the Class G&C Program falls within the federal government's jurisdiction, as legally mandated by the Canadian Space Agency Act (), and its objectives are aligned with Government of Canada priorities in science, technology, and innovation. The space sector is recognized as making a critical contribution to Canada's prosperity, and space-related public investments are essential for developing new sciences and technologies and pushing the boundaries of knowledge. According to the Canadian government, as the economic opportunities stemming from the application of new knowledge increase, it is crucial for Canada to possess a strong S&T base, robust research infrastructure, and a highly qualified workforce.

The Class G&C program's Research component is also aligned with the CSA's strategic outcomes and departmental priorities. However, the Awareness and Learning component was not evaluated because initiatives aimed at elementary and secondary school students have not been funded since and this component represented only $4M of the $40M in total program spending over the course of the evaluation period. Going forward, the Class G&C Program's Terms and Conditions should be reviewed to determine whether the Awareness and Learning component remains aligned with the CSA's priorities.

In terms of continued need, the Class G&C Program's objectives demonstrate relevance in addressing the needs of Canadians by contributing to capacity development for the Canadian space sector, by supporting collaboration between industry, government, academia, and international partners, and by supporting activities that contribute to the fulfillment of the CSA's mandate and priorities. To further respond to the needs of the Canadian space sector, key informants suggested continuing program funding in domains targeted by prior AOs.

Effectiveness of Program Performance

Achievement of Expected Outputs

Evaluation findings showed that the following G&Cs products were directly produced by the Class G&C Program and its Research component during the evaluation period ( to ):

  • Ten AOs were launched through the program's Research Component (two of which had yet to award funding agreements by the end of the evaluation period);
  • Ranked lists of applicants were produced for each of the eight AOs for which funding agreements had been awarded;
  • $40M in G&Cs were awarded by the Class G&C Program;
  • 91% ($36M) of all Class G&C funding pertained to the program's Research component;
    • 76% of Research component funding was awarded by the CSA's Future Canadian Space Capacity program (PAA 1.3);
    • 98% of Research component funding was awarded to universities;
  • 195 Research component funding agreements were awarded;
    • 40% stemmed from unsolicited proposals;
    • 60% were solicited via AOs (two of which were launched prior to the approval of the Class G&C Program's Terms and Conditions in ); and
  • 94% of the requested progress and final reports were completed by funding recipients, representing 94% of Research component-funded projects.

According to most key informants, effective funding application mechanisms were in place, the AO selection process was fair, and the reporting requirements were similar to those of OGDs. Though ranked lists of funding priorities applicable to all G&C initiatives (a planned program output) had not been produced by each of the CSA's branches, an assessment of whether the ten AOs launched during the evaluation period were aligned with CSA priorities was conducted on a case-by-case basis. The CSA should standardize the process for identifying ranked lists of funding priorities applicable to all G&C initiatives.

With respect to solicited proposals, evaluation findings showed that information regarding both the nature of the proposal selection process and delays in the selection process was not provided to applicants consistently across AOs, and constructive feedback on proposals was also provided inconsistently. In addition, there were redundancies within AO application forms and inconsistencies across AOs application processes. To address these shortcomings and in keeping with TBS recommendations for transfer payment programs, the CSA should standardize the application, selection, and feedback processes across AOs and clearly communicate these processes to the Canadian space community. To facilitate funding applicants' internal planning, key informants also suggested increasing the predictability of the timing of AOs.

Unlike solicited projects funded via AOs, evaluation findings indicate that there are no clearly articulated selection criteria for unsolicited proposals. Furthermore, key informants raised concerns about the fairness and transparency of the unsolicited proposal selection process, as well as about the extent of alignment between unsolicited projects and CSA funding priorities. As with AOs for solicited proposals, introducing standardized and transparent application, selection, and feedback processes for unsolicited proposals would address the challenges identified with the unsolicited proposal funding mechanism. Furthermore, the recommendation made above regarding producing a ranked list of funding priorities for each branch of the department would ensure alignment between unsolicited projects and CSA priorities.

The Class G&C Program's PM Strategy identifies performance indicators, baseline data, and targets for the program's expected outcomes, but not for expected outputs, indicating a need to update the program's PM Strategy to ensure that it contains output-related indicators and baseline data and targets. Also, some of the G&C data stored in the CSA's databases were inaccurate (e.g., more than one identification number per agreement in some cases, erroneous list of completed progress and final reports), indicating a need to improve the data entry process in order to ensure archival data validity.

Achievement of Expected Outcomes

Evaluation findings showed that the projects funded by the Class G&C Program's Research component successfully contributed to increasing space S&T knowledge – though clear alignment between the types of knowledge gained and the CSA's priorities was more evident for solicited projects than for unsolicited projects, thereby reinforcing the need for a standardized unsolicited proposal selection process. Evidence also showed that the knowledge produced by funded projects resulted in important technological or scientific breakthroughs, as well as in the development of new ideas for integration in future space missions, the use of satellite data, the development of applications and algorithms, and the creation of new R&D projects. Furthermore, the extent of knowledge dissemination increased; presentations and publications were widely disseminated both nationally and internationally, as well as within academic circles and for the general public. However, according to CSA and funding recipient key informants, the lack of continuity in funding for S&T domains targeted by prior AOs curtailed some funding recipients' abilities to bring their R&D projects to a higher level of maturity and reap their full potential for knowledge gain.

In addition to increasing knowledge generation and dissemination, the Class G&C Program effectively increased Canada's focus on space. For example, funding recipients brought new players into space-related research fields and the number of organizations participating in funded projects' research teams increased considerably over the course of the evaluation period. In addition, projects funded by the Research component led to notable increases in HQP and space-related expertise capacity within those organizations that received Class G&C funding, as well as outside of these organizations – though several funding recipients expressed concern that the capacities developed could not be maintained following project completion.

Class G&C funding also fostered new and maintained collaborations between funding recipients and both national and international partners, which produced leveraged funds for about half of the funded projects. Furthermore, as a result of the diversity among organizations participating in funded projects' research teams and among partnering organizations, funding recipients established multidisciplinary and multi-institutional collaborations that produced achievements specifically related to multidisciplinary research in half of the funded projects.

Due to changes in the progress and final report templates over time, approximately one-third of the report data provided by recipients were only accessible one year at a time and could not be linked to related data collected during other years in the evaluation period, thereby precluding some aggregated, multi-year analyses of the program's performance. This methodological limitation indicates a need to ensure that performance data from recipients' progress and final reports are systematically stored in an accessible, multi-year format conducive to supporting evaluation.

Program Efficiency and Economy

The Class G&C Program is unique in its targeted support of Canadian space-related S&T development and produces good value with respect to use of public funds by achieving its expected outcomes and bolstering Canada's reputation for space R&D through modest and flexible investments. In line with the conclusions reported thus far, value for money could be improved upon by standardizing the application, selection, and feedback processes of solicited and unsolicited proposals.

Salary dollars allocated by the CSA's main programs for managing G&C initiatives were not tracked during the evaluation period, thereby precluding an examination of the ratio of administrative costs to program expenditures. Data pertaining to the Class G&C Program's full administrative costs should be tracked to ensure data availability.

Available data show fairly close alignment between forecasted budgets and actual spending on G&C agreements (especially in recent years), and less spending than had been forecasted on the costs associated with the CoE. With respect to actual spending, G&C agreement expenditures increased and costs associated with the CoE decreased over time.

These spending trends are forecasted to continue after the evaluation period, calling into question whether the resources allocated to the CoE are sufficient and utilized efficiently for carrying out their responsibilities in implementing the Class G&C Program in an economically sound manner. Evidence suggests that the accountabilities and roles and responsibilities of both the CoE and of those employees who work on G&C initiatives are not uniformly understood in the same manner by CSA key program stakeholders. Furthermore, some CSA key informants reported delays in the launch of AOs resulting from the CoE's oversight function, which aims to ensure consistency in the implementation of G&C initiatives across the CSA (as per TBS's Policy on Transfer Payments that calls for standardization). Together, these findings point to a need to review and better communicate the accountabilities and roles and responsibilities of the CoE and the employees who work on G&C initiatives, as well as to ensure that the level of resources allocated to the CoE is commensurate with CoE roles and responsibilities. This review is currently underway, as a result of Management Action Plans developed in order to address recommendations stemming from the Evaluation of the Cassiope Contribution Program () and the Audit Report of the CSA's G&C Program ().

Summary of Recommendations

Based on evaluation findings and conclusions, the CSA's Evaluation function recommends the following:

  1. The Class G&C Program's Terms and Conditions should be reviewed to determine whether the Awareness and Learning component remains aligned with the CSA's priorities.
  2. For both solicited and unsolicited proposals, the application, selection, and feedback processes should be standardized and clearly communicate to the Canadian space community.
  3. The program's PM Strategy, as well as data entry, collection, and storage processes, should be revised to ensure performance data availability, validity, and accessibility. In addition, the process for identifying ranked lists of funding priorities applicable to all G&C initiatives should be standardized across the CSA's main branches.

5. References

Management Response and Action Plan

  Responsibility
Organization / Function
Management response Details of action plan ScheduleTable note 31
Recommendation # 1
Review the Class G&C Program's Terms and Conditions with respect to the Awareness & Learning component ensuring alignment with CSA priorities.

Responsibility:
Vice-president
with the support of the:
G&C Steering Committee and G&C Center of Expertise

While Management agrees with the recommendation, TBS is currently reviewing its policy on Transfer Payments; therefore, we will wait for the new policy and guidelines before reviewing the Class G&C Program's Terms and Conditions. The G&C Center of Expertise will work in collaboration with the G&C Steering Committee in order to review the Class G&C Program's Terms and Conditions.
Recommendation # 2
For both solicited and unsolicited proposals, the application, selection, and feedback processes should be standardized and clearly communicate to the Canadian space community. Responsibility: Vice-president
with the support of the :
G&C Steering Committee and G&C Center of Expertise

Management agrees with the recommendation. This recommendation includes various elements that will be addressed differently:

  1. Standardized process for AO and unsolicited proposal across the CSA.
  2. Clear communication of the processes with the Canadian space community.

 

  1. In collaboration with the G&C Steering Committee, the G&C Center of Expertise will review the various G&C processes at the CSA in order to standardize them across the CSA.
  2. The G&C Center of Expertise will update the CSA website to ensure that all the information is available so the space community is well-informed about the CSA's Class G&C program
Recommendation # 3
Revise and update the program's PM Strategy, as well as data entry, collection, and storage processes, to ensure performance data availability, validity, and accessibility. In addition, the process for identifying ranked lists of funding priorities applicable to all G&C initiatives should be standardized across the CSA's main branches.

For review of Performance measurement strategy and process for identifying funding priorities:
Responsibility:
Vice-president
with the support of the:
G&C Steering Committee and
G&C Center of Expertise

For availability, validity and accessibility of performance data: Responsibility: Chief Financial Officer

Management agrees with the recommendation. There were three aspects to this recommendation:

  1. Performance measurement strategy
  2. Standardized process for identifying ranked lists of funding priorities.
  3. Standardized process for data entry and format database to retrieve and provide data in an accessible format

 

  1. The CSA will review the performance measurement strategy for the Class G&C Program.
  2. Through the G&C Steering Committee, multi-year plans for G&C investments will be developed and funding priorities ranked. Then, priorities will be communicated to the space community.
  3. The CSA will standardize data entry process and maximize linkages between SAP and Unitas in order to make effective and efficient use of available data.

Table notes

Table note 31

The Schedule dates may be influenced by the new Policy on Transfer Payments and the requirements of Deliverology.

Return to table note 31 referrer

Page Footnotes

Footnote 8

Corporate risk was not assessed because the Class G&C Program's spending is financed by numerous programs across the CSA's PAA.

Return to footnote 8 referrer

Footnote 9

As per the Class G&C Program's PM Strategy (CSA, 2013a), baseline data and targets were established based on performance measurements in specific years and were identical to one another for each indicator used in departmental reporting.

Return to footnote 9 referrer

Footnote 10

Verification signatures for progress and final reports are submitted in hard-copy, rather than electronically.

Return to footnote 10 referrer

Footnote 12

Including one former CoE staff member.

Return to footnote 12 referrer

Footnote 16

t(34) = 4.12, p < .001

Return to footnote 16 referrer

Footnote 17

t(64) = 3.68, p < .001

Return to footnote 17 referrer

Footnote 18

t(51) = 4.80, p < .001

Return to footnote 18 referrer

Footnote 19

t(65) = 2.21, p = .03 for ease of finding information and t(64) = 2.42, p = .02 for clarity of information

Return to footnote 19 referrer

Footnote 20

Perceptions regarding how the Class G&C Program's selection process compares with that of OGDs were only asked of CSA employees because they are better acquainted with the details of the CSA's funding selection process than are funding recipients and non-recipients.

Return to footnote 20 referrer

Footnote 21

t(45) = 2.70, p = 0.10 for solicited proposals and t(34) = 2.16, p = 0.38 for unsolicited proposals

Return to footnote 21 referrer

Footnote 22

t(10.37 adjusted for unequal variances) = 4.31, p = .001 for fairness and t(61) = 4.12, p < .001 for transparency

Return to footnote 22 referrer

Footnote 23

t(45) = 4.87, p < .001 for funding recipients and t(9) = 3.00, p = .015 for non-recipients

Return to footnote 23 referrer

Footnote 24

Three large projects were initially proposed in response to requests for proposals through the contract funding mechanism. Prior to the project start dates and as suggested by PWGSC, they were converted to G&Cs in order to better align the nature of the work with the funding mechanism. All three projects were funded through the Class G&C Program based on the request for proposals' evaluation process as well as on the G&C process, resulting in two grants (totaling $1.5M) and one contribution of $3.6M.

Return to footnote 24 referrer

Footnote 26

t(33)= 3.33, p = .002

Return to footnote 26 referrer

Footnote 27

The proportion of funding recipients who leveraged funds (79%) is higher than the number of funded projects for which funds were leveraged (52%) because some funding recipients received funding for more than one project during the evaluation period (as described in the Methods section of this report). However, the possibility that a sampling bias among online questionnaire respondents exaggerated the proportion of funding recipients who leveraged funds cannot be discounted.

Return to footnote 27 referrer

Footnote 28

HQP were defined in the online questionnaire as individuals with university degrees at the bachelors' level and above.

Return to footnote 28 referrer

Footnote 29

A multivariate analysis of variance was run with respondent type (recipients, non-recipients, and CSA employees) as the independent variable and "overall satisfaction" and "comparison with other programs" as the dependent variables. There was a significant statistical difference in perceptions of program quality based on respondent type, F(4,166) = 11.03, p < .001. Respondent type significantly affected both "overall satisfaction," F(2, 84) = 15.81, p < .001, and "comparison with other programs," F(2, 84) = 16.67, p < .001. Post hoc tests showed that non-recipients were significantly less satisfied with the program than both recipients (p < .001) and CSA employees (p < .001). In addition, non-recipients were significantly less likely than both recipients (p < .001) and CSA employees (p = .004) to rate the program favourably compared with other funding program, and recipients were significantly more likely than CSA employees to rate the program favourable compared with others (p = .007).

Return to footnote 29 referrer