Language selection

Search

Top of page

Evaluation of the Canadian Space Agency Sun-Earth System Science Business Line

For the period from to

Project # 17/18 – 02-02

Prepared by the Audit and Evaluation Directorate

Table of Content

  1. Introduction
  2. Description of the SESS-BL
  3. Evaluation approach and methods
  4. Evaluation findings
  5. Conclusions and recommandations

List of tables and figures

Acronyms used in the report

Missions list

Executive Summary

This report presents the findings of the evaluation of the Canadian Space Agency (CSA)' Sun-Earth Science Business Line (SESS-BL). The SESS-BL's fundamental purpose is to support Canada's contribution to the advancement of the scientific knowledge concerning the Sun-Earth system, and the application of this scientific knowledge in a range of policy areas such as weather forecasting, environmental monitoring, climate change, natural disaster management and mitigation, and the protection of private and public infrastructures in space and on Earth. The evaluation covers a five-year period, from to , and examines the SESS-BL's relevance and performance.

The evaluation was conducted by PRA Inc., on behalf of the CSA's Audit and Evaluation Directorate, between and . The evaluation is included in the CSA's five-year evaluation plan and was conducted in accordance with the Treasury Board of Canada's Policy on Results ().

Relevance

A number of federal departments and agencies rely on existing data concerning the Sun-Earth system to administer their programs and services, and are exploring future opportunities that will emerge as new scientific missions are maturing. As the centre of expertise on space, the CSA collaborates with these departments and agencies, and with a range of other space agencies, institutions, and stakeholders, to advance the scientific knowledge concerning the Sun-Earth system, and its operational applications. In this context, the SESS-BL provides the overall framework and vision for the CSA's involvement in these scientific missions and activities. It offers guidance for both the activities undertaken within the CSA that address specific dimensions of the Sun-Earth system, and for the collaborations between the CSA and external stakeholders, in Canada and internationally.

During the period covered by the evaluation, the range and nature of activities undertaken through the SESS-BL have been directly aligned with government priorities in areas such as climate change and the monitoring of natural hazards. These activities also reflect research priorities identified by Canadian researchers and scientists. The establishment, in , of three CSA science advisory committees is widely perceived as a meaningful process to further support a shared vision among Canadian key stakeholders engaged in activities concerning the Sun-Earth system where Canada is best positioned to make a valuable contribution.

Finally, the activities of the SESS-BL reflect the legislative mandate of the CSA, and do not duplicate programs and activities undertaken by other federal departments and agencies.

Performance

Over the five years covered by the evaluation, the CSA invested $42.3 million in activities directly covered by the SESS-BL. These resources were predominantly directed toward the operations and data utilization of ongoing missions and activities, and also served to fund concept and Phase 0 studies, as well as project development activities associated with upcoming scientific missions and activities.

The evaluation provided an opportunity to document how these SESS-BL activities contribute to expanding the body of knowledge concerning the Sun-Earth system. Among other things, Canadian scientists have been directly involved in missions and activities led by the CSA and other space agencies, and their publications are regularly cited around the world. Along the same lines, the data produced through activities funded by the CSA has been used by scientists and researchers around the world to support their research priorities. One challenge that Canada continues to share with other space-faring nations is the management and storage of the data produced through their respective missions and activities. At the time of the evaluation, the CSA was still exploring options to facilitate access to the data it produces, reflecting the open data direction adopted by the federal government as a whole.

The development and operations of satellites and instruments used in support of Canada's scientific missions on the Sun-Earth system have also engaged the Canadian space sector. The range of benefits to the industry includes the opportunity to mature their technology, to build space heritage, to attract and develop HQPs, and to open-up spin-off opportunities.

The evaluation also provided an opportunity to explore some of the policy and programming applications that have derived from the scientific knowledge gained to date on the Sun-Earth system. The list of such applications that are documented in this report includes the improvement of weather forecasting in Canada and of the mitigation of floods and drought on agriculture activities through data from the SMAP and SMOS missions, the integration of space weather information in the ongoing management of electric grids, and the use of ACE/SCISAT and OSIRIS data to monitor progress on international commitment relating to climate change. These examples illustrate the fact that, while the full journey from the identification of a scientific research need to the operationalization of the ensuing scientific data may extend over a long period, it is a cycle that is now fully integrated in a number of federal programs and activities, and is expected to remain as such for the foreseeable future.

To maximize its contribution to the scientific body of knowledge on the Sun-Earth system, the CSA has to make choices that must consider a range of factors such as the priorities set by other space agencies, operational needs of federal departments and agencies, and the needs and capacities of Canadian scientists and the Canadian space industry. To this end, the CSA has implemented processes that will support a more systematic consideration of these factors, in consultation with other federal departments and agencies (through senior management committees) and with Canadian experts (through advisory committees).

Finally, at the time of the evaluation, the CSA was implementing the performance reporting requirements established in the federal Policy on Results. The evaluation provided an opportunity to better assess the range of SESS-BL performance data collected to date, and in doing so, identified gaps that can be addressed as part of this implementation process. This performance information can support the ongoing management and decision-making process respecting SESS-BL activities, in addition to enhancing the capacity of the CSA to share information on activities carried-out and results achieved by the SESS-BL with key stakeholder groups in academia, governments, and industry. The CSA's advisory committees can support the sharing of this performance information with external stakeholder groups.

Since the report does not include recommendations, a Management Action Plan is not required.

1. Introduction

This document constitutes the final report of the evaluation of the Canadian Space Agency (CSA)'s Sun-Earth System Science Business Line (hereafter the SESS-BL). It covers a range of scientific activities that contribute to a greater understanding of the Earth's space environment and its interaction with the Sun (including space weather), the chemical and physical processes found in the Earth's atmosphere, as well as the processes and interactions found within the Earth system. This evolving knowledge finds operational applications in a range of areas such as weather forecasting, environmental monitoring, climate change policy-making and assessment, natural disaster management and mitigation, and the protection of private and public infrastructures (both in space and on Earth). These scientific missions and activities on the Sun-Earth system are typically carried-out through international collaborations.

Sun-Earth System Sciences graph. Text version below:
Text version of Sun-Earth System Sciences graph

Represents three ovals surrounding a circle. In each oval, there is one the elements included in the Sun-Earth System Sciences, represented in the circle in the middle of the figure. The three elements in the ovals are as follows:

  • Atmospheric science
  • Solar-Terrestrial science
  • Earth System science

The evaluation covers a five-year period, from to , and examines the relevance, design, and performance of the SESS-BL. It is included in the CSA's five-year evaluation plan and was conducted in accordance with the Treasury Board of Canada's Policy on Results. It constitutes the first evaluation being conducted of the SESS-BL.

The evaluation was conducted by PRA Inc. on behalf of the CSA Audit and Evaluation Directorate, between and .

2. Description of the SESS-BL

This section of the report includes a brief description of the SESS-BL and of the broader context in which it operates. It covers the SESS-BL's key components, governance model, resource allocation, and expected outcomes.

2.1 The broader context of the SESS-BL

2.1.1 Advancing scientific knowledge of the Sun-Earth System

With its colossal objects and forces at play, the Sun-Earth system creates among the most challenging scientific phenomena to explore and comprehend. Such scientific knowledge is nonetheless critical, as life on Earth both affects and is affected by these dynamic forces. Expanding technological capabilities, particularly through ever more sophisticated satellite missions, are enhancing our understanding of the Sun-Earth system and of the environmental, social, political, and economic ramifications of such phenomena as space weather, climate change, or the evolving composition of the Earth's surface and atmosphere.

Over the last two decades in particular, the National Aeronautics and Space Administration (NASA) and the European Space Agency (ESA) have led several space missions focussed on the Sun-Earth system, and many more space agencies and organizations in countries such as Brazil, Canada, China, France, Germany, India, and Japan have either led or contributed to scientific missions and activities expanding our knowledge of the Sun-Earth system.

These space initiatives emerged in a context marked by a greater recognition that the interconnectivity of activities occurring on Earth and within its atmosphere has far reaching consequences. One illustration was provided by the international mobilization that occurred in the 1970s and 1980s around the need to reverse the ozone layer depletion, which led to the Montreal Protocol on Substances That Deplete the Ozone Layer, a United Nations treaty signed by 46 states in and since ratified by another 151 countries (United Nations, ). Beyond the direct harmful impact of UV radiation on humans that comes from a depleting ozone layer, this phenomenon also affects photosynthesis in plants, which may disturb the entire food chain (NASA, ). In early , NASA published a report providing the first direct proof of ozone hole recovery since the passing of the Montreal Protocol (NASA, 2018b). The ongoing monitoring of the ozone layer, commissioned by the World Meteorological Organization (WMO), and other studies on the ozone layer, have consistently relied on data provided by several satellites carrying scientific missions, including Canada's SCISAT or the OSIRIS instrument, provided by Canada and located on the Swedish satellite ODEN.

Another illustration came less than two years after the signing of the Montreal Protocol, when in , a particularly powerful solar flare triggered a solar storm that struck Earth's magnetic field. Among other things, this created electrical currents in the ground of a large part of North America that caused various forms of damages, including the shutting down of the entire power grid in Québec, leaving households in the dark for 12 hours, and halting airports in the province and the Montréal subway. Additional impacts of this solar storm included the disturbance of some satellites that were left out of control for several hours (NASA, ). Recognizing the range of significant impacts that solar storms may cause to infrastructures on Earth and in space (where the number of satellites is increasing exponentiallyFootnote 1), governments around the world have invested in scientific missions focused on space weather, with the goal of enhancing the capacity of public authorities and private entities to mitigate the impacts of solar storms and other related phenomena. Canada's CASSIOPE satellite with its ePOP instrument, ESA's SWARM mission (which includes the UVI instrument provided by Canada), the CSA's Geospace Observatory Canada Initiative, and the proposed SMILE mission, led by ESA and China and including a Canadian instrument, are some of the space activities focussing specifically on varying dimensions of space weather.

Other phenomena, such as the intensity and severity of floods, droughts, forest fire, costal erosion, melting glaciers, or raising ocean levels, have also illustrated the many ramifications of the dynamics unfolding within the Sun-Earth system.

2.1.2 Canada's participation in missions and activities related to the Sun-Earth system

The CSA is engaged in scientific activities related to the Sun-Earth system and the SESS-BL is the primary conduit through which these activities are channeled. As further elaborated in the next sub-section, the SESS-BL provides a framework for CSA's involvement that covers the entire space science life cycle, from science preparation to mission data analysis and use.

In addition to the SESS-BL, the CSA manages other programs that may address, at a secondary level, issues related to the Sun-Earth system. For instance:

  • The Space Technology Development (STD) initiative provides selected Canadian entities with financial support to advance the development of targeted space technologies. The program helps to reduce technological unknowns of future space missions and develop Canadian expertise to support such missions. While the primary goal of the STD initiative is the development or maturation of space technologies and expertise in a variety of areas, it may include technologies that are expected to be used in support of scientific missions exploring the Sun-Earth system.
  • The ESA Contribution Program, which supports the implementation of the Canada-ESA Cooperation Agreement, provides opportunities for Canadian entities (primarily industry) to participate in ESA missions and activities. The primary goal of the program is to support the space industry in Canada, particularly through an expansion of their international markets. In doing so, the nature of the activities and missions in which Canadian entities participate may relate to the Sun-Earth system.

These examples are offered for illustrative purposes only, as other activities undertaken by the CSA (such as CSA's Science, Technology and Expertise Development in Academia (STEDiA) initiative) may also address, in some capacity, issues associated with the Sun-Earth system. They nonetheless illustrate an important consideration for the purpose of this evaluation, namely that the assessment of the SESS-BL ought to consider activities directly covered by the business line, in addition to other activities that, while pursuing very separate goals and objectives, may also contribute to the achievement of the outcomes of the SESS-BL.

2.2 The SESS-BL

2.2.1 SESS-BL overview and objectives

During the period covered by the evaluation, the SESS-BL fell within the CSA's departmental program entitled Space Data, Information and Services (CSA, 2016a).Footnote 2 This departmental Program focussed on space missions related to Earth observation, satellite communications, and scientific missions (Sun-Earth system sciences). As illustrated in Figure 1, the range of activities covered have included mission design (and the development of the required technology), ground infrastructures, and the access and use of the data and imagery, as applicable, which have resulted from these missions. While some of the missions undertaken through this departmental Program are pursuing research and development (R&D) goals, other missions are primary operational in nature, in that they provide ongoing data and images that support the implementation of programs and services offered by a range of federal departments and agencies.

Figure 1. Text version below:

Figure 1

Figure 1 - Text version

Departmental Program : Space Data, Information and Services

A table with three rows containing the different subprograms and three columns containing the sub-subprograms under each of the sub programs.

  • Sub-program: Mission and technology development
  • Sub-program: Ground infrastructures
  • Sub-program: Data, imagery and services
    • Sub-sub-program Earth observation
    • Sub-sub-program Satellite communication
    • Sub-sub-program Scientific missions

The column of the SSP Science Missions is surrounded to demonstrate what the Sun-Earth system includes.

The CSA has used the concept of "business lines" to bridge the nature of missions undertaken and the various operational dimensions that typically characterize space missions. In the specific case of the SESS-BL, it therefore covers the range of activities related to mission design and technology development, the required ground infrastructures, as well as the use of data and images, all of which are specifically associated with scientific missions exploring the Sun-Earth system.

Since missions targeting the Sun-Earth system often involve international collaborations and tend to be led by large space agencies, Canada's projects and activities may involve only some dimensions of the business line. For example, they can include the provision of a specific instrument to be incorporated on a foreign satellite; the use of Canadian ground infrastructures or the undertaking of calibration or validation activities to support scientific mission led by other space agencies; or the use by Canadian researchers of scientific data from foreign missions to advance scientific knowledge on specific dimensions of the Sun-Earth system. Canada's participation may also involve satellite missions entirely led by Canada (such as ACE/SCISAT), in which case all dimensions of the business line would apply, in addition to involving other international partners or stakeholders as applicable (to provide complementary support, or to use the Canadian data for analysis).

Ultimately, the goal of the SESS-BL is to provide an overall framework and direction for the range of scientific missions and activities on the Sun-Earth system that the CSA engages in directly through the SESS-BL, or indirectly through other programing areas.

2.2.2 Program activities

Scientific space missions typically progress through various cycles, including the preparatory work leading up to the potential missions (science preparation and maturation, and mission development), and the ongoing operations of actual missions, including data analysis and use. As these missions typically unfold over an extended period of time, the work carried out through the SESS-BL may relate to missions that have been in operations for quite some time, as well as more recently launched missions. For illustrative purposes, Figure 2 includes the key missions in which the SESS-BL has been engaged in some capacity over the evaluation period.

Figure 2. Text version below:

Figure 2

Figure 2 - Text version

Selected missions related to the Sun-Earth System by launch date. The figure is a timeline with an arrow pointing to the right. On this line, the years to are aligned in five-year intervals. Below the timeline, 13 other lines representing missions related to the solar-terrestrial system by departure date. A framework defines the evaluation period.

  • MOPITT ( – ongoing)
  • OSIRIS ( – ongoing)
  • SCISAT ( – ongoing)
  • THEMIS ( – ongoing)
  • CLOUDSAT ( – ongoing)
  • SMOS ( – ongoing)
  • ePOP ( – ongoing)
  • SWARM ( – ongoing)
  • SMAP ( – ongoing)
  • GO-Canada (CGSM) ongoing)
  • EarthCare ()
  • SMILE ()
  • SWOT ()

More specifically, the SESS-BL carries out the following types of activities:

  • Provision of financial support: The SESS-BL provides financial support to external partners and stakeholders. This includes providing support for the exploratory work on potential scientific missions (concept and Phase 0 studies), for the development of new missions (phases A to D), for ongoing operational missions, including calibration and validation activities (phase E), and for the analysis of data provided through scientific missions (phases E and F). This support is managed by the CSA personnel assigned to the SESS-BL and may be provided through contracts, grants and contributions, or Memorandum of Understanding (MOU).
  • Support to other CSA activities: The personnel assigned to the SESS-BL provides support (e.g., acting as scientific authority) or collaborates with other programs within the CSA on activities concerning the Sun-Earth system. The list of these activities includes (but is not limited to) funded projects under the STD program or the STEDiA program, particularly the Flight and Fieldwork for the Advancement of Science and Technology (FAST) initiative.
  • Engage external stakeholders: The personnel assigned to the SESS-BL also interacts with a number of external stakeholders who are also involved in space missions and activities on the Sun-Earth system. The list of these external stakeholders includes other federal departments and agencies, international partners (other space agencies, universities, and international groupings and associations of researchers involved in relevant scientific fields).

2.2.3 Program resources

Over the five years covered by the evaluation, the CSA invested a total of $42.2 million in the SESS-BL.

As indicated in Table 1, the vast majority of these resources (88%) were directed to various forms of financial support to external entities undertaking scientific activities on the Sun-Earth system. A more complete analysis of these investments is included in sub-section 4.2.2 of this report. The remaining expenditures were directed towards the ongoing management of the SESS-BL. As for human resources, between 7.5 and 9 full time equivalents (FTE) were assigned annually to the SESS-BL.

Table 1: Summary of resources allocated to the SESS-BL ('000)
Activities - - - - - Total
Program activities
Contracts $6,237 $3,633 $4,029 $2,551 $2,895 $19,345
Grants $1,890 $2,074 $2,118 $1,956 $3,682 $11,720
Contributions $0 $590 $1,215 $1,200 $1,109 $4,114
MOU $637 $382 $329 $428 $380 $2,155
Sub-total $8,764 $6,680 $7,690 $6,135 $8,066 $37,335
Program management
Number of FTEsTable 1 note * 9.0 9.0 8.0 7.5 7.5 -
Salaries $954 $1,062 $944 $901 $785 $4,645
O&M (excluding contracts) $72 $92 $40 $19 $25 $248
Sub-total $1,026 $1,154 $984 $920 $810 $4,894
Total $9,790 $7,834 $8,674 $7,055 $8,875 $42,228

Source: Financial data provided by the CSA

2.2.4 Management structures

The Director General of the Space Utilization Branch holds the overall responsibility for activities carried out under the SESS-BL. In doing so, he operates within the set of applicable policies from the Treasury Board of Canada Secretariat (CSA, 2016b, p. 10), as well as the Project Management Policy and the Integrated Governance and Monitoring Framework Directive adopted by the CSA. This latter policy includes a Project Complexity and Risk Assessment tool that supports the assessment of the nature and complexity of each project being considered by the Space Utilization Branch, including missions on the Sun-Earth system.

To support the ongoing coordination of activities related to all satellite missions within the Space Utilization Program (earth observations, communications or scientific), the Director General of the Space Utilization Program chairs the Space Utilization Management Committee. It is the responsibility of each director to implement the decision of this Committee as applicable.

The CSA personnel assigned to the SESS-BL includes project managers and engineering professionals. Other professionals within the CSA provide support on a project by project basis (matrix support).

2.2.5 Program logic

The program logic of the SESS-BL is described in its Performance Measurement Strategy (CSA, 2016b, p. 18). The logic model, which illustrates the SESS-BL's results chain, is presented in Figure 3.

Graph. Text version below:

Figure 3

Figure 3 - Text version

Sun-Earth system science Logic model

Utimate Outcome

  • The GoC offers more diversified or cost-effective programs and services and academia is able to do research due to its utilization of space-based solutions (PMF 1.1)

Intermediate Outcomes

  • GoC Organizations use space data and services to deliver their mandates, and academia uses it for research (PMF 1.1.1)
  • Expressed Canadian and foreign data needs are fulfilled by reliable national ground infrastructure (PMF 1.1.2)
  • GoC Organizations use space based solutions to deliver their mandates, and academia uses it to increase their ability to perform research (PMF 1.1.3)

Immediate Outcomes

  • 1.1.1.3 Scientific missions: Scientific missions provide GoC organizations and academia with data and information (PMF 1.1.1.3)
  • 1.1.2 Ground infrastructure 1.1.2.1 and 1.1.2.2
    • CSA's satellites are functioning as per operational requirements (PMF 1.1.2.1)
    • Foreign satellites missions are supported (PMF 1.1.2.1)
    • Satellite and ground-based data is provided to GoC organizations and Academia (PMF 1.1.2.2)
  • 1.1.3.3 Scientific data utilization: The SESS Community generates new knowledge and enhances its ability to turn SESS data into useful products and services (PMF 1.1.3.3)
  • 1.1.1.3 Scientific Missions, 1.1.2: Ground Infrastructure 1.1.2.1 et 1.1.2.2, 1.1.3.3 Scientific data utilization
  • 1.1.1.3 Scientific Missions: Agreement to gain access to SESS data

Outputs

  • 1.1.1.3 Scientific missions 1.1.2 Ground infrastructure 1.1.2.1 et 1.1.2.2, 1.1.3.3 Scientific data utilization: Sun-Earth System Science Strategic documents
  • 1.1.1.3 Scientific missions, 1.1.2 Ground infrastructure 1.1.2.1 et 1.1.2.2, 1.1.3.3 Scientific data utilization: Publications, Presentations, Workshop, Reports
  • 1.1.1.3 Scientific missions: Canadian instrument on foreign satellite
  • 1.1.1.3 Scientific missions: Canadian satellite
  • 1.1.1.3 Scientific missions, 1.1.2 Ground infrastructure 1.1.2.1 and 1.1.2.2: Canadian Ground Infrastructure
  • 1.1.2 Ground infrastructure 1.1.2.1 et 1.1.2.2: Tasking of satellite & ground infrastructure systems & Management of data
  • 1.1.2 Ground infrastructure 1.1.2.1 and 1.1.2.2, 1.1.3.3 Scientific data utilization: Working groups & reports
  • 1.1.3.3 Scientific data utilization: contracts / agreements

Activities

  • 1.1.1.3 Scientific missions 1.1.2 Ground infrastructure 1.1.2.1 et 1.1.2.2, 1.1.3.3 Scientific data utilization: Needs assessment of GoC organizations and Academic Community
  • 1.1.1.3 Scientific missions 1.1.2 Ground infrastructure 1.1.2.1 et 1.1.2.2, 1.1.3.3 Scientific data utilization: communication et sensibilisation
  • 1.1.1.3 Scientific missions: Space-based Scientific Data requirements
    • 1.1.1.3 Scientific missions:
      • Scientific instrument & Mission concepts
      • Feasibility studies, Science and Technology Development & Demonstration (Phase 0)
      • Development of Canadian instruments for foreign missions (Phases A-B-C-D)
      • Development of Canadian missions (Phases A-B-C-D)
  • 1.1.1.3 Scientific missions 1.1.2 Ground infrastructure 1.1.2.1 and 1.1.2.2, 1.1.3.2 Scientific data utilization: Satellites operations & Ground infrastructure requirements
    • 1.1.1.3 Scientific missions, 1.1.2 Ground infrastructure 1.1.2.1 and 1.1.2.2: Feasibility studies
    • 1.1.1.3 Scientific missions, 1.1.2 Ground Infrastructure 1.1.2.1 and 1.1.2.2: Development of Canadian Ground infrastructure
      • Ground infrastructure 1.1.2.1 and 1.1.2.2: Satellites Operations
      • Ground infrastructure 1.1.2.1 and 1.1.2.2: Data handling
  • 1.1.1.3 Scientific missions, 1.1.2 Ground infrastructure 1.1.2.1 et 1.1.2.2, 1.1.3.2 Scientific data utilization: Data utilization support requiements
    • Ground infrastructure 1.1.2.1 and 1.1.2.2, 1.1.3.3 Scientific data utilization: Planning, coordinating and moniroting SESS data usage & requests
    • 1.1.3.3 Scientific data utilization: Planning, coordinating and monitoring of SESS data utilization initiative.

3. Evaluation approach and methods

This section of the report provides a brief description of the methodology used to conduct the evaluation of the SESS-BL. It clarifies the purpose and scope of the evaluation, describes the key evaluation issues being addressed, and the methods used to gather evaluation findings. It also identifies the limitations that the evaluation faced, along with the strategies used to mitigate these limitations.

3.1 Purpose and scope

This report fulfills the commitment included in the CSA's Departmental Evaluation Plan ( to ) to conduct the evaluation of the SESS-BL. It covers a five-year period, from to .

The evaluation covers the relevance and performance of the SESS-BL. The following six evaluation questions are addressed:

Relevance
  1. Does the SESS-BL continue to address a demonstrable need?
  2. Is the SESS-BL aligned with departmental strategic outcomes, federal government priorities, and the priorities of Canadian research institutions involved in scientific research related to the Sun-Earth system?
  3. To what extent do activities undertaken through the SESS-BL reflect an appropriate distribution of roles and responsibilities among key stakeholders?
Performance (effectiveness)
  1. Has the SESS-BL achieved its immediate and intermediate outcomes?
  2. To what extent has the SESS-BL contributed to the achievement of its ultimate outcome?
  3. What processes has the CSA implemented, if any, to enhance the efficiency of SESS-BL activities?

3.2 Methods

Evaluation data were collected through a number of research methods, which are briefly described in this sub-section.

3.2.1 Overall approach

As this is the first evaluation of the SESS-BL, the methodological approach was designed to offer a comprehensive picture of the range of activities undertaken by the SESS-BL, and of the contribution that the knowledge generated by the SESS-BL is expected to make to a number of public policy goals and objectives. It also considers the fact that the full achievement of the SESS-BL's expected outcomes unfolds over a long period of time, well beyond a five-year period. As such, the goal of the evaluation was to offer a meaningful assessment of the activities undertaken during the targeted five-year period ( to ), as well as a description of the benefits realized and outcomes achieved from activities that have occurred prior to that period.

The following sub-sections describe each of the methods used to address the evaluation questions.

3.2.2 Document, performance and financial data review

The document and literature review informed all evaluation questions. The document review portion of this task included both information provided by the CSA on the SESS-BL, as well as documents identified or provided by other stakeholders, particularly as a result of the interviews conducted as part of the evaluation. The literature review portion of this task focused on the relevance of the SESS-BL and covered activities undertaken by other countries in scientific missions and activities in which Canada has also been involved. Since the SESS-BL aims to produce new knowledge, this task also included a bibliometric analysis that was prepared for the CSA.

As it relates more specifically to performance data, the CSA approved the SESS-BL's Performance Measurement Strategy in . As such, some but not all performance indicators had been documented at the time of the evaluation. The performance information also included a variety of operational documents, administrative data, and financial information.

3.2.3 Key Informant Interviews and case studies

Key informant interviews contributed to the in-depth understanding of SESS-BL activities, including results achieved and challenges faced by key stakeholders. These interviews also corroborated, explained, or further elaborated on findings from other data sources. A total of 63 individuals were consulted (through both individual and group interviews) from seven different stakeholder groups. The distribution of these interviews is included in Table 2.

Table 2: Distribution of interviews conducted as part of the SESS-BL evaluation
Key informant groups # of individuals
CSA 12
Academia 10
Federal departments and agencies (other than the CSA) 11
Industry 9
Advisory Committee Members 9
Other space agencies (NASA, ESA) 6
Other space organisations (NOAA, NSF, UN) 6
Total 63

A total of 14 of the interviews conducted focussed more specifically on three missions selected to act as case studies. These missions are SWOT, SWARM, and OSIRIS. The criteria used to select these missions included the date that the missions were launched (to include both older and newer missions), the nature of the Canadian participation, and the sub-field of the Sun-Earth system to which they belong (space weather, atmospheric sciences, and Earth system sciences). The purpose of these interviews was to focus on specific missions (rather than Sun-Earth activities more generally) to allow for a more in-depth understanding of the implementation and results associated with these missions. The information gathered as part of these interviews is included throughout this report, as applicable.

3.2.4 Limitations

This section describes the limitations encountered in the evaluation and how they were addressed.

Nature and scope of SESS-BL activities

As previously noted, the scientific exploration of the Sun-Earth system is a highly complex and multi-dimensional process that involves a wide range of stakeholders around the world, and that unfolds over an extended period of time. In this context, isolating the SESS-BL's contribution, particularly those activities carried-out under the SESS-BL over a five year period, was bound to raise some challenges. As noted in sub-section 3.2.1, the decision was made early on to approach the evaluation at two levels. First, the evaluation would document the range of activities in which the SESS-BL has been involved over the targeted five years, providing a better understanding of the nature and scope of Canada's participation in this world-wide endeavour. Second, the evaluation would also serve to document some of the applications of the scientific knowledge gained to date on the Sun-Earth system that manifested themselves during the evaluation period, even if such applications were based on activities that had occurred well before the strict evaluation period covered by the evaluation. This would provide a better understanding of the scope and nature of the impact of the knowledge gained on the Sun-Earth system.

Performance measurement strategy development

The SESS-BL's performance measurement strategy was developed and approved in . Consequently, the data collected at the time of the evaluation addressed only some of the performance indicators. In addition, the CSA was in the process of implementing the new reporting structure established by the Policy on Results (Government of Canada, 2016c), particularly through the required Performance Information Profiles, which provide a new structure for gathering and reporting performance information. As a result, the evaluation gathered all available administrative data and proceeded with the required analysis to be included in this report.

Most interviewees have a vested interest in the Sub-Program

This limitation was mitigated by requiring interviewees to explain their perspectives and provide examples where appropriate. In terms of the overall report, the findings from the key informant interviews were triangulated with findings from other data sources.

4. Evaluation findings

This section of the report describes the evaluation findings. It first explores the relevance of the SESS-BL, before turning to the performance of the SESS-BL.

4.1 Relevance

4.1.1 The rationale for the SESS-BL

Finding: The SESS-BL provides the overall framework and vision for CSA's involvement in scientific missions concerning the Sun-Earth system. It guides both activities undertaken within the CSA, and the collaboration between the CSA and external stakeholders in Canada and internationally. (Evaluation question: Relevance #1)

Canada has historically collaborated with other countries and with international organizations in enhancing the global knowledge of the Sun-Earth system, and in expanding the operational capacity to better forecast and mitigate the impact of events occurring within that system. For instance, after more than 15 years, Canada's ACE/SCISAT mission continues to provide insights on the ozone layer depletion (CSA, ); scientists at Environment and Climate Change Canada (ECCC) have been collaborating with agencies such as EUMETSAT in Europe or NOAA in the United States to enhance space-based solutions used to forecast and monitor weather and other environmental patterns (ECCC, ; EUMETSAT, ). Canada is also actively engaged in policies and initiatives on climate change, such as the WMO's Global Climate Observing System (GCOS) that monitors Essential Climate Variables (ECV) (World Meteorological Organization, 2018a) or the WMO's Global Framework for Climate Services that supports the integration of science-based climate information in ongoing policy and program planning and implementation (World Meteorological Organization, 2018b). Canada has also joined other countries in enhancing the scientific knowledge on space weather, and on the resulting impacts that may derive from solar storms.

In fact, during the period covered by the evaluation, a higher level of global mobilization emerged around the need to more systematically address the potential impacts of solar storms and of Near-Earth Objects (NEO), such as asteroids or comets whose orbital paths may affect the Earth. In , ESA renewed the funding for its Space Situational Awareness Programme, which focusses on space weather, the monitoring of NEO, and space surveillance and tracking, with a view of enhancing Europe's capacity to "watch for objects and natural phenomena that could harm satellites in orbit or infrastructure such as power grids on the ground" (European Space Agency, ). In , the United States tabled its National Space Weather Strategy (National Science and Technology Council, 2015b) and its companion National Space Weather Action Plan (National Science and Technology Council, 2015a) with a goal of enhancing the capacity of the United States monitor and respond to space weather and other related events occurring within the Sun-Earth system. The Strategy notes that "unlike terrestrial weather events (e.g., a hurricane), space weather has the potential to simultaneously affect the whole of North America or reach even wider geographic regions of the planet", and that capabilities in observing and forecasting space-weather events "depend on international cooperation and coordination" (National Science and Technology Council, 2015b, p. 3).

The need for a sustained effort to monitor space weather and NEO and for adequately preparing for their potential impacts has driven a number of space missions and instruments, including the SWARM mission led by ESA and launched in , the ePOP instrument included on CSA's CASSIOPE satellite launched also in , the recently launched GOLD mission led by NASA (NASA, 2018c) and the upcoming SMILE mission led jointly by ESA and the Chinese Academy of Sciences (to be launched in ) (European Space Agency, 2018a).

In this context, Canada's involvement in activities concerning the Sun-Earth system relies to a great extent on the CSA, which in turn, collaborates with other federal departments and agencies, and with other space agencies and institutions around the world. The SESS-BL provides the overall framework within which CSA's activities on the Sun-Earth system are carried-out. As previously mentioned, other programs of the CSA may include activities that concern the Sun-Earth system, but only the SESS-BL is specifically dedicated to this field of space activities. Moreover, the SESS-BL is expected to support and coordinate all programs and activities of the CSA that have a connection to the Sun-Earth system. In the absence of the SESS-BL, the CSA does not have another mechanism to coordinate its scientific activities respecting the Sun-Earth system.

4.1.2 The SESS-BL's alignment with government and CSA priorities

Finding: The SESS-BL is directly aligned with the priorities of the federal government on climate change and on the monitoring and mitigation of the impacts of space weather on public and private infrastructures on Earth and in space. It also supports the commitment of the federal government to facilitate the integration of scientific knowledge in policy and programming decisions. (Evaluation question: Relevance #2)

For the purpose of this evaluation, the alignment of the SESS-BL with government and CSA priorities is considered at a number of distinct levels.

Starting with space weather, it is now recognized as one of the natural hazards in Canada being monitored by Natural Resources Canada (NRCan), alongside earthquakes, volcanoes, marine geological hazards, flood, and landslides (Natural Resources Canada, ). NRCan provides ongoing forecasts of geomagnetic activity in Canada and the impact of space weather on technology. It is also operating the Canadian Geomagnetic Observatory Network, which provides data on the Earth's magnetic field (Natural Resources Canada, ), and is managing the Public Safety Geoscience Program, which supports research on risk reduction from the effect of space weather and other natural hazards (Natural Resources Canada, ). While NRCan uses the available operating data on space weather, the CSA, through the SESS-BL, supports research activities that expand the overall scientific knowledge respecting the Sun-Earth system, which ultimately expand the range of monitoring and tracking data becoming available for operational purposes, such as those undertaken by NRCan.

As already noted, Canada is also actively engaged in addressing the various challenges relative to climate change. During the period covered by the evaluation, the federal government, along with provincial and territorial governments, tabled their Pan-Canadian Framework on Clean Growth and Climate Change, which provides a broad action plan on reducing greenhouse gas emissions and on adapting to current and expected climate changes (Government of Canada, 2016a). As these governments pursue specific goals in the reduction of greenhouse gas emissions, operational data is required for monitoring purposes. The commercially available space-based data on greenhouse gas emissions offered by the Canadian firm GHGSat, or the data provided by the TROPOspheric Monitoring Instrument launched in on board ESA's Sentinel-5 Precursor are some examples of ongoing space-based data that complement other measurement tools used by governments, industry and other stakeholders involved in climate change initiatives. In this context, atmospheric and Earth system sciences are expected to support climate change policies, programs, and activities by broadening the scientific knowledge on a wide range of related phenomena, and by paving the way for additional monitoring tools and data to emerge. The SESS-BL allows the CSA to specifically engage in scientific research activities that ultimately support actions on climate change.

The evaluation also provided an opportunity to assess the extent to which the range of activities undertaken through the SESS-BL are aligned with the research priorities of Canadian scientists active in Sun-Earth system sciences. Taken as a whole, the range of missions in which Canada has been participating, along with the funding opportunities provided by the CSA, including the announcement of opportunities issued through the SESS-BL, have allowed Canadian researchers and scientists to pursue what they consider to be research priorities. During the interviews conducted as part of this evaluation, academics have pointed to missions such as SMAP, MOPITT, OSIRIS and ACE/SCISAT as examples of missions that are well aligned with their research priorities. They also noted the important contribution made by the CSA through its support for ground-based and stratospheric balloon activities, including calibration and validation activities, which complement satellite-based missions and activities.

Looking forward, the decision by the CSA to establish, in , three science advisory committeesFootnote 3 to "support the planning, validation and updating of long-term roadmaps and program strategies for the CSA" (CSA, ) was well received by researchers and scientists consulted as part of this evaluation. As they assemble experts from academia, government departments and agencies, and the CSA, they are seen as an important contributor to an ongoing dialogue that can facilitate the alignment of research priorities among these stakeholder groups. The establishment of these committees is also aligned with the importance given to science in informing decision-making within the federal government (Government of Canada, ).

4.1.3 Appropriate distribution of roles and responsibilities

Finding: The SESS-BL supports a specific range of activities that falls within the mandate of the CSA, and which complement what other funding agencies are offering. Evaluation findings did not unveil cases of overlap or duplication with activities undertaken by other stakeholders engaged in scientific activities on the Sun-Earth system. (Evaluation question: Relevance #3)

At a fundamental level, and in accordance with section 4 of the Canadian Space Agency Act, the CSA has a legislative mandate to "advance the knowledge of space through science", which is the primary purpose of SESS-BL activities. While other departments and agencies, such as ECCC, Agriculture and Agri-Food Canada (AAFC) or NRCan, are also involved in activities pertaining to the Sun-Earth system, they typically pursue operational goals, such as the monitoring or the ongoing measurement of phenomena occurring in the Sun-Earth system. Other funding agencies, such as the Natural Sciences and Engineering Research Council (NSERC) or Canadian Foundation for Innovation (CFI) may also support fundamental research that relate to the Sun-Earth system, but they do not directly pursue the goal of advancing this type of scientific knowledge. Consequently, in the absence of the SESS-BL, there would be no other program in Canada whose primary purpose is to allow Canadian researchers and other Canadian space stakeholders to participate in advancing fundamental scientific knowledge specific to the Sun-Earth system.

By engaging key stakeholders through mechanisms such as the three CSA scientific advisory committees, by participating in interdepartmental bodies, such as those led by ECCC on climate change, or by engaging in academic coordinating bodies such as the Atmosphere-Related Research in Canadian Universities (ARRCU) or the Division of Atmospheric and Space Physics (DASP) of the Canadian Association of Physicists, the SESS-BL further ensures that the activities it undertakes plays a complementary role. During interviews conducted as part of this evaluation, stakeholders emphasized the fact that the personnel assigned to the SESS-BL offers unique skills and perspectives that can facilitate the coordination of efforts in Canada on the wide range of scientific activities that fall within the realm of the Sun-Earth system.

4.2 Performance

4.2.1 The selection of SESS-BL missions and activities

To adequately assess the performance of the SESS-BL, it is helpful to clarify how its missions and activities are planned and selected in the first place. Evaluation findings indicate that three main drivers must be considered in that regard:

  • First, the CSA may decide to lead an entire mission, such as the ACE/SCISAT mission launched in . In that case, the CSA has been responsible for the satellite itself and for one of the instruments onboard (while another instrument was development by ECCC); the mission's Principal Investigator and other scientists are from Canadian universities, and the satellite and payloads were developed by the Canadian space sector. A total of approximately $100 million has been invested to date by Canada in this mission.
  • Second, the Canadian government, or the CSA more specifically, may be approached by international partners to contribute to a foreign scientific mission respecting the Sun-Earth system, through the development of an instrument, or through other activities such as calibration and validation activities. In such cases, Canadian scientists and/or industries may be solicited to contribute to such missions.
  • A third and somewhat related avenue is when Canadian scientists or industries proactively approach foreign partners to engage in new scientific activities on the Sun-Earth system. If such projects are seen as being aligned with current research priorities, and funds are available, the CSA may support these initiatives.

In addition, the availability of new scientific data, resulting from missions in which the Canada may or may not have been directly involved, allow Canadian scientists to proceed with new analyses that will contribute to the overall body of knowledge produced in connection with the Sun-Earth system. The CSA may fund such analyses through announcements of opportunities.

Ultimately, the level of financial resources available to support missions and activities concerning the Sun-Earth system largely dictates the extent to which any of the above-mentioned avenues can be pursued. During the period covered by the evaluation (- to -), the CSA did not proceed with significant investments supporting new missions. As further discussed in sub-section 4.2.2, the SESS-BL focussed on supporting ongoing missions or missions that became operational during the evaluation period.

4.2.2 Activities undertaken during the evaluation period

Finding: During the evaluation period, the SESS-BL has provided financial support for a range of activities that cover the various cycles of scientific missions, from concept studies to data utilization. (Evaluation question: Performance #4)

4.2.2.1 Provision of financial support

As noted in sub-section 2.2.3 of this report, CSA invested a total of $42.3 million in the SESS-BL over the five years covered by the evaluation. As illustrated in Figure 4, these resources were predominantly invested in contracts, as well as in grants and contributions (G&C), in support of missions and activities in various forms of development. Some investments were also directed toward memoranda of understanding (MOU) with other federal departments and agencies, as well as operations and maintenance (O&M).

Figure 4: SESS-BL Program spending by key categories
- - - - -
Contracts 6,237 3,633 4,029 2,551 2,895
G&C 1,890 2,665 3,333 3,156 4,791
MOU 637 382 329 428 380
O&M 1,026 1,154 984 920 810

Source: CSA financial data

More specifically, the SESS-BL was used to support the following activities:

  • Concepts and Phase 0 studies: The CSA regularly funds concept and Phase 0 studies concerning the Sun-Earth system. These studies provide an initial assessment of the feasibility and of the required technologies to carry out potential missions. For instance, prior to the evaluation period, in and , the CSA funded six concept studies under the Atmospheric Processes of Climate and its Changes (APOCC) initiative, including the Thin Ice Clouds in Far IR Experiment (TICFIRE) and the SnowSat instrument, both of which later progressed into the science and technology development phase (CSA, ). During the evaluation period itself, the SESS-BL provided financial assistance for concept studies on some potential missions and activities, such as the Chemical and Aerosol Sounding Satellite (CASS), which is meant to serve as a follow up to ACE/SCISAT in exploring some dimensions of the ozone depletion, or the Canadian Atmospheric Tomography System (CATS), which is meant to be used as a follow-up to the Optical Spectrograph and InfraRed Imager System (OSIRIS) that observes atmospheric gases and aerosols. Some of these projects, including CATS, have moved into science and technology development, funded mainly through the STD initiative.
  • Project development: During the evaluation period, a few projects moved into project development phases (Phases A to D).Footnote 4 It includes, for example, a major component of the main instrument to be included in the upcoming SWOT mission (surveying and monitoring the oceans and freshwater bodies around the globe), the validation and calibration algorithms used by Canadian scientists in support of the SMAP mission measuring the amount of water in the top 5 cm of soil on the Earth's surface, the Canadian instrument on the SWARM mission focussing on the measurement of the magnetic field, as well as the ePOP instrument on Canada's CASSIOPE satellite focussing on space weather.
  • Mission operations, including data handling, and data utilization: The largest investments made through the SESS-BL during the evaluation period were associated with the operations and data utilization for ongoing missions and activities. This covered a range of missions, some of which having been launched back in in the case of the MOPPIT instrument included on the Terra satellite (focusing on clouds, water vapor, aerosol particles, trace gases, and terrestrial and oceanic properties), the OSIRIS instrument launched in (focussing on the ozone layer), the CloudSat mission launched in (focussing on the composition of clouds, rain and snow), and the Canadian ACE/SCISAT mission launched in (focussing on the ozone layer). This category of funding also included activities under the Geospace Observatory (GO) Canada initiativeFootnote 5, which funds ground-based instruments exploring the region of geospace above Canada to conduct scientific research on space weather and the transformation of this resulting scientific knowledge into applications, such as an increased resilience of Canadian infrastructures to space weather phenomena.

Looking at the actual level of investments provided by the SESS-BL during the evaluation period by key missions and activities, Figure 5 indicates that close to two-thirds of these investments were directed towards the operations and use of SCISAT ($11.3 million) and the funding provided through the GO Canada initiatives ($9.3 million). In the latter case, funding is provided both for the design of the instruments and data component, which provides the infrastructure of Canada's geospace observatory, and the science and application component, which covers activities such as computer-based modelling and the development of applications. Announcements of opportunities for GO Canada were issued in (instruments and data component) and in (science and applications component).

In , the SESS-BL has also funded two announcements of opportunities aimed at supporting the use of space-based data concerning the Sun-Earth system. The first of these announcements targeted the field of Earth system sciences, and provided funding for projects carrying out scientific investigations that "will help advance understanding of physical and chemical processes of the atmosphere as well as Earth-surface processes that affect atmospheric composition" (CSA, 2016c). These projects use a variety of data sources, including those from the SCISAT satellite, the MOPITT and OSIRIS instruments, and other satellites from international partners, and are expected to improve forecasting and prediction capabilities in managing climate change. The second announcement of opportunity targeted the field of Solar-Terrestrial sciences, and provided funding for scientific investigations on physical processes occurring in the geospace that generate space weather and for the development of models that will support "improved forecasting and now-casting capabilities that will contribute to increased resilience to the impacts of space weather" (CSA, 2016d).

Figure 5: SESS-BL funding by key missions and activities
Contracts G&C + MOU
SCISAT
(op and data)
10,207 1,112
GO Canada 2,944 6,385
Themis 2,276  
SMAP   2,187
OSIRIS 1,668  
SWARM   1,360
ePOP   1,440
SWOT   880
Earth
system AO
(data)
  884
Solar-
Terrestrial
AO (data)
  720

Source: CSA financial data

4.2.2.2 Other types of activities

As noted in the program description, the personnel assigned to the SESS-BL also provides matrix support to other programs of the CSA, such as STD and STEDiA, where in some cases they act as internal clients or scientific authority. The group also liaise with other federal departments and agencies, and with international partners. These types of activities have occurred during the evaluation period, including in particular the work in support of the three advisory committees.

4.2.3 SESS-BL outcomes

Finding: The SESS-BL has engaged scientists and the space industry in missions that have produced scientific knowledge on the Sun-Earth system that has since been operationalized to support the ongoing delivery of public programs and activities. (Evaluation question: Performance #4 and 5)

In light of the range of activities that have been undertaken through the SESS-BL, this sub-section explores the extent to which the SESS-BL has achieved its expected results. In doing so, the report first addresses how the SESS-BL has facilitated the engagement of Canadian scientists in enhancing the overall knowledge of the Sun-Earth system. The report also examines the impacts of SESS-BL activities on the Canadian space industries. Finally, the report provides insights on various applications of the scientific knowledge on the Sun-Earth system in managing public programs in Canada.

When considering these broader results, it is important to acknowledge that space missions and activities on the Sun-Earth system are assessed as a whole. In other words, Canada may provide instruments onboard foreign satellites, or carry out validation and calibration activities in support of a space-based mission, but when it comes to assessing the impact of these Canadian contributions, the entire missions within which these activities have occurred are considered, as they ultimately provide the new data and knowledge on the Sun-Earth system that is sought. This simply reflects the fact that, when it comes to Sun-Earth system sciences, international collaboration and cooperation are paramount, and any resulting impacts must be considered with this premise in mind.

Also, as mentioned in the description of the methodology used for this evaluation, the range of activities considered for this assessment of expected results is not limited to activities that have been initiated during the evaluation period. It takes into account activities that have been implemented prior to the evaluation period, but whose results have been experienced during the evaluation period.

4.2.3.1 Engagement of Canadian scientists

Evaluation findings confirm that Canadian scientists operating either in an academic or government setting, are actively engaged in scientific research pertaining to the Sun-Earth system. With the support of the CSA, these researchers have participated in a number scientific teams assigned to space missions and activities, including acting as Principal or Co-Principal Investigators. For example:

  • Government scientists from ECCC, AAFC, and the Department of Fisheries and Oceans (DFO) have participated in science teams for a number of missions, including Canada's ACE/SCISAT mission, the SWOT mission (led by NASA and the CNES), the SMAP mission (led by NASA), the CloudSat mission (led by NASA), and the upcoming EarthCare mission (led by ESA and Japan). A number of MOU signed between the CSA and these federal departments and agencies have facilitated their participation.
  • University scientists have also played a significant role within scientific teams supporting Sun-Earth system scientific missions, as Principal Investigator or as member of scientific teams. Their involvement includes Canada's ACE/SCISAT mission, the SWARM mission (led by ESA), the SMAP mission (led by NASA), the ODIN mission with Canada's OSIRIS instrument (led by Sweden), the CloudSat mission (led by NASA), the ePOP instrument included on Canada's CASSIOPE mission, the THEMIS mission (led by NASA) and the SWOT mission (led by NASA and the CNES). Again, the CSA has facilitated the participation of these scientists through funding assistance.

In addition to participating in scientific mission teams, Canadian researchers have also used available data on the Sun-Earth system to produce studies and analyses. These research activities have been facilitated, in part, through financial support provided by the CSA, including the two announcements of opportunities described earlier in sub-section 4.2.2.1.

During the evaluation period, scientists who received funding from the CSA produced more than 1,200 articles published in peer-reviewed journals, in addition to other publications, and participated in a number of conferences, seminars and workshops. Figure 6 provides further details on these activities.

Figure 6: Number of publications and related activities undertaken by researchers funded by the CSA during the - to - period
Number
Peer-reviewed
publications
1,248
Other
publications
263
Conferences, seminars,
workshops
1,612
Other
presentations
156

Source: CSA administrative data

The assessment of CSA's contribution must also include the range of scientific data that it makes available for any scientist and researcher around the world to use to support their own analyses and publications concerning the Sun-Earth system. As indicated in Figure 7, more than 2,000 articles were published between and that used, in some capacity, data produced by the Sun-Earth system initiatives of the CSA, including both ground-based data, and satellite data.

While these numbers provide helpful insights, it is important to emphasize that they may not represent the entire set of publications that have used data produced by Canadian missions and instruments. This is due to the fact that this data is openly available, and only when a researcher formally acknowledges having used this data, can it be included in these statistics. As noted repeatedly during interviews conducted as part of the evaluation, this is a challenge that all space agencies are facing when it comes to assessing the extent to which their data has been used by scholars and other published researchers.

Figure 7: Number of publications by the international scientific community using data from CSA missions and activities
Number
GO Canada 709
Themis 479
SCISAT 219
SWARM 199
SMAP 188
MOPITT 168
OSIRIS 74
ePOP 23

Source: CSA administrative data, for the period of to

Another important measure of the Canadian scientists' contribution to the knowledge base regarding the Sun-Earth system is provided by the bibliometric study that the CSA conducted in , which covers the period between and . One of the primary findings that emerged from this study, and that is particularly relevant for the purpose of this evaluation, relates to the extent to which publications on the Sun-Earth system from Canadian scientists are used by other researchers around the world. One indicator used to measure this impact is the Average Relative Citation (ARC). Put simply, the ARC "measures the impact of publications produced by a given entity as reflected in citations. An ARC score over 1 indicates that the entity publishes publications that are more highly cited than the world average" (Council of Canadian Academies, , p. 3).

The data presented in Figure 8 indicate that, on average, publications from Canadian scientists are more cited when compared to the world average, and they also tend, on average, to be more cited than publications produced by researchers from the OECD countries considered as a whole.

Figure 8: Average of relative citation (ARC) of Canadian and OECD publications in Space Science (-)
Canada OCDE
All space
sciences sub-fields
1.60 1.17
Solar-Terrestrial
Science
1.03 1.04
Earth-Surface
Science
1.50 1.39
Atmospheric
Science
1.54 1.29

Source: CSA bibliometric study

As noted during interviews, the current trend towards open-data as applied to space missions and activities on the Sun-Earth system is seen as a determining factor in facilitating the engagement of scientists. At this point, data sets from NASA and ESA, as well as data produced through Canadian missions and activities are readily made available to researchers at no cost. In particular, the NASA's Atmospheric Science Data Center is hosting a range of data sets concerning the Sun-Earth system, including the data from the Canadian MOPITT instrument (NASA, 2018d). This trend, however, is also raising significant challenges for all space agencies involved. While NASA is regularly cited as an example when it comes to making data available for research purposes, representatives from NASA who were consulted as part of this evaluation emphasized the significant technological and financial requirements that are associated with this open-data policy. Data sets produced by space missions and activities are practically unmatched in size, and tend to expand significantly due to the long-term nature of these missions and activities.

At the time of the evaluation, the CSA did not have the capacity to host data sets resulting from the missions and instruments it supports. This data is either hosted on systems managed by other agencies such as NASA, or they rest with academic institutions involved in these project. For instance, data from the ACE/SCISAT mission is hosted by the University of Waterloo, while the data from OSIRIS is hosted by the University of Saskatchewan. As the federal government as a whole is moving towards open data and the full implementation of its Directive on Open Government (Government of Canada, ), the CSA was, at the time of the evaluation, exploring a number of options to facilitate access to all its data, including those from space missions and activities.

4.2.3.2 Engagement of the Canadian space industry

Missions and activities related to the Sun-Earth system directly engage the space industry, including a number of Canadian companies that have participated in concept and Phase 0 studies, or that have built satellites and instruments used in missions and activities concerning the Sun-Earth system. The list of these companies includes COM DEV – Honeywell, MDA, ABB, Communications & Power Industries (CPI), Magellan Aerospace, along with many other small and medium enterprises involved in software development and other support services. Funding for these activities has come from various programs within the CSA, including but not limited to the SESS-BL.

Interviews conducted with industry representatives provided insights on the impacts of their involvement in these activities. Four themes emerged from these interviews:

  • Participation in concept studies or in building actual satellites, instruments, or components thereof, provide space companies with the opportunity to mature their technologies. This may be ultimately applied to the mission or instrument under consideration, or it may be applied as part of other opportunities where the same types of technologies are being considered.
  • Technologies that end up being used as part of a mission acquire space heritage, which becomes a significant valued asset in positioning space companies. As noted by one key informant, "getting a contract for a Canadian mission on the Sun-Earth system was an important breakthrough; it provided us with space heritage that we were able to promote in other countries and to other space agencies, and it led to new contracts".
  • Concept studies and technology development activities provide meaningful opportunities to attract new employees and to provide them with training opportunities, therefore contributing to the development of highly qualified personnel (HQP) in the Canadian space sector.
  • Finally, for those companies that are involved in a variety of sectors (beyond space), their participation in missions and activities on the Sun-Earth system has provided them with spin-off opportunities where they could apply new technologies developed in the context of Sun-Earth scientific activities to other sectors.
4.2.3.3 Contributions to public policies and programs

One of the primary drivers behind scientific missions and activities regarding the Sun-Earth system is to allow communities to better manage a number of phenomena occurring within that system, such as climate change, solar storms or NEO. It is widely recognized that the continuous journey from fundamental research, to scientific knowledge, and to policy or program applications unfolds both at multiple levels and over extended periods of time. As one scientist noted during the interviews conducted as part of this evaluation, "space weather today stands where weather forecasts stood 30 to 40 years ago; our goal is to significantly improve, over the next 20 to 30 years, the accuracy of space weather predictions". During that same period, advances in atmospheric and earth system sciences are also expected to pave the way for strategies to deal with climate changes and other related phenomena.

Keeping this in mind, the evaluation provided an opportunity to better understand how the transition from scientific knowledge to policy and program applications is currently experienced. While this sub-section presents some of these applications, it is not meant to be an exhaustive list of such applications.

Improving weather forecasting in Canada

Two scientific missions, SMAP and SMOS, have been providing data that are now moving into operational applications. Led by NASA, with calibration and validation activities carried out by Canadian teams, the SMAP scientific mission was meant to enhance our understanding of soil moisture, particularly in the top 5 cm of soil. The mission has been operating since . A few years before that, in , the Centre national d'études spatiales (CNES) in France and the Center for Technological and Industrial Development (CDTI) in Spain collaborated to launch the scientific mission SMOS, where Canadian scientists have also collaborated through calibration and validation activities. The mission has been exploring a number of critical Earth system variables, including soil moisture and ocean salinity.

The ECCC's Meteorological Research Division has been including SMAP and SMOS data to improve the Canadian Land Data Assimilation System (CalDAS). According to ECCC representatives consulted as part of the evaluation, the inclusion of this additional data has improved weather forecasting by 10 to 15 percent. At the time of the evaluation, ECCC was in the process of making SMAP and SMOS data fully operational by the end of , which will impact ECCC's predictions concerning precipitations, surface temperatures, and clouds, as well as its weather forecasting.

This new dataset will also improve hydrology predictions, such as river flows, and flood forecasting models, which will better inform public authorities that have to deal with the resulting impact of floods.

Monitoring and mitigation of weather impacts on agriculture

The agriculture sector being particularly affected by weather patterns, there have been considerable efforts around the world to improve the ability to forecast and manage unfavorable weather conditions that may lead to drought or floods. Just like ECCC, AAFC became particularly interested in the scientific data emerging from the SMOS and SMAP. At the time of the evaluation, AAFC's National Agroclimate Information Service had integrated information from these two missions into its risk assessment methods and techniques. Among other things, it is now being used to support the management of compensations offered to producers who cannot seed as a result of water excess in soil.

At the other end of the spectrum, AAFC has also turned to space-based data to monitor draught and its impact on the agriculture sector in Canada. To this end, data from the SMAP mission is fed into the AAFC-operated Canadian Drought Monitor (CDM), which is "responsible for providing monthly assessments of drought for Canada that feed directly into the North American Drought Monitor, a cooperative effort between drought experts in Canada, Mexico and the United States" (AAFC, ). Information from the CDM supports business risk management activities within AAFC and crop insurance companies.

Managing electric grids using space weather information

In the period that followed the black-out in Québec, Hydro-Québec engaged in a number of activities to enhance its ability to monitor and manage solar storms that could affect its electrical grid. While other regions of the world are directly affected by solar storms, a number of characteristics found in Québec render this territory particularly vulnerable. Among other things, it is positioned on a large rock shield that prevents the electric current from solar storms to flow through the earth, pushing it towards a power grid that has relatively long transmission lines, which further increases the grid's vulnerability (Hydro-Québec, ).

A task force was established shortly after the black-out, which proposed a number of corrective measures that covered both the monitoring of solar storms and the management of their effects on Hydro-Québec's infrastructures. One of these measures was the "establishment of real-time alert system that measures disturbances on the power grid during magnetic storms" (Hydro-Québec, ). At the time of the evaluation, this monitoring system was including data from Space Weather Canada (NRCan), from NOAA in the United States, and from Solar Terrestrial Dispatch, a private space weather services company based in Alberta. The CSA has been supporting the Space Weather Canada initiative, particularly through the funding it provides to scientists in Canadian universities who design and operate instruments gathering relevant data that on space weather (CSA, 2017a). Hydro-Québec was also using magnetometers to monitor electric currents in the ground that result from solar storms.

Since then, there has been a constant monitoring of solar storm activities at Hydro-Québec and protocols are in place to make operational changes and adjustment based on the level of solar activity experienced. These different measures have led to significant improvements in the ability of Hydro-Québec to maintain the integrity of its electrical grid. As evidence, there have been intense magnetic storms that have hit the electrical grid in the province since , but the measures in place have allowed the service to remain undisturbed (Hydro-Québec, ). As noted during interviews conducted during this evaluation, the data on space weather have played a significant role in preventing other black-outs, but there are still many gaps remaining in the range of space weather information that a corporation like Hydro-Québec requires.

In addition, Hydro-Québec is actively monitoring data on climate change, which also affects the electrical grid in Québec. To this end, the organisation has joined Ouranos, a non-profit organization that supports a network of 450 researchers, experts, practitioners and policy-makers in Québec who are involved in monitoring and mitigating climate change.

Monitoring international commitments on climate change

As previously noted in sub-section 2.1.1, data from the ACE/SCISAT mission and from the OSIRIS instrument are among the range of dataset on the Sun-Earth system that is currently used by the WMO for its assessment of the ozone layer done every four years in support of the implementation of the Montreal Protocol on Substances That Deplete the Ozone Layer. This type of data is also being used to monitor a range of other commitments. This is the case, for instance, with the reports produced every seven years by the WMO's Intergovernmental Panel on Climate Change in support of the UN Framework Convention on Climate Change, which focusses on greenhouse emissions, and in the quality forecasting models being used to monitor the US-Canada Air Quality Agreement and the Canada Clean Air Act (ECCC, ).

4.2.4 Program efficiency

Finding: The CSA has implemented processes that will facilitate the coordination of activities between the CSA and other federal departments and agencies, and the integration of expert knowledge in investment decisions concerning scientific activities on the Sun-Earth system. The CSA is also in the process of addressing the requirements of the Policy on Results, which provides an opportunity to address current gaps in performance measurements. (Evaluation question: Performance #6)

Maximizing benefits derived from SESS-BL investments

In providing the overall framework for CSA's involvement in scientific activities concerning the Sun-Earth system, the SESS-BL seeks to maximize the benefits gained through the resources invested by the CSA in such activities. As with all areas of space, the range of possible missions, activities and projects on the Sun-Earth system in which the CSA could participate far exceeds the current level of resources available. During interviews conducted as part of this evaluation, there was a clear desire expressed by all stakeholder groups to see Canada remain actively involved and, to the extent possible, expand its participation in this field. This, in itself, speaks to the contribution made to date by the Canadian space sector to this important body of scientific knowledge. However, the extent to which the CSA can proceed with new investments in Sun-Earth sciences is a question that needs to be considered at an agency-wide level and, as such, it falls well beyond the scope of this evaluation.

Looking back at the evaluation period, a number of structures and processes have been established by the CSA to support investment decisions, including those regarding Sun-Earth system sciences. In particular, the establishment, in and , of the Deputy Minister's Governance Committee on Space, the Assistant Deputy Minister's Space Program Integration Board, and the Director General's Space Program Integration Board are examples of processes that are facilitating the emergence, among the CSA and other departments and agencies, of a government-wide vision of investment priorities for space missions and activities (CSA, ). As noted during interviews, CSA investment decisions on scientific missions on the Sun-Earth system have tended to be made on an ad hoc basis, based on specific opportunities that have presented themselves. The governing structure implemented during the evaluation period allows for a more systemic and government-wide approach to such planning.

In addition, the establishment of the three science advisory committees within the CSA further supports the planning of current and future activities of the CSA concerning the Sun-Earth system. As these committees assemble scientists and experts in the various fields of Sun-Earth system sciences, they can bring an informed perspective on emerging priorities related to these various fields of scientific activities. As noted during interviews, there are large planning processes in place, such as the Decadal Survey for Earth Science and Applications from Space produced by the National Academies of Sciences, Engineering and Medicine in the United States (NASA, 2018a), which set broad priorities for scientific activities on the Sun-Earth system. Having the three advisory committees in place enhances the capacity of the CSA to monitor and integrate this type of strategic information in its decision-making process.

Monitoring and reporting on program performance

As noted in the methodology section, the SESS-BL's Performance Measurement Strategy was approved in . Since then, the program has documented some performance indicators, but has not produced performance reports based on this data. At the time of the evaluation, the CSA was proceeding with the implementation of Performance Information Profiles for all its activities, and performance information will therefore be tailored to accommodate this new reporting framework.

This evaluation, which was the first conducted of SESS-BL activities, has provided strategic insights on the importance of collecting strong data information. As illustrated throughout the report, the CSA has been involved in a relatively large range of missions and activities on the Sun-Earth system, which have involved multiple stakeholders within and outside the CSA, and which have unfolded over an extended period of time. Being able to successfully monitor and report on the outputs and results of all these activities is critical to fully appreciate the broader impacts and applications of the scientific knowledge being produced.

While the current management structure of the SESS-BL allows for some performance data to be collected, evaluation findings indicate that this data is not consistent over time, and more importantly, it has yet to be fully integrated in an actual monitoring and performance measurement strategy that would provide a performance analysis of the SESS-BL on an ongoing basis. Having such a monitoring and performance strategy in place would not only support the ongoing management of the SESS-BL, but it would greatly support the next evaluation of the SESS-BL.

Integrating Gender-Based Analysis Plus considerations

In , the federal government released its new Policy on Results (Government of Canada, 2016c), along with its associated Directive on Results (Government of Canada, 2016b). This new framework clarifies expectations related to gender-based analysis plus (GBA+). First, it confirms that in establishing their performance measurement strategy, program managers must include, where relevant, a gender-based plus analysis. It also identifies, as a mandatory procedure, that all evaluations be planned to take into account, where relevant, GBA+ analysis.

In , the CSA approved its own policy and procedures governing gender-based analysis, which is based on the concept of GBA+, as defined by the Status of Women Canada (Status of Women Canada, ). This framework explores how "diverse groups of women, men and gender-diverse people may experience policies, programs and initiatives". As such, GBA+ is an analytical tool that is expected to be used to support all programming cycles, from initial design, to implementation and evaluation with a view of ensuring equitable access and benefits.

During most of the period covered by the evaluation, the policy and procedures of the CSA concerning GBA+ were not in place. Consequently, this evaluation is not assessing the extent to which proper GBA+ analysis has been performed as part of the management of the SESS-BL. Moving forward, however, the management of all activities falling under the SESS-BL is expected to take into account the procedures that the CSA has established concerning GBA+. Both the performance measurement activities, and the next evaluation of SESS-BL activities will provide an opportunity to assess the extent to which GBA+ considerations have successfully been implemented into the management of these activities.

5. Conclusions and recommandations

This section of the report concludes by summarizing the key findings of the evaluation, and includes observations and recommendations where applicable. More detailed information, substantiating each of these statements, is included in the preceding sections of the report.

5.1 Relevance

Governments world-wide have recognized the need to adequately monitor phenomena occurring within the Sun-Earth system, and mitigate their potential impacts on Earth, and on space-based infrastructures and assets. Solar storms, NEOs whose orbital path may affect the Earth, and climate change are illustrations of these complex phenomena that call for informed policies, programs and collaborative efforts.

Much has yet to be understood when it comes to the Sun-Earth system, and nations are collaborating in broadening the scientific knowledge of the Sun-Earth system, and in transitioning this knowledge into policy decisions. From a space perspective, this means supporting both scientific missions that uncover new knowledge, and operational missions that feed relevant data on an ongoing basis for monitoring and decision-making purposes.

The CSA has been using the SESS-BL to provide the overall framework and vision for its involvement in Sun-Earth system sciences. As space missions and activities may serve both scientific and operational purposes, the SESS-BL has been focussing on the scientific portion, paving the way for other federal departments and agencies to operationalize these data to support their activities and programs.

The CSA is expected to maintain a central role in supporting the actions of the Canadian government respecting climate change and the protection of public and private infrastructures on Earth and in space. In implementing this role, the CSA is using the SESS-BL to manage a number of activities that are directly funded by the SESS-BL, in addition to supporting the coordination of other activities within the CSA that relate to the Sun-Earth system. The SESS-BL also supports the CSA's coordination efforts with other federal departments and agencies, and other national and international stakeholders. In the absence of the SESS-BL, there would be no other program areas within the CSA whose principal focus is the Sun-Earth system, and the advancement of the scientific knowledge on that system that can benefit the Canadian society as a whole. This work is therefore aligned with the current priorities of the federal government, and with the legislative mandate of the CSA.

5.2 Performance

Activities undertaken by the SESS-BL during the evaluation period

Overtime, the CSA's involvement in scientific missions and activities on the Sun-Earth system has taken various forms, including launching and leading entire missions such as SCISAT, providing instruments and calibration activities as part of missions led by other agencies, participating in scientific mission committees, or supporting the use of data as part of fundamental research activities undertaken by Canadian scientists and researchers.

During the evaluation period, the SESS-BL supported a number of concept and Phase 0 studies, including some that moved into the science and technology development phase. In addition, some instruments, including those included on the SWARM and CASSIOPE missions, or the upcoming SWOT mission, moved through project development phases (Phases A to D). The SESS-BL also provided financial assistance in support of ongoing mission operations and data utilization, particularly in relation to SCISAT and the GO Canada initiative. Finally, the SESS-BL provided matrix support to other programs of the CSA or acted as internal client to these programs.

Engaging Canadian scientists and the Canadian space sector

The evaluation confirms that the SESS-BL has contributed to the engagement of Canadian scientists in a range of scientific missions and activities concerning the Sun-Earth system. These scientists have shared their knowledge and expertise through publications, conferences, and other related activities. The contribution of Canadian scientists to the body of scientific knowledge on the Sun-Earth system is further illustrated by the fact that publications from Canadian scientists are regularly cited around the world. Additionally, missions and activities funded through the CSA have provided scientific data that has been used by scientists and other researchers around the world.

One challenge that Canada faced along with all nations involved in space missions and activities is the management and storage of the data produced through their respective missions and activities. At the time of the evaluation, the CSA was still exploring options to facilitate access to the scientific data it produces, in line with the open data direction adopted by the federal government as a whole, and to find options for storing and managing these large datasets over extended periods of time.

The development and operations of satellites and instruments used in support of Canada's scientific missions on the Sun-Earth system have also engaged the Canadian space sector. The benefits to the industry include the opportunity to mature their technology, to build space heritage, to attract and develop HQPs, and to open-up spin-off opportunities.

Informing public policies and programs

The evaluation provided an opportunity to document some of the policy and program applications realized to date that have been supported by the scientific knowledge acquired on the Sun-Earth system, to which Canada has contributed. The relatively long journey from the identification of a scientific research need to the launching of a scientific mission and the operationalization of the resulting data and knowledge requires that these outcomes be considered over an extended period of time. The improvement of weather forecasting in Canada and of the mitigation of floods and drought on agriculture activities through data from the SMAP and SMOS missions, the integration of space weather information (including NRCan's Space Weather Canada) in the ongoing management of electric grids, and the use of ACE/SCISAT and OSIRIS data to monitor progress on international commitment relating to climate change are some of the key applications that have emerged during the evaluation period.

Maximizing the benefits of SESS-BL investments

With its set level of resources assigned to scientific activities on the Sun-Earth system, the CSA has to make choices that must consider a range of factors such as the priorities set by other space agencies, operational needs of federal departments and agencies, and the needs and capacities of Canadian scientists and the Canadian space industry. During the period covered by the evaluation, the CSA has implemented processes that will support a more systematic consideration of these factors, in consultation with other federal departments and agencies (through senior management committees) and with Canadian experts (through advisory committees). While these advisory committees are expected to support the decision-making process within the CSA, they also present an opportunity to further engage, through knowledge translation and sharing activities, stakeholders who have a direct stake in the Sun-Earth system scientific knowledge and its application. Such activities could focus on targeted missions, new operational applications of scientific data, key policy statements from other space agencies (such as the NASA's Decadal Survey), or other issues as identified by committee members.

At the time of the evaluation, the CSA was implementing the performance reporting requirements established in the federal Policy on Results, which will necessarily cover the set of activities undertaken through the SESS-BL. The evaluation provided an opportunity to better assess the range of performance data collected to date, and in doing so, identified gaps that should be addressed moving forward. In particular, while some performance data has been collected, it has yet to be integrated in an actual performance monitoring and reporting strategy. Such a strategy will not only support the ongoing management and decision-making process respecting SESS-BL activities, but it will also provide considerable support for future evaluations, in addition to enhancing the capacity of the CSA to share information on activities carried-out and results achieved by the SESS-BL with key stakeholder groups in academia, governments, and industry. The CSA advisory committees can support the sharing of this performance information with external stakeholder groups.

The full implementation of the requirements established in the Policy on Results will also provide an opportunity for the CSA to integrate GBA+ considerations in the planning, delivery and reporting of SESS-BL activities.

Since the report does not include recommendations, a Management Action Plan is not required.

Management Response and Action Plan

Responsibility
Organization / Function
Management response Details of action plan Schedule
Recommendation # 1
N/A N/A N/A N/A N/A

Bibliography

Date modified: