Evaluation of the Qualifying and Testing Services Program (David Florida Laboratory) for the Canadian Space Agency

Download the PDF version (2.2 MB)

For the period from April 2009 to September 2014

Project # 14/15 - 02-03

Prepared by the Audit and Evaluation Directorate

December 2014

Table of Contents

  1. Introduction
  2. Background
  3. Evaluation Scope and Methods
  4. Results
  5. Conclusions and Recommendations

List of Tables and Figures

Abbreviations Used in the Report

  • AIAC: Aerospace Industries Association Canada
  • AIT: Assembly, integration and testing
  • CAGR: Compound annual growth rate
  • CARIC: Consortium for Aerospace Research and Innovation in Canada
  • CFI: Canada Foundation for Innovation
  • CRC: Communications Research Centre
  • CRF: Consolidated Revenue Fund
  • CRIQ: Centre de recherche industrielle du Québec
  • CRIAQ: Consortium for Research and Innovation in Aerospace in Quebec
  • CSA: Canadian Space Agency
  • DFL: David Florida Laboratory
  • DND: Department of National Defence
  • DRDC: Defence Research and Development Canada
  • ECG: Evaluation Consultative Group
  • EMC: Electromagnetic compatibility
  • EMI: Electromagnetic Interference
  • ESA: European Space Agency
  • ESTEC: European Space Research and Technology Centre
  • EU: European Union
  • FTE: Full-time equivalent
  • GC: Government of Canada
  • GFE: Government furnished equipment
  • GFS: Government furnished service
  • GoCo: Government-owned Contractor-operated
  • HQ: Headquarters
  • HQPs: Highly Qualified Persons
  • IP: Intellectual property
  • ITAR: International Traffic in Arms Regulations
  • JWST: James Webb Space Telescope
  • kN: Kilonewton
  • NASA: National Aeronautics and Space Administration
  • NRC: National Research Council
  • O&M: Operations and maintenance
  • OGD: Other government department
  • PCW: Polar Communication and Weather Mission
  • PDRP: Professional Development Recruitment Program
  • PIM: Passive inter-modulation
  • PM: Performance Measurement
  • PMO: Project Management Office
  • PPP: Public–private partnership
  • PWGSC: Public Works and Government Services Canada
  • RCM: RADARSAT Constellation Mission
  • RF: Radio Frequency
  • RFQF: Radio Frequency Qualifying Facility
  • S&T: Science and Technology
  • SQF: Structural Qualification Facility
  • TBS: Treasury Board Secretariat
  • TQF: Thermal Qualifying Facility
  • WED: Western Economic Diversification Canada
  • S&F: Security and Facilities

Executive Summary

Background

This report contains the findings of the Evaluation of the Qualifying and Testing Services Program of the Canadian Space Agency (CSA). The David Florida Laboratory (DFL), located in Ottawa, Ontario, is the main mechanism through which the program is delivered. This program carries out specialized activities and services for the assembly, integration and testing (AIT) of space hardware (e.g., satellites and components) on behalf of the CSA and external clients, both in Canada and internationally. The program exists to ensure that mission-assigned technology and entire systems can safely and reliably meet the rigours of space.

The study was conducted between April and September 2014 on behalf of the CSA Audit and Evaluation Directorate by a consortium of firms led by Kelly Sears Consulting Group. The evaluation is a requirement of the CSA five-year evaluation plan, which, in accordance with Treasury Board of Canada Secretariat's Policy on Evaluation (2009) requiring that all federal programs be evaluated every five years. An Evaluation Consultative Group, consisting of representatives from the CSA (St. Hubert and the DFL) guided the work of the consulting team.

The evaluation involved the following methods: a review of a considerable amount of documentation and literature; the conduct of 44 interviews within the CSA and with external DFL clients; and a benchmarking study of eight other AIT facilities in other countries.

Conclusions and Recommendations

Relevance/Need

The objectives of the Qualifying and Testing Services Program (i.e., the DFL) are aligned with federal priorities as outlined in the Science & Technology Strategy (2007), the Space Strategy (2003), and Canada's Space Policy Framework (2014). The program aligns with strategic policy issues related to sovereignty and national reputation on the international stage by maintaining Canada's standing as a Tier 2 country in terms of the development and autonomy of its space program and industry. The program also aligns with policy issues related to employment and industrial development, particularly through its role in supporting the space industry, which is an important driver of research and innovation, and which employs several thousand highly qualified persons (HQPs). Delivery of the program falls within the federal government's jurisdiction via the Canadian Space Agency Act (1990).

Regarding the future need for a government-operated facility such as the DFL, several factors indicate that the level of demand may decrease in the coming years. Although there is evidence of a continuing need for testing services such as that provided by the DFL, the space industry is evolving towards smaller satellites and increased testing in-house on the part of large companies. In addition, the current uncertainty with respect to new space projects at the CSA makes it more difficult to anticipate the demand for testing at the DFL.

There is currently no mechanism within the CSA to identify the testing needs of the space sector (and other sectors such as telecommunications and aerospace) as they evolve. There is also a risk of overlap and duplication of testing services, along with the possibility that certain gaps may exist.

Recommendation 1: The CSA should set up a process for consulting with stakeholders representing industry (space and other sectors as appropriate), other federal departments and agencies, universities and CSA clients to provide guidance on current and future testing needs. This consultation process would serve to minimize areas of overlap and duplication with other government and private facilities.

Achievement of Outcomes

The program has generally been successful in achieving the immediate and intermediate outcomes as depicted in the program's logic model. However, the evaluation was hampered by a lack of quantitative data on performance. A Performance Measurement Strategy has been developed but has not been fully implemented. Many of the performance indicators are not appropriate given the way the DFL operates.

Recommendation 2: The CSA should revise the existing Performance Measurement Strategy to make it more viable and improve performance measurement to better reflect how the DFL operates and the environment it operates in. The client surveys should be modified as necessary to align with the Performance Measurement Strategy.

The program has been able to meet the needs of its clients as demonstrated by high levels of client satisfaction which were confirmed by the many clients interviewed for this evaluation. However federal government-wide restrictions on funding for travel to networking events has meant that program staff are less able to stay abreast of technological and market developments in the space industry. This in conjunction with a lack of sufficient investment in DFL infrastructure may be putting at risk the DFL's standing as a world-class facility.

Ongoing issues with DFL building maintenance pose a significant challenge from the perspective of both the CSA and DFL clients. The original DFL building was built in 1971 and then expanded in 1978, 1985, 1997, and most recently in 2011. The DFL building was not designed so that maintenance could easily be performed on some parts of the building while keeping the rest open and fully operational. Without an extensive redesign of the building there are few options for resolving the issue of periodic shutdowns. The CSA is currently exploring solutions which are within the CSA budget capacity, such as building a parallel distributed power system consisting of two independent power networks, so that operation can continue during system tests and upgrades (and power outages). This would decrease the frequency of complete shutdowns of the DFL.

The challenges related to building maintenance are compounded by frequent slippage in DFL clients' schedules which make planning shutdowns well in advance difficult. Resolving this issue is challenging because, to some extent, such delays are characteristic of satellite projects and as a client service organization it is difficult for the DFL to refuse to accommodate changing client needs. However the CSA does strive to maintain collaboration among its different sectors and clients in order to ensure coordination of maintenance and testing requirements (especially when there are project delays or schedule conflicts) and limit the impact of the shutdowns.

There is some evidence that the lack of sufficient investment in the DFL's human resources and the relatively large number of recent and pending retirements have resulted in the program being unable to maintain human resources at a level to meet demand for testing during peak periods. This has been compounded by the fact that program staff tend not to be transferrable across testing areas. There is a sense that having access to more engineers rather than technicians would alleviate this challenge because engineers are seen as having a more developed skill set. The CSA has recently put in place a recruitment process and developed a training program for new recruits. In the context of the DFL this would include "cross training" all new staff so they are more readily transferrable across testing areas. In addition, there are currently about 200 engineers employed by the CSA. Sharing of engineering staff between the CSA headquarters (HQ) in St. Hubert and the DFL could serve to meet some of the demand for engineers at the DFL.

Recommendation 3: The CSA should explore a program of sharing staff, particularly engineers, between CSA HQ and the DFL. This option may allow the DFL to overcome some of its staffing issues over the medium term, while providing experience to engineers at the CSA.

There is a sense within the CSA and among clients that the DFL is not sufficiently utilized, and, as noted above, there are concerns that utilization may decline in future years. However, analysis of past, current and forecast utilization is hampered by the lack of robust, consistently measured utilization data.

Recommendation 4: The CSA needs to develop and put in place a consistent and robust approach to measuring utilization at the DFL which is consistent with the approach used in other similar facilities internationally.

Based on the DFL's utilization statistics from 2001-02 to 2013-14, there is evidence that over this time period, the DFL's utilization has fluctuated somewhat but utilization levels from 2009-10 to 2012-13 were in fact higher than those for earlier periods. Utilization rates plummeted in 2013-14 as a direct result of the 6-month building shutdown in the 2013 calendar year.

Despite the lack of robust utilization data, there is agreement that the utilization of the DFL should be increased, which would require some increases in resources for hiring technical staff, marketing, etc. There is a sense that in the absence of increased activity at the CSA which could bring in more testing to the DFL, there is only limited potential to increase testing for the Canadian space sector. Key alternate markets for DFL services are believed to be the EU and the US although attempting to access these two markets would present significant challenges. Another option is for the DFL to expand its testing services to other industries, something which the DFL already does to some extent. To date no market research has been done to assess whether these are viable options.

Recommendation 5: The CSA should undertake a study to identify realistic potential markets for the DFL beyond the space sector. Based on the study, implement a strategy in keeping with its mandate as a federal agency to develop a small number of potential markets for the DFL which have long-term potential. This strategy should be appropriately resourced in order to increase the likelihood of success.

There is evidence of a need for increased marketing of DFL services. The DFL is unable to undertake formal marketing because it lacks the human and financial resources to do so. In the past the DFL was able to undertake informal marketing activities including attendance at conferences, workshops and other events where there are potential clients. This approach to marketing is consistent with the approach of similar organizations such as ESTEC and Intespace.

There is evidence that the DFL is meeting clients' needs with respect to the provision of technical information.

Almost all clients indicated that they have used the DFL for at least some of their testing needs and will continue to do so. However, larger firms indicated that they have been moving away from relying on the DFL for all of their testing needs and now increasingly use the DFL for overflow (when their own facilities are fully booked) or when they require larger, more specialized equipment they do not possess. There was general agreement on the need for a national testing facility such as the DFL in order to maintain national autonomy in testing and to ensure access to reliable testing facilities for all firms in Canada, regardless of size. The need to maintain Canada's standing as a Tier 2 country in space was also linked to the need for a national testing facility – without a national testing facility Canada would, in effect, become a Tier 3 country and thus dependent on other countries for testing space hardware.

Efficiency and Economy

Based on available information, the program has delivered its outputs and outcomes efficiently and the program is delivered economically.

A key limitation to addressing efficiency and economy for the Qualifying and Testing Services Program is the lack of consistent and reliable utilization data and cost data against which to assess the achievement of outputs and outcomes. In the absence of quantitative data, the evaluation has relied largely on qualitative information from interviews and documentation.

The DFL costs Canadian taxpayers an average of less than $10M (after factoring in revenues deposited into the CRF) annually over the five-year period of the evaluation. In contrast, the space industry employs several thousand HQPs and generated revenues of $3.3B in 2012. In addition, having a well-equipped state-of-the-art facility available to all Canadian companies means that companies are not leaving Canada to undertake testing and thus contributing to the economies of other nations. Money spent on testing at the DFL along with the related expenditures remains within Canada.

Key areas that affect the DFL's efficiency and economy include human resources, facilities management, pricing, and utilization.

With respect to human resources, the lack of staff that can be moved from one testing area to another hinders the DFL's efficiency because the DFL sometimes lacks sufficient staff to meet demands for testing. The CSA is working to remedy this by ensuring new staff are portable across all testing areas within the DFL. The use of more engineers rather than technicians is also seen as a possible solution because engineers are seen as having a more flexible skill set. Another suggestion made related to reviewing the human resources allocation within the DFL through consolidation of groups.

The inability to shut down only some areas for maintenance is seen as inefficient and adds to the cost of maintenance.

Given the limitations on obtaining pricing information from other testing facilities it was not possible to benchmark the DFL's fee schedule. However there is evidence that the costing model/fee schedule is outdated because neither the internal nor external costing model have been reviewed and updated in recent years.

Recommendation 6: The CSA should review and revise the costing model for the DFL and ensure that it is updated on a regular basis.

Although the lack of consistently measured utilization limits a fair assessment of whether the DFL is adequately utilized relative to other similar testing facilities, there is general agreement that increasing the DFL's utilization would boost efficiency and economy by more effectively using the DFL's facilities and staff.

With respect to an alternative operating model for the DFL, there is evidence that Government-owned Contractor-operated (GoCo) or public–private partnership (PPP) options for operating the DFL merit further investigation. This is currently being done by the CSA in parallel to this evaluation.

1. Introduction

This report presents the findings of the Evaluation of the Qualifying and Testing Services Program of the Canadian Space Agency (CSA). The David Florida Laboratory (DFL), located in Ottawa, Ontario, is the main mechanism through which the program is delivered. The evaluation was conducted on behalf of the CSA Audit and Evaluation Directorate by Kelly Sears Consulting Group in collaboration with Beechwood Consulting and Research Inc., BBMD Consulting Inc., and Hickling Arthurs Low Corp. between April and October 2014. The evaluation is a requirement of the CSA five-year evaluation plan and was conducted in accordance with the Treasury Board of Canada Secretariat's Policy on Evaluation (2009) requiring that all federal programs be evaluated every five years.

2. Background

This chapter provides a brief profile of the Qualifying and Testing Services Program, consisting of the David Florida Laboratory (DFL). The Qualifying and Testing Services Program corresponds to the CSA's Program Alignment Architecture (PAA) sub-program (SP) 1.3.3. This sub-program consists of specialized activities and services for the assembly, integration and testing (AIT) of space hardware and involves space qualifying technology, sub-units, units or entire spacecraft developed by Canadian academic institutions, government organizations, and industry, as well as international partners and clients.

2.1 Program Profile

Officially opened in September 1972, the DFL, located at 3701 Carling Avenue in Ottawa, then consisted of a single bay clean room, integration and storage area, a series of small thermal vacuum chambers, a 53 kilonewton (kN) vibration table, an anechoic chamber, and office space. The DFL was transferred to the CSA in March 1989 upon the establishment of the Agency. As part of the CSA, the DFL provides facilities and environmental testing services to both private and public sector clients on a fee-for-service basis.

According to the program's Performance Measurement Strategy (PM Strategy, March 2014), the mandate of the DFL in delivering the Qualifying and Testing Services Program on a fee-for-service basis is to:

  • Provide ongoing support to meet the objectives of the Canadian Space Program through the qualification of flight hardware;
  • Provide environmental and radio frequency testing at the system and subsystem level to a range of domestic and offshore clients on space-based and terrestrial programs;
  • Manage the ongoing schedule of test support to maximize the quality and integrity of the test environment, and minimize any risk to people, facilities, and the test article's safety;
  • Acquire and develop test technology to provide high-value test services and facilities to effectively support the Canadian Space Program;
  • Provide a mechanism for the transfer to industry of appropriate environmental and radio frequency test technology;
  • Conduct, in conjunction with industry, activities aimed at increasing opportunities for DFL use for Canadian industry and academia; and
  • Ensure the cost-effective operation of the DFL by establishing usage priorities and procedures consistent with optimum use of the facilities. This includes the provision of test support for non-space-related work to achieve uniform facility loading.

The program is necessary to ensure that mission-assigned technology and entire systems can safely and reliably meet the rigours of space. The program serves to demonstrate the suitability and effectiveness of new Canadian space technologies in contributing to space missions and provides an effective base for increasing Canada's capability to participate in future space programs.

In addition to its main facilities, the DFL is supported by an operational infrastructure in keeping with its broad scope of activities. This includes offices, conference rooms, storage areas, check-out rooms and in-house mechanical, electrical and electronic shops for custom design and device construction.

Program clients include other branches of the CSA who manage space projects. For these CSA-led projects, DFL facilities are made available to contractors and subcontractors who are furnishing equipment to the Government of Canada (GC) for assembly integration and testing of space-borne systems, spacecraft, satellites and related subsystems and components. For each project the DFL appoints one or more Program / Technical Representatives as interfaces between the DFL and the client. To facilitate planning and a smooth AIT campaign, the DFL forms an AIT Working Group (WG). These WGs convene meetings on a regular basis to discuss and plan the eventual AIT execution at the DFL. The programs request test services via a test request form, and the DFL tracks the costs as tests are executed. The CSA programs are invoiced at the end of each test.

The DFL also provides testing services to many non-CSA clients,Footnote 1 including clients from the private sector and other federal government departments and agencies. For these external clients, a similar project management mechanism is in place and a fee schedule has been devised, although these have not been updated since 2009-10 for the external fee schedule and 2010-11 for the internal. Revenues collected for testing done for private sector clients are not retained by the CSA but are deposited into the federal government's Consolidated Revenue Fund (CRF).

The DFL is an ISO certifiedFootnote 2 laboratory and adheres to strict quality assurance principles. The ISO certification requires regular internal and external audits of compliance to the ISO standard. The DFL's ISO certification for the laboratory was obtained in 2000, and the clean room was certified in 2009.

Prior to 2012 DFL shutdowns for maintenance and upgrading did not follow any set schedule. Starting in 2012, the DFL has been shut down for approximately two weeks every spring and every fall for regularly scheduled maintenance and upgrading. In addition, the DFL was shut down for the greater part of 2013 and into 2014 in order to perform extensive maintenance and upgrading to the facilities. The work was conducted in three phases:

  • April to September 2013: encompassed all the high bay / clean room facilities.
  • August 2013 to March 2014: covered all the environmental qualification facilities.
  • September 15 to October 31, 2013: consisted of a general maintenance shutdown.

2.2 Governance, Roles and Responsibilities

The federal government is responsible for delivering certain space missions in order to provide services to Canadians. The nature of such missions is determined from time to time in consultation with departments, and the role of the CSA in each case is determined through this process. In most cases other departments or Canadian companies define the mission requirements according to their own mandates, but in some cases the mission mandate is uniquely determined by the CSA. Missions managed by the CSA, for whichever purpose, usually require the services of an outside industrial team to develop the operational system. Development requires verification that space systems are qualified to operate in space, and the requisite AIT services are frequently provided to this team by the CSA through the DFL.

Development of a CSA space mission, including assessment of performance, is the responsibility of a Project Management Office (PMO), defined and identified according to CSA policy on governance. The PMO manages all aspects of the entire life cycle of a mission, including the relationship with users, partners, and stakeholders, industrial contracts, verification, operation, and disposal. The DFL acts as a supplier to the PMO for provision of AIT services, in a manner analogous to other members of the overall development team. In this sense the performance of the DFL is assessed by the PMO on behalf of the CSA and its partners.

The Qualifying and Testing Services Program conducts its business and provides services under the leadership of the DFL Director. The DFL Director reports to the Director General of the Space Science and Technology Branch of the CSA.

For most projects, the program utilizes a tri-partite working group including the client, the prime contractor / builder, and the DFL as the subcontractor for testing to ensure that the DFL meets the client's and prime contractor's needs within the constraints of costs and capabilities. Further, the program has in place a formal process for measuring client satisfaction. The annual operational work plan process is used to list all planned activities, the date on which they will occur or by which they will be accomplished, the resources that will be required, and the person/group responsible for carrying them out.

Security and Facility Management (S&F) within the CSA is responsible for all aspects of facility operations, maintenance, infrastructure improvements and projects as well as electronic and physical security at the DFL facility. S&F is responsible for maintaining and inspecting a broad range of equipment within the DFL including air conditioning and ventilation, cranes and lifts, sprinkler systems, power/electrical systems and building structure.

2.3 Stakeholders, Partners and Clients

Clients relying upon the services of the DFL include other CSA programs, other government departments, the Canadian space industry and academic institutions as well as industrial clients from outside Canada.

Other federal government departments and agencies rely on the DFL for providing testing services. The DFL has been and will be used for testing space systems built by Defence Research and Development Canada (DRDC). Many systems at DRDC and the Communications Research Centre (CRC) undergo testing at the DFL.

Space companies rely on the DFL to qualify their space hardware and to position themselves for bidding on international space programs and projects and for meeting the requirements of resulting contracts.

Universities such as University of Toronto, York University, University of Calgary, University of Alberta and Carleton University have used DFL services for qualification of their systems for space-related projects and for scientific instruments/technologies developed for space research. Other universities such as the Royal Military College, University of Ottawa, and École Polytechnique have not used the DFL for testing but have used the expertise of DFL personnel.

DFL facilities are also utilized by international organizations. For example, specialized facilities at the DFL for passive inter-modulation (PIM) testing are used by European Space Agency (ESA) projects and ESA contractors such as Thales Alenia Espace, Astrium and others for the Galileo Satellite Antennas. The Canadian Space Agency / INMARSAT agreement designates the DFL as INMARSAT's sole authorized aeronautical antenna assessment organization. Many European space systems and components are tested at the DFL, and the DFL is an important component of their supply chain.

There are no other organizations involved in the delivery of the program, as all activities and outputs are delivered by the DFL and other divisions of the CSA.

2.4 Resource Allocation

The total funding and actual expenditures for the program over the five-year period from 2009-10 to 2013-14 are presented below. Expenditures for the DFL include two components, operations and expenditures on security and facilities (i.e., maintenance of the building). The funds spent in supporting the building and infrastructure are covered by internal services, i.e., not directly by the program.

Table 1: DFL Funding and Actual Expenditures, 2009-10 to 2013-14

2009-10 2010-11 2011-12 2012-13 2013-14
David Florida Laboratory
FTEs (#) 40.94 40.91 40.93 37.47 32.7
Salaries ($K) 3,460 3,302 3,362 3,187 3,151
O&M ($K) 1,351 1,897 1,775 1,439 1,580
Capital ($K) 1,678 1,931 1,533 1,657 751
Total Budget ($K) 6,489 7,130 6,670 6,283 5,482
Actual Expenditures ($K) 6,256 6,932 6,434 6,083 5,105
Security & Facilities (for the DFL)
FTEs (#) 3.85 5.7 6.55 7.49 4.65
Salaries ($K) 374 501 517 582 500
O&M ($K) 3,370 3,710 4,030 3,437 3,767
Capital ($K) 1,199 2,124 899 1,351 2,120
Total Budget ($K) 4,943 6,335 5,446 5,370 6,387
Actual Expenditures ($K) 4,875 6,343 5,447 4,902 6,138
Total DFL + S&F
FTEs (#) 44.79 46.61 47.48 44.96 37.35
Salaries ($K) 3,834 3,803 3,879 3,769 3,651
O&M ($K) 4,721 5,607 5,805 4,876 5,347
Capital ($K) 2,877 4,055 2,432 3,008 2,871
Total Budget ($K) 11,432 13,465 12,116 11,653 11,869
Actual Expenditures ($K) 11,131 13,275 11,881 10,985 11,243

Source: CSA SAP database.

The CSA calculates costs for DFL services from an approved fee schedule, allowing internal (i.e., CSA) projects to account for costs associated with each project. However, as the DFL has an A-base budget, projects only pay incremental costs.

In addition to providing testing services for systems built by the CSA and its contractors, the DFL also provides qualification services to Canadian and international clients (i.e., external to the CSA) on a "fee-for-service" basis when the capacity is available within the DFL. Again, the revenues received for such services are deposited into the Consolidated Revenue Fund (CRF). The amounts deposited in the CRF via the DFL from 2009-10 to 2013-14 are summarized below.

Table 2: DFL Revenue Contributed to Consolidated Revenue Fund, 2009-10 to 2013-14

2009-10 2010-11 2011-12 2012-13 2013-14
DFL Revenue contributed to the Consolidated Revenue Fund ($K)Footnote 3 1,660 3,650 5,311 2,193 1,940

Source: CSA SAP database.

2.5 Prior Evaluations and Audits of the Program

This is the first evaluation conducted of the Qualifying and Testing Services Program. Previous reviews and audits of other programs have only touched on the DFL.

The DFL "Good-Faith Review" was conducted in 2008 in response to a directive received from the CSA's Executive Committee that was looking at updating the mandate of the DFL as part of an overall CSA strategic review. This review was undertaken in two phases. The first phase of the review included revisiting the mandate of the DFL, looking at the history of the DFL, and describing the facilities and capabilities available at the DFL. The second phase of the study was focused on a comparative analysis of the facilities, services and capabilities provided by the DFL with other similar facilities across Canada. A report by Facet Consulting (2008) concluded that:

When comparing DFL's services and related rate schedules with the other similar Canadian environmental test facilities that provided rate information, the DFL's rates are within market norms. A few of DFL's specific items may be higher or lower than some of the equivalent services from other facilities in certain cases, but they are typically within the highest or lowest limits of pricing for similar services. It can also be concluded that at least with respect to Canada's large space industries and their specific facilities, testing for outside interests is not a priority and not a primary area of business. Also very few external labs have a schedule of rates for their services and only one has the level of detail similar to the DFL. Most labs provide all-inclusive figures that encompass the range of services from that particular facility. As a result, it is difficult for users to accurately estimate the overall test costs and for the labs to precisely account for their costs. The DFL approach offers clients and users a straightforward and thorough method for cost estimations and for effective project management control, which serve to instill confidence in DFL's services to the outside marketplace. Finally, and perhaps most importantly, the DFL facilities, services and capabilities in the environmental test domain are unique in Canada and Canada's space companies rely on the DFL to qualify their space hardware and to position themselves for bidding on international space programs and projects.

Two internal audits of the DFL have been undertaken by the CSA. An audit of the Management Framework of the DFL was completed in November 2005 and an audit of the Management of Testing Facilities was completed in November 2010. The 2005 audit concluded that the DFL's management framework was appropriate to achieving the objectives of the organization. The audit made a number of recommendations focused on the user fees charged to external clients and improving the quality of financial information. The 2010 internal audit concluded that the planning and management processes of the testing facilities made it possible to meet the requirements of internal and external clients of the DFL efficiently and effectively. The 2010 internal audit made three recommendations:

  • The planning process between the DFL and internal testing facilities users should be formalized and documented.
  • The DFL should draft a document outlining its overall strategy for use of facilities and draw up an associated implementation plan.
  • The DFL should review the output indicators and results indicators to ensure they are relevant and adequately measure DFL performance.

2.6 Program Theory

The Qualifying and Testing Services Program is primarily concerned with access and quality of services offered to its clients, thus the logic model and performance measurement framework are focused on information and services provided and client satisfaction. The logic model and the performance measurement framework were developed internally by the CSA.

To provide services and to carry out activities to meet its mandate the DFL requires highly qualified personnel, capital and operating funds. These resources are used to maintain extensive specialized systems and processes for AIT of space systems.

According to the PM Strategy (2014) the logic of the Qualifying and Testing Services Program can be divided into three broad streams that, together, support the nurturing and enhancement of future Canadian space capacity:

  • Service maintenance and upgrades to maintain access to reliable services;
  • Business development to foster and increase utilization of the AIT facilities; and
  • Qualifying and testing activities to maintain and increase informed decision making.

The logic model is presented graphically after the narrative below.

Service Maintenance and Upgrade

Maintain and improve baseline AIT services:

The various DFL systems used to provide AIT services require regular maintenance, calibration, certification, and inspection to continue to meet baseline technical performance and to minimize risk of failure. As technology progresses and client demands evolve, it is necessary for the systems to evolve accordingly, while remaining within scope of the baseline services. This means replacing ageing and obsolete equipment, and taking advantage of more cost-effective methodologies as they become available.

This activity is necessary to ensure that baselines services (Op1) are maintained and ready for testing space systems and other equipment.

Develop new services:

In order for the DFL to be able to meet future needs of the Canadian Space Program and other DFL clients, DFL personnel engage in a variety of networking activities in order to stay abreast of current measurement technologies such as visiting similar facilities; developing a network with other professionals engaged in providing similar services worldwide; participating in working groups and professional organizations engaged in the development of new instrumentation, related test technologies and procedures, testing standards and practices, useful to the DFL; and gathering intelligence regarding most recent developments and progress in appropriate technologies and in determining future strategic needs and opportunities.

It is expected that service maintenance and upgrades will maintain access to reliable space AIT facilities (Oc1) and thus contribute to reducing the risk of a gap between services provided and users' needs. Providing access to a full range of testing services should prevent Canada's space sector from depending on the facilities of other nations and thus contribute to the Canada's autonomy in space qualification (Oc4).

Business development

Engage in business development:

To understand the needs of present and past clients of the program, to attract new business, and to ensure optimal utilization of facilities, DFL staff engage in a variety of networking activities with industry, OGDs, academia and other international organizations. Activities in this area include presentations to clients, conferences and symposia, guided tours and other suitable opportunities to show visitors DFL facilities and capabilities.

This activity establishes and sustains relationships with the clients (Op3). It is expected that access to reliable space AIT facilities (Oc1) and good business relationships will maintain utilization of the facilities (Oc2).

AIT activities

Qualify and Integrate Space Systems:

The DFL provides services using a large inventory of facilities, instrumentation, and associated expertise, and by using procedures designed to determine if components or hardware qualify for the rigours of launch and the space environment. The equipment, when not being used for space components and systems, is also used by DFL clients for other ground-based system testing.

Serve clients:

To be able to provide testing services, there are many tasks necessary to serve clients for whom the tests are performed. These tasks include identifying needs of clients, receiving test requests, preparing cost estimates, scheduling the tests, resolving schedule conflicts, preparing the necessary legal contracts and documentation, negotiating the conduct of tests, providing technical and logistics support, accounting, billing, invoicing and overall quality assurance and record keeping.

The outputs of space qualification testing are the test data and qualification results (Op 4) that provide detailed performance and technical data about systems tested at the DFL.

It is expected that the data supplied by the DFL to its clients are suitable in determining whether or not their systems will survive the launch and function as required in the space environment throughout the mission, thus contributing to maintained or increased availability of reliable information and services to assist clients in meeting project technical requirement (Oc3). Second, these data serve as references in resolving anomalies should these occur during a mission's operational phase. This outcome is the expected result of the program, as per the CSA's performance measurement framework.

Finally, achievement of the three immediate outcomes, namely maintained access to reliable space AIT facilities (Oc1), maintained utilization of the facilities (Oc2) and maintained or increased availability of reliable information and services to assist clients in meeting project technical requirement (Oc3), will allow the CSA to maintain Canada's autonomy in space qualification (Oc4).

Figure 1: Qualifying and Testing Services Program Logic Model

Description is above at section 2.6 Program Theory

3. Evaluation Scope and Methods

3.1 Purpose, Evaluation Issues and Scope

In keeping with the Treasury Board Policy on Evaluation (2009), the purpose of this evaluation was to provide an evidence-based assessment of the value for money with respect to the ongoing rationale and performance of the CSA's Qualifying and Testing Services Program. The intended users of this evaluation are the DFL's Director, the DG of Space Science and Technology, managers and staff as well as the President of the CSA and other senior managers tasked with making decisions about the DFL's ongoing and future direction. Other intended users include the Minister of Industry, Parliamentarians, the space sector and the Canadian public.

The evaluation focuses on the five core issues identified in the Treasury Board of Canada Secretariat's Directive on the Evaluation Function (2009), which includes issues of relevance (continuing need, alignment with federal government priorities, alignment with federal roles and responsibilities) and performance (achievement of expected outcomes, demonstration of efficiency and economy). The Evaluation Matrix, outlining the evaluation issues and questions, indicators and data sources, is presented in Appendix A. The specific questions addressed by the evaluation are listed below.

Relevance

  1. To what extent is there a continued need for the DFL? How are stakeholder needs expected to evolve over the coming years (i.e., 5 to 10 years)? How does the DFL support the Canadian space industry?
  2. Is the DFL aligned with federal government priorities?
  3. Is the DFL consistent with federal roles and responsibilities?

Performance

  1. To what extent has the DFL achieved its immediate outcomes?
  2. To what extent has the DFL achieved its intermediate outcome of maintaining Canada's autonomy in space AIT?
  3. What, if any, challenges has the DFL experienced in meeting its objectives? How has the DFL responded to these challenges (e.g., human resources, fee schedule, and retention of revenues)?
  4. Have there been any unexpected outcomes (positive or negative) as a result of DFL services (Qualifying and Testing Services Program)?
  5. Is the DFL producing its outputs and outcomes (immediate) in the most efficient manner?
  6. Are there alternative, more efficient or effective ways of delivering AIT services for the benefit of the Canadian space sector?
  7. Are there any opportunities for improving the overall economy of DFL services?

3.2 Approach and Methods

Although this is the first evaluation of the Qualifying and Testing Services Program, the DFL has been in existence since 1972. Since that time, its activities, outputs and outcomes have remained relatively stable as has the delivery of its activities. The focus of this evaluation was on the relevance and performance of the Qualifying and Testing Services Program and such is considered a summative evaluation according to the TBS definition.

In the terminology of evaluation research methodology, the design chosen for the Evaluation of the Qualifying and Testing Services Program was a basic non-experimental design, whereby the evaluation team assessed the extent to which the DFL delivered on its objectives over the time period being evaluated (April 1, 2009, to September 2014). Although the DFL has been collecting client satisfaction data throughout the time period being evaluated, the PM Strategy developed by the Qualifying and Testing Services Program was only finalized in March 2014 and so only limited performance data was available.

The methodological approach and level of effort for this evaluation were determined using a risk-based approach. The Qualifying and Testing Services Program is of a relatively low materiality (expenditures ranged from a high of $13.3M to a low of $11.0M, before taking into account funds deposited into the CRC, during the period being evaluated). Taking into consideration the level of risk and the need to consult with DFL clients, the evaluation used interviews, a benchmarking study and a document and file review as data sources. The evaluation design and the data sources made it possible to gather data for each evaluation question from multiple lines of evidence.

3.2.1 Approach

The evaluation team worked closely with an Evaluation Consultative Group (ECG). Members of the ECG included CSA managers and staff as well as representatives from the Audit and Evaluation Directorate within the CSA. The evaluation team sought input and feedback from the ECG on key deliverables for the evaluation, including Project Work Plan; Evaluation Plan; interview guides; presentation of preliminary findings; and the final report. The ECG also provided names of individuals to be interviewed.

3.2.2 Data Sources

3.2.2.1 Document Review

A review of existing documentation that relates to the Qualifying and Testing Services Program was undertaken to help address all evaluation issues of relevance, effectiveness, efficiency, and economy. A number of documents were provided by the Project Authority and other members of the ECG, and these documents were reviewed as part of the process of developing the Work Plan, Evaluation Matrix and Evaluation Plan. Additional documents were identified by the research team through an Internet search. All documents were reviewed systematically, using a template based on the evaluation matrix, during the data collection phase of the evaluation.

Although no challenges were encountered during the document review, it must be noted that a few documents received were marked as confidential and could not be quoted. Although these documents provided important context to the evaluation team in interpreting the findings, the information contained in the documents could not, in many cases, be used in the report. This challenge was mitigated through the use of other documents and interviews.

A list of documents reviewed is included in Appendix B.

3.2.2.2 Key Informant Interviews

The selection of interviewees was focused on individuals who were the most familiar with the DFL and who would be well placed to respond to the evaluation questions – i.e., selection of key informant interviewees was purposeful. The interviewees were identified by members of the ECG and represented individuals who had been directly involved in managing the Qualifying and Testing Services Program or who were previous clients of the DFL, including foreign clients, universities, other federal government departments and agencies, CSA project representatives (RADARSAT, CASSIOPE, NEOSSat, etc.), and industry/prime contractor representatives. All of the DFL's industry clients from 2009 to 2014 were contacted by our evaluation team and given the opportunity to participate in an interview. The clients interviewed represented firms from across Canada and included small, medium and large firms as well as universities and other federal government departments and agencies. Interviews were conducted both by telephone and in person.

The evaluation methodology described in the Evaluation Plan anticipated conducting key informant interviews with a small number of Canadian non-clients of the DFL, i.e., firms, universities or other federal departments and agencies that required testing facilities or services within the past five years but chose not to use the DFL. Our team consulted with members of the ECG; however no Canadian non-clients were identified. Specifically DFL staff were unable to identify firms that solicited information and a price quote from the DFL but did not proceed with testing at the DFL. It was not possible to assess how many organizations in the space sector could have used the DFL for testing but did not.

The original proposal from the consultants included the conduct of a survey of the DFL's Canadian clients. This survey, described in the Evaluation Plan, would have meant that all Canadian DFL clients who were not included in the key informant interviews would have been invited to complete an online survey. However, as the project progressed towards finalizing the data collection tools and sampling, it was discovered that there were too few clients to merit a survey. The finalized list of clients sent to our project team by DFL staff included client contacts of firms that were no longer in business and those that were on the list of key informants. Removing those that were no longer in business and those that had already been contacted for an interview left 10 firms – all of which were contacted for an interview to replace the survey. This means that all Canadian DFL clients over the past five years that are still in business were given the opportunity to provide input into the evaluation.

In all, 44 key informant interviews were completed. Of the individuals identified for the key informant interviews, some could not be completed because they were unavailable during the interview period, because they were no longer with the organization or because they felt someone else within the organization was better placed to respond to the interview questions (in which case these individuals were contacted instead). In two cases industry representatives were on the list of key informants as well as the list of interviewees for the benchmarking study (described below), these individuals were interviewed for the benchmarking study only in order to minimize response burden. The response rate for the second wave of interviews with the DFL's Canadian clients was much lower. This is not entirely surprising since the first wave of interviews included individuals identified as being the most engaged with the DFL and thus the second group had less regular contact with the DFL and were thus more difficult to engage.

Table 3: Key Informant Interviews by Category

Category of Interviewee Number of Potential Interviewees Number of Interviews Completed
CSA-St. Hubert Managers 17 14
CSA-DFL Management and Staff 8 8
Canadian Clients (Wave 1) 17 17
Foreign ClientsFootnote 4 2 1
Canadian Clients (Wave 2 – supplementary interviews) 10 4
Total 54 44

Interviews were conducted in the preferred official language of the interviewee. Interviews ranged in length from approximately 20 minutes to slightly over 60 minutes and averaging about 40 minutes. Interview guides are included in Appendix C.

A key limitation in the interview findings is that the majority of interviewees have a vested interest in the DFL and its continued operations. However, the findings from the key informant interviews have been triangulated with findings from other lines of evidence to the extent possible.

With respect to analysis, the specific roles and responsibilities of each interviewee meant that the interviewees were not in a position to respond to all questions posed, even though the guides had been tailored to the three broad categories of interviewee. Furthermore in the case of clients the need to keep corporate information confidential and the qualitative nature of key informant interview data mean that reporting interview findings using counts of interviewees (i.e., how many said what) is neither relevant nor appropriate.

3.2.2.3 Benchmarking Study

The benchmarking study was intended to provide additional information on what AIT facilities and services are available within Canada and internationally and, to the extent possible, the costs of these services. The benchmarking criteria were identified based on the evaluation issues, questions and indicators set out in the Evaluation Matrix for which benchmarking is identified as a data source. The process for undertaking the benchmarking study is described in this section.

A key step in the benchmarking study was to identify the organizations that the DFL can usefully be compared against both in Canada and abroad. This was done by reviewing previous benchmarking studies undertaken by theFootnote 5 CSA as well as undertaking web research to identify testing facilities similar to the DFL in the USA and Europe. The following eight facilities were examined:

  • Space Systems Loral, USA.
  • Lockheed Martin Space Systems Company, USA.
  • NASA Glenn Research Center, Plum Brook Station, USA.
  • NASA Jet Propulsion Lab, USA.
  • Thales Alenia Space Italia, Italy.
  • Intespace, France.
  • IABG Space Test Centre, Germany.
  • ESA/ESTEC, the Netherlands.

The data for the benchmarking study was collected through web research on the selected comparison organizations and interviews with key individuals in these organizations. The interviews verified and updated the information gathered from the web research and provided supplementary information on the role the organizations and their AIT services play in meeting the space needs of the respective countries. The interview guide used for the benchmarking study is presented in Appendix D.

The process for conducting the interviews for the benchmarking study was identical to the process described above for the key informant interviews.

A challenge encountered in conducting the interviews for the benchmarking study was the identification of the most appropriate contact within each facility. Our team consulted with CSA-DFL representatives and used our own network of contacts in the space industry to identify the most appropriate individual(s) within each organization.

A key limitation for the benchmarking study, also identified in previous benchmarking studies contracted by the CSA, was an inability to obtain cost information on the various services provided by AIT facilities. Some of this is the result of confidentiality issues, lack of direct comparability across facilities/services and a lack of a price list for facilities that are owned by companies that undertake only internal testing, i.e., they do not provide testing services outside their companies.

3.3 Limitations

The main limitations faced by the evaluation along with the implications and mitigation strategies are described in this section.

Lack of price information for other AIT facilities. The benchmarking study was not able to obtain information on the fees charged by comparable AIT facilities in Canada and elsewhere. In the absence of this data, we have provided a qualitative assessment based on interviews with DFL clients.

Lack of interviews with non-clients. As noted above, the evaluation team endeavoured to interview representatives from organizations that could have used the DFL for testing space hardware but did not. These interviews were expected to provide information on viable alternatives to the DFL for Canadian companies and allowed for a larger variety of perspectives. In the absence of interviews with non-clients, we used available qualitative information from interviews with current clients of the DFL as well as information from the benchmarking study.

Lack of baseline data and targets. Because a PM Strategy was only finalized in March 2014, no baseline data was available for many of the outcomes in the logic model. This is a problem common to many federal government evaluation studies. However, the DFL has been collecting client satisfaction feedback via a short questionnaire over the entire time period being evaluated. The evaluation gathered primarily qualitative evidence from key informants evaluation issues related to the performance of the DFL. Interviews were undertaken with a range of clients representing (Canadian, foreign, large firms, small firms, universities, other government departments) in order to obtain a variety of perspectives.

4. Results

This chapter presents evaluation findings related to the relevance and performance of the DFL.

4.1 Relevance

The relevance of the Qualifying and Testing Services Program was evaluated with regard to 1) the extent to which there is a continued need for the DFL to support the Canadian space industry; 2) the linkages between program objectives and federal priorities; and 3) the role and responsibilities for the federal government in providing qualifying and testing services to the Canadian space industry.

4.1.1 Continued Need for the Program

4.1.1.1 Current and Anticipated Needs for Assembly, Integration and Testing (AIT)

Given the timelines and cost involved in putting technology into space, developers and buyers proceed stepwise, first testing space technology as rigorously as possible on the ground and then testing a small prototype or components in space to gain "flight heritage." For on-the-ground testing, industry and government use facilities such as the DFL. As noted in the Aerospace Review (2012) the DFL provides specialized facilities, equipment, and support personnel necessary to assemble and check the space-worthiness of entire spacecraft, their subsystems, and major components.Footnote 6

As an indicator of demand for AIT internationally, there is evidence of a strong market for satellites with Euroconsult (2014) estimating approximately 115 satellites to be launched each year between 2014 and 2023.Footnote 7 The Aerospace Review Space Working Group Report (2012) also notes that the global space sector is experiencing exponential growth and transformation. All G20 countries have space programs and more than 50 countries have their own domestic satellite system.

Based on an analysis of the international global satellite industry conducted by Euroconsult (2014) government clients will be responsible for more than 75% of the $248 billion in revenues expected from the manufacturing and launch of the 1,155 satellites between 2014 and 2023. Euroconsult (2014) estimates that internationally close to 90% of the government market value will remain concentrated in the 10 countries with an established space industry, with growth in the government market coming largely from new satellite systems in 35 nascent space countries, creating a market of $2 billion on average per year to be provided principally by foreign suppliers as local industry capabilities develop simultaneously.

In the commercial space sector, Euroconsult anticipates a total of 350 satellites to be launched over the decade, most of which will be for the replacement of satellites currently in orbit.

This would suggest that, internationally, there will be a continued demand for AIT services in general. A strong area of demand is internationally with governments that do not have their own space program and/or testing facilities.

The next question is whether companies in the space industry will require the services of an outside testing facility such as the DFL. Several factors are and will continue to affect demand for AIT services generally and for the services provided by the DFL. One factor is the increasing emphasis within the space sector on smaller satellites. This trend was noted by some of the CSA key informants and supported by the literature. For example, The Economist recently reported that some 1,000 nanosatellites (usually under 500 kg) are expected to be launched in the next five years.Footnote 8 Further, there is also the emergence of "cube" satellites ("CubeSats"), which are smaller than nanosatellites, with a volume of 1 litre (a 10 cm cube) and weighing 1.33 kg or less. Given their greatly decreased cost, CubeSats are seen as requiring less rigorous testing due to the decreased financial risk. For example, they are often assembled in "clean enough rooms" rather than in the typical clean room that is the standard for larger satellites.Footnote 9, Footnote 10 This trend towards small satellites does not necessarily mean a corresponding reduction in demand for DFL services in the coming years. Although originally built to accommodate large spacecraft, the DFL is scalable, i.e., it can easily adapt to the needs of smaller satellite projects. The DFL has thermal vacuum chambers ranging in size from 1 square metre to 7x10 metres.

Another major factor influencing the current and anticipated demand for DFL services noted by some CSA representatives is the Canadian space sector's dependence on the CSA and other federal departments and agencies for satellite design and construction contracts. According to several interviewees, the limited investment in new space projects on the part of the federal government in recent years has resulted in less demand for AIT and thus for the services provided by the DFL. According to these interviewees, the federal government has not funded any large new space projects in recent years. A design contract with MDA for the RADARSAT Constellation was announced back in 2007 and the contract to build RADARSAT Constellation was signed in 2013. There have been a few smaller projects, such as CASSIOPE, Sapphire and NEOSSat. Looking ahead, the CSA has partnered with DND and Environment Canada on the Polar Communication and Weather Mission (PCW), but funding for the further development of this mission has not been approved.

In their analysis of Canada's space sector, Euroconsult (2014)Footnote 11 reports that Canada's CAGR (compound annual growth rate) for investment in space was 8%. Euroconsult estimated Canadian government expenditures for space at $648 million in 2013, an all-time high resulting from the combined peak funding in civil programs and spending by the Department of National Defence (DND) in space. The overall CSA space budget increased to $488 million from $388 million in 2012. According to the Euroconsult (2014) report funding increases were due to an extra $179 million allocation to the RADARSAT Constellation Mission (RCM) from the fiscal framework and other Canadian governmental organizations. According to Euroconsult (2014) this increasing budget masks the fact that since 2012, the baseline CSA budget, set at approximately $300 million, has been decreasing. The DND does not disclose its annual spending for space, however Euroconsult estimates that DND's annual spending on space was C$160 million in 2013, based on its programs' life cycles.

Although the global space sector is growing and the most recent statistics in the State of the Canadian Space Sector (2012) indicate that the Canadian space sector is relatively healthy with revenues of $3.327 billion in 2012, of which 20% were from public (i.e., government) sources,Footnote 12 these statistics mask major variations in the growth of the various segments of the industry. According to the Aerospace Review's Space Working Group, telecommunications revenues dominate the statistics in the Canadian space sector, accounting for 80% of the sector's revenues and which have increased by 24% over the previous five years. As noted in the Aerospace Review's Space Working Group's most recent report (2012):

It is important to note that these statistics are dominated by the growing communications satellite services and applications sector, which grew by almost 16% in 2010. The strength of this segment masks a worrying trend in other segments of the space industry. For instance, while the services and applications segment has experienced growth in the past five years of 67%, the space segment has only grown by 1%, the research segment by 8% and the ground segment has contracted by 4%.Footnote 13

Delving deeper into the State of the Canadian Space Sector (2012) reinforces this point. Over the past five years, revenues from navigation technologies have decreased 33%, space exploration has decreased 9%, and space science has decreased 22%. While revenues from earth observation have increased 61% over the past five years, these are all downstream services, not space hardware which is what requires AIT at facilities such as the DFL.

Another key factor for the DFL is that some of the larger firms in the industry have established in-house AIT facilities. This trend was noted by some of the CSA interviewees and several client representatives. Some of the companies interviewed noted that there are several reasons for this decision. First, there is a desire to be more self-sufficient for their testing needs, a goal which is more attainable given the growth in demand for smaller satellites which can be fully tested in house. Another factor is a desire to reduce costs, as it is more expensive to have two testing teams, one situated at the DFL and another in-house. A related factor is human resource management, e.g., by keeping in-house staff more fully utilized, they are less likely to leave. A final factor is convenience, i.e., the company has more control over the scheduling of tests.

Some provincial governments and federal departments / agencies / programs have subsidized the implementation of testing facilities. MDA has received a grant from the Quebec provincial governmentFootnote 14 while Magellan has received funding from the Manitoba government, Western Economic Diversification Canada (WED) (federal government) and the University of Manitoba. The University of Toronto's Space Flight Laboratory received funding through the Canada Foundation for Innovation (although the CFI is a non-governmental body, it is funded entirely by the federal government). Interviewees noted that these facilities do not contain all the equipment and the scale of equipment available at the DFL. The DFL's large thermal vacuum chamber was frequently noted. Such a facility could cost a company approximately $30 million to put in place – a cost which is beyond the means of even large firms, not to mention the cost of the expertise required to operate the equipment. A few interviewees representing large Canadian firms that have their own testing facilities noted that their own facilities complement those of the DFL, i.e., company facilities do not entirely displace the DFL.

Based on interviews with industry representatives, small- and medium-sized firms and universities are currently very dependent on the DFL for testing, and this is expected to remain the case. Several interviewees noted that small- and medium-sized firms and universities generally cannot afford to build in-house test facilities. The use of other (larger) firms' testing facilities is not seen as a viable option due to the desire to protect intellectual property.

Larger firms anticipate their need for the DFL will continue to decrease in the coming years due to the factors noted above. However, most representatives from these firms stressed that they will continue to use the DFL for some testing, e.g., tests requiring a large thermal vacuum chamber, for very sophisticated and customized tests, or when independent third-party testing is desirable for military contracts. One representative from a large Canadian firm noted that using the company's US facilities will be dependent on resolving International Traffic in Arms Regulations (ITAR) issues, which can make it difficult to bring a satellite back into Canada. This interviewee explained that ITAR has significant implications for transporting satellites and equipment across the US border and thus adds to the time required as well as paperwork and cost.

4.1.1.2 Extent to Which Needs for AIT are Met by the DFL

This question concerns the extent to which the DFL has been able to meet the needs of its clients in recent years. Two previous benchmarking studies contracted by the CSA (Athena Global, July 2006 and Facet Consulting Ltd., 2008) noted that the DFL provides in a single location all the testing required by the Canadian space industry. The Facet study concluded that the DFL is the only test facility in Canada that is capable of meeting all of the environmental test requirements of a large satellite program within one large facility. The DFL facilities provide easy access between test areas avoiding the high risk of crating and transportation of the test item between sites to perform critical tests.Footnote 15 This conclusion was confirmed by the benchmarking study conducted for this evaluation.

Based on findings from the benchmarking study and interviews with DFL clients, the key factors influencing the selection of a testing facility by a user include i) the testing facility has facilities large enough for the size and weight of the objects to be tested; ii) the recognized skills and experience of the facility's staff; iii) the facility offers competitive pricing for testing services; and iv) location within Canada. Location within Canada is viewed as an important factor due to ITAR restrictions which make transporting equipment across the Canada–US border administratively difficult and time-consuming and thus costly.

Overall, the majority of CSA interviewees and client representatives stated that the DFL continues to meet the needs of the CSA and the Canadian space (and telecommunications) industries for reliable AIT testing facilities. A few client representatives (that lack in-house facilities) noted that having access to the DFL means they do not need to invest in expensive testing equipment which they might need to use only once every few years.

4.1.1.3 Importance of the DFL in Supporting International Competitiveness and Potential Impact of the Absence of the DFL

There was general agreement among interviewees from all groups that the DFL plays a critical role in supporting the competitiveness of the Canadian space industry. Most CSA representatives noted that the DFL, as part of the CSA, has played a key role in developing the Canadian space industry – the CSA through provision of contracts to the space industry and the DFL through the provision of reliable AIT facilities to the contracted firms.

There was general agreement among interviewees from all groups that the DFL plays a critical role in leveling the playing field among Canadian firms by ensuring access to AIT facilities for small- and medium-sized firms. Unless they are subcontractors to large firms that have their own AIT facilities, the absence of the DFL would leave smaller firms without ready access to AIT facilities. They would be required to seek AIT internationally, which would increase the cost and risk to the project, or to try to access the AIT facilities of other firms in Canada which would put their intellectual property at risk. As noted earlier, using facilities in the US would mean having to deal with ITAR regulations in addition to incurring higher costs and greater risks to equipment.

Overall the DFL is seen as contributing to making the Canadian space industry more cost-competitive because it has access to the DFL which provides reliable AIT facilities in one location, eliminating the need to travel and transport very expensive satellites to multiple locations. The more frequently space equipment is moved, the greater the risk of damage to sensitive components. CSA and client representatives also noted that availability of AIT facilities in Canada also eliminates the need for moving equipment to the US and having to deal with the related ITAR requirements. The DFL is also seen as enhancing the competitive bids on international project proposals by being viewed as an independent, world-class testing facility that has Government of Canada ownership (therefore viewed as neutral with no vested commercial interest).

According to the study conducted by Facet Consulting (2008), major aerospace companies interviewed for that study reported that if the DFL did not exist they would be forced to test their instruments in the US and suffer the consequences of ITAR restrictions, schedule delays and significantly increased costs. However there is evidence from the present evaluation that the industry has evolved since 2008 with larger firms being less dependent on the DFL.

There is general agreement among interviewees that although AIT facilities are available in the US and internationally, dependence on these facilities would result in increased costs and risks to the Canadian space industry. Risks related to using facilities in the US or internationally include risk of damage to equipment, risks related to intellectual property (IP), and risks to timelines/schedule – all of these risks have the potential to significantly negatively affect the budget for a project. Using US facilities would further result in additional paperwork and (potentially) delays related to ITAR.

4.1.1.4 How other Countries Meet their Needs for AIT

The benchmarking study undertaken as part of this evaluation found that virtually all countries with a developed space program also have a national AIT centre focused on providing services to their country's space sector. This was confirmed by interviewees. The one notable exception is the US where all prime contractors have their own facilities and so there is less of a need for a government-run facility like the DFL. According to interviewees, if specialized equipment is required, the US government will purchase it and then sell it to the prime contractor for a nominal price. This is seen as a subsidy to the industry by the US government. NASA AIT facilities are used almost exclusively for NASA programs.

In Japan and the EU, facilities are operated as government-owned, contractor-operated (GoCo) facilities. Contracts for the operation of the AIT facilities are renewed periodically, usually every five years. The EU facilities are reported to be very busy with numerous spacecraft in the pipeline. One CSA representative noted that the Japanese government has continued to strongly support their space industry, despite the country being in a recession over much of the past 10 years. In France, the AIT facility is operated by a consortium of the EU's two main prime contractors. The consortium is operated at arm's length from the two firms in order to prevent a monopoly type situation. The facility in Germany, IABG, is employee owned and reported to be very well utilized by space industry and non-space industry clients.

Turkey, Malaysia, India, China and Brazil all have their own facilities similar to the DFL which are operated by the government. In some cases the DFL has assisted these other countries in setting up their facilities. It was noted by a number of interviewees in the benchmarking study that emerging space countries tend to build testing facilities first as a key part of their national space infrastructure and to establish their credibility as space nations.

4.1.1.5 Alternatives to the DFL

The interviews conducted as part of this evaluation agree that there are few alternatives to the DFL in Canada, particularly for the testing of large spacecraft or antenna. A few examples of alternatives were provided, including the Centre de recherche industrielle du Québec (CRIQ) and a variety of private testing labs. However none of those facilities has all the testing equipment available at the DFL. Although a few larger firms and universities have AIT facilities, these are not intended to serve the industry at large – they do not have fee schedules and internal projects take precedence over external ones. This perspective is echoed in both the Facet Consulting Ltd. (2008) and Athena Global (2006) studies.

According to a few interviewees, no other facility within Canada has a large thermal vacuum chamber required for testing of large satellites. Prime contractors that design and build large satellites and are able to do much of the required testing in house will still go to the DFL to use its large thermal vacuum chamber. Also no other facility in Canada, with the possible exception of MDA, has a large anechoic chamber such as that used for testing assembled spacecraft or large antenna. The DFL has a database of approximately 20 facilities across Canada able to provide some AIT services. This database is updated periodically and there are plans to expand to add AIT facilities in other countries. In terms of alternatives available outside Canada, the benchmarking study conducted for this evaluation identified the following:

  • Loral and Lockheed Martin (testing facilities at the two major US space companies): Representatives of these companies stated that their facilities are similar to those at the DFL but are of larger capacity. For example, the large thermal vacuum chamber at Loral has an internal diameter of 8.2 metres and a length of 15.2 metres, compared to an internal diameter of 6.7 metres and a length of 10.7 metres for the largest thermal vacuum chamber at the DFL. The capacity difference is not only a function of the size of the testing equipment but also a function of the number of facilities for any particular service such as thermal vacuum where Loral has a number of chambers close to the size of the largest chamber at different locations.
  • Intespace: Differs significantly from the DFL in the scope of its testing which covers non-space sectors (aerospace, transportation and energy) in addition to the space sector. Testing in these other sectors tends to be on subsystems which need to be qualified. The equipment is similar for the testing in all sectors except some of the equipment such as the vibration tables are designed to take much heavier loads than needed for testing space products. The Radio Frequency (RF) and Electromagnetic Compatibility (EMC) testing equipment is used in all sectors. The thermal vacuum chambers cover a range of temperature testing but the vacuum component is not needed outside the space sector so the chambers are less expensive.
  • IABG: A detailed comparative analysis of DFL and IABG services was undertaken by IABG in response to our inquiries. IABG's analysis showed a high degree of similarity in the range of services offered by the DFL and IABG.

4.1.2 Alignment with Federal Priorities

This section describes the alignment of the Qualifying and Testing Services Program with federal government priorities as described in key federal documents relating to space, science and technology and federal government priorities overall.

Canada's Space Policy Framework: The importance of the space industry in relation to federal government priorities is described in Canada's Space Policy Framework (2014), which notes that the well-being of Canadians depends on the services the space industry provides. The skills the industry requires are those of an advanced, knowledge-based economy. The jobs it creates are demanding and rewarding. And the profits it generates domestically and through international sales and partnership are a powerful benefit to the national economy.Footnote 16

According to some CSA representatives, the Qualifying and Testing Services Program has always played an important role in supporting the development of the Canadian space industry. The need for this role is reflected in Canada's Space Policy Framework. However some representatives from the CSA noted that the federal government currently lacks a clearly articulated plan with respect to the space industry. A recent report by Euroconsult (2014)Footnote 17 also notes the lack of a clearly articulated plan for the space industry on the part of the federal government. The Euroconsult report, however, notes that the federal government's release of Canada's Space Policy Framework (2014) promises more direction for the industry by outlining broad national goals for the space program.

Canadian Space Strategy: The program's PM Strategy describes how the DFL is directly aligned with the CSA's strategic vision as described in the Canadian Space Strategy (CSA, 2003), which is the current framework that guides the CSA in implementing Canada's Space Program. This strategy outlines the four core programmatic thrusts in which the CSA will lead Canada's space activities. Qualifying and testing services are one of the key elements outlined in the Canadian Space Strategy required in order to meet its target results as identified in each of the four core thrusts. In particular, the Canadian Space Strategy (CSA, 2003) states that "in order for Canada to pursue its space program to the fullest, we must preserve our ability to assemble, integrate and test the space systems of our choice. Canada must also have the ability to operate the critical components of its ground and space-based infrastructure. This requires adequate test, integration, and operation infrastructures. The Canadian Space Agency will encourage private-public partnerships to maximize the efficient utilization of facilities and equipment based in Canada. We will also open those to our international partners, provided Canadian interests and requirements are protected."

Canada's S&T Strategy (2007) states that "the Strategy and its policy commitments will be guided by four core principles, one of which is Promoting World-Class Excellence. The Government of Canada will ensure that its policies and programs inspire and assist Canadians to perform at world-class levels of scientific and technological excellence." By virtue of the specialized space qualification services it provides the DFL contributes to this Government of Canada priority. The process of testing feeds into the process of refining and developing a technology – it is part of the innovation process.

CSA representatives explained that internationally, Tier 1 countries are those which have the ability to develop, test and launch spacecraft; Tier 2 countries are those which are able to develop and test but which, like Canada, lack launch facilities; and Tier 3 countries are able to develop spacecraft but are unable to test and launch. Interviewees believe that a healthy space industry depends on Canada remaining a Tier 2 country and maintaining its autonomy by having access to reliable testing facilities. Without the DFL the Canadian space sector would be dependent on outside testing facilities thereby making Canadian firms less cost-competitive and exposing them to increased risk due to transporting equipment and potentially putting timelines at risk. The DFL thus contributes directly to maintaining Canada's autonomy in space AIT and a healthy space industry.

4.1.2.1 Alignment with Federal Roles and Responsibilities

Bill C-16, the Canadian Space Agency Act, states, in part, that:

  • one of the "objects" of the Agency is "to ensure that space science and technology provide... economic benefits for all Canadians."
  • one of the Agency's functions is to "encourage the commercial exploitation of space capabilities, technology, facilities and systems."
  • in carrying out its objects, the Agency may "construct, procure, manage and operate space research and developments... facilities... and... provide services and facilities to any person."

Consequently, on a fee-for-service basis, the DFL provides facilities and environmental testing services to both private and public sector clients. Its task is twofold:

  • to maintain and operate an environmental test facility capable of meeting the current and future needs of Canada's space program.
  • to monitor and respond to the test requirements of future space missions of interest to Canada.Footnote 18

Some CSA representatives explained that if the federal government wants a healthy and growing space industry then it needs to ensure the availability of reliable and accessible AIT services. The availability of AIT services at an entity such as the DFL is seen as being particularly important for small, growing companies which would not be able to compete with large companies. Notably a few interviewees representing the CSA and its clients commented that although federal support is required, the AIT facility need not be operated by the federal government.

Some CSA representatives also noted that maintaining AIT capacity is critical to the ability of Canada to implement a space program, particularly space exploration projects should it decide to do so in the future. Some CSA representatives noted that continued support for the program must be considered in the context of Canada's future space activity. If the federal government intends to implement space projects in the near future then the DFL must remain open.

The CSA mandate, as defined in the Canadian Space Agency Act (1990 & Supp. 2012), is to "promote the peaceful use and development of space, to advance the knowledge of space through science and to ensure that space science and technology provide social and economic benefits for Canadians" (Canadian Space Agency Act, Section 4, 1990). The DFL's activities are in alignment with the Canadian Space Agency's mandate because it provides an essential service to ensure that CSA projects are successful and have been qualified before launch in space.

In carrying out its objects, the Agency may "construct, acquire, manage, maintain and operate […] facilities and systems […] and provide services and facilities to any person" (Canadian Space Agency Act, Section 5, 1990). As noted in the PM Strategy, through its activities and operations, the Qualifying and Testing Services Program contributes to meeting the overall goals and objectives of CSA programs and projects, and thus to the federal government's priorities of prosperity and sovereignty.

None of the CSA representatives interviewed believe that the DFL would be a better fit with another department or agency. A few clients noted that they also obtain testing services from other federal government facilities including the National Research Council; however this appears to be mainly for acoustic testing.

4.2 Performance

This section addresses the evaluation questions related to performance including the achievement of outcomes and economy and efficiency.

4.2.1 Achievement of Immediate Outcomes

This section presents evaluation findings related to the achievement of immediate outcomes as depicted in the logic model, specifically:

  • Maintained access to reliable space AIT facilities;
  • Maintained utilization of facilities; and
  • Maintained or increased availability of reliable information and services to assist clients in meeting project technical requirements.
4.2.1.1 Maintained Access to Reliable Space AIT

Most CSA and client representatives interviewed believe that the DFL has been successful at meeting the needs of clients for reliable access to AIT services.

4.2.1.1.1 Client Satisfaction with the DFL

The DFL has been conducting client satisfaction surveys for a number of years. At the completion of each testing contract, the DFL sends the client a client satisfaction survey. Table 4 below summarizes the response rate for the DFL's client survey for 2009-10 to 2013-14.

Table 4: Response Rate for DFL Client Satisfaction Survey, 2009-10 to 2013-14

Fiscal Year Number of Surveys Sent Number of Responses Response Rate (%)Footnote 19
2009-10 85 37 44
2010-11 95 36 38
2011-12 82 28 34
2012-13 56 23 41
2013-14 33 7 21

Survey results indicate a consistently high level of satisfaction on the part of clients with the DFL consistently achieving almost 100% client satisfaction. Figure 2 below summarizes overall client satisfaction for the previous five fiscal years. The DFL also consistently meets or exceeds client expectations with respect to the delivery of quality, cost-effective and timely service as summarized in Figure 3 below. Key informant interviews conducted with DFL clients as part of this evaluation confirmed a high level of satisfaction with the DFL. Additional client satisfaction results are presented in Appendix E.

Figure 2: Overall Client Satisfaction with the DFL, 2009-10 to 2013-14

Overall Satisfaction

Graph. Data table follows.
Fiscal year Exceeded Met Not Met N/A
2009-10 59% 40% 0% 1%
2010-11 45% 51% 3% 1%
2011-12 50% 47% 0% 3%
2012-13 64% 35% 0% 1%
2013-14 60% 36% 2% 2%

Source: DFL Client Satisfaction Survey.

Figure 3: Satisfaction with DFL Quality, Cost-Effectiveness and Timeliness, 2009-10 to 2013-14

Delivering quality, cost effective, and timely services i.e., as scheduled?

Graph. Data table follows.
Fiscal Year Exceeded Met Not Met N/A
2009-10 48% 48% 4% 0%
2010-11 41% 53% 6% 0%
2011-12 57% 43% 0% 0%
2012-13 59% 41% 0% 0%
2013-14 80% 20% 0% 0%

Source: DFL Client Satisfaction Survey.

The DFL also administers a client satisfaction survey at the CSA program level. Three CSA satellite programs have completed AIT at the DFL over the time period from 2009 to 2014 – CASSIOPE (2009), Sapphire (2013) and M3MSat (2013). To maintain confidentiality of responses, the DFL produces a summary analysis of the survey results which it then provides to all respondents.Footnote 20 For all three programs, respondents rated all DFL facilities and services used very highly (at least 8 out of 10 for all facilities across all three programs). The summaries of responses to the program level client satisfaction surveys also indicated a very high level of satisfaction with DFL facilities and service.

A few CSA-St. Hubert and client representatives noted that there are instances of scheduling conflicts at the DFL. These interviewees report that these are generally the result of clients who frequently experience project delays. Scheduling conflicts which implicate CSA projects are generally resolved within the CSA (i.e., among the Director Generals involved). A few clients interviewed commented that the DFL has proven to be very accommodating to schedule changes. According to representatives from CSA-DFL, the number of tests not completed due to schedule conflicts is close to zero. When a schedule conflict occurs it is usually because one or another of the projects seeking DFL services has experienced delays and the DFL staff work with the clients in order to accommodate the projects' testing needs. Utilization of the DFL is discussed further in Section 4.2.1.2 below.

4.2.1.1.2 Maintained and Upgraded Services to Meet Client Needs

CSA-St. Hubert and client representatives stated that the DFL has generally kept pace with the evolving needs of clients. These interviewees noted that DFL staff are very well trained and qualified and the equipment is adequate to meet their needs. The DFL has successfully maintained its ISO 9001:2008 certification requiring the DFL to meet all the requirements of the annual audits. However, some CSA-St. Hubert representatives commented that there has been underinvestment in DFL staffing and infrastructure over the past few years and this has had an impact on the DFL's ability to keep pace with evolving needs and technologies over the long term. An internal review undertaken by the CSA in 2013 also concluded that there has been insufficient investment in the DFL in the past 5 to 10 years which has resulted in some deterioration of its facilities. Government-wide budget cuts have meant that the program's salary budget has inhibited the ability to hire new staff. Currently the DFL is only able to term employees (i.e., cannot hire permanent employees). Staffing challenges at the DFL are further compounded because the CSA is reviewing the business plan for the DFL, and there is thus some reluctance on the part of the CSA to hire new staff until the future of the DFL is decided. Although CSA expenditures on the DFL have remained relatively constant throughout the period being evaluated, there have been no adjustments for inflation and no increases to permit upgrading of infrastructure and the purchase of highly specialized equipment to maintain the DFL's world-class status.

The issue of shutdowns of the DFL in recent years was the subject of considerable discussion during several of the client interviews. Prior to 2012 there were no scheduled shutdowns at the DFL; however, extensive maintenance was required, and this was started in 2012. The DFL was constructed in 1971 and was not built in order to easily allow shutdowns of only specific areas of the facility, and thus for many types of maintenance the entire facility has had to be closed to clients. Other testing facilities around the world are constructed to easily allow for a shutdown of some areas while other areas continue to operate. Although the CSA has a policy of notifying clients well in advance of each shutdown, clients interviewed for the evaluation noted that the shutdowns have had a significant adverse impact on the ability of the DFL to meet their needs. Although the DFL strives to meet the needs of its clients, CSA representatives stated there is currently little that can be done to eliminate the need to shut down the facility periodically in order to perform routine maintenance or to upgrade the facility.

4.2.1.1.3 Developed New Services to Meet Client Needs

The DFL has developed new services to meet the needs of clients primarily for specific testing programs (i.e., on a case-by-case basis). For example, a Canadian space company needed some shock testing done and had arranged to have it done at a facility in England. This firm asked the DFL whether they would be able to develop a system for shock testing. The DFL now has shock testing capability at the unit/equipment level and the firm is able to have this testing done at the DFL at a significant cost saving. Shock testing is seen as increasingly important as satellites become smaller. Another recent example is the implementation of a gaseous helium system to support the thermal vacuum test specifications required by the JWST.

Some CSA-DFL representatives noted that cuts to the travel budgets along with policies directed at limiting travel within the federal government have curtailed the ability of DFL staff to travel to conferences and network with those who operate other facilities and with clients and potential clients.Footnote 21 Cuts to the CSA travel budget and other policies aimed at limiting work-related travel by federal employees are part of a government-wide effort to control costs related to travel.Footnote 22 Notably, the DFL has spent more than its budget allocation for travel in each fiscal year since 2009-10 indicating that perhaps the DFL requires more budget for travel than it is allocated. This is in contrast to the CSA (excluding the DFL) who have spent less than allocated for travel in each fiscal year. However it should be noted that decisions on travel within the federal government are made by senior managers based on a number of factors such as budget, need, human resource demands, etc. Table 5 below summarizes the CSA-DFL and CSA (excluding CSA-DFL) budget allocation and actual expenditures.

Table 5: Allocated and Actual Travel Expenditures, 2009-10 to 2013-14

2009-10 2010-11 2011-12 2012-13 2013-14 % change over time period
CSA-DFL Travel Budget
Travel Budget Allocation 130,000 120,000 120,000 75,000 41,100 -68
Actual Expenditures 117,607 122,831 122,197 74,504 51,738 -56
CSA Travel Budget (excluding the DFL)
Travel Budget Allocation 5,600,800 7,103,000 7,045,000 4,379,182 3,388,111 -40
Actual Expenditures 5,507,776 5,620,218 5,486,599 3,638,604 2,547,469 -54

Attendance at conferences and other networking events allows CSA-DFL staff to keep up to date on industry needs and technological changes (e.g., new measurement techniques, equipment, and what is needed to test new types of spacecraft). Some CSA-DFL staff believe it is now more difficult to stay abreast of how industry needs and standards are evolving and thus to ensure that the facilities continue to meet client needs.

4.2.1.2 Maintain Utilization of Facilities

The DFL's utilization statistics are limited by a lack of consistent approach to measuring utilization across test areas. For example, Thermal Qualifications Services calculates utilization on the basis of the actual test time, not including pre- and post-test activities, nor factoring in the maintenance and calibration required to ensure equipment is operational and available for testing services. RF Qualification services include pre- and post-test activities associated with the particular test being undertaken. Structural Qualification services follow a similar calculation as RF Qualification services. Essentially the DFL is currently only reporting "time on test" and thus may be under estimating utilization since they are not including the entire testing process (i.e., initial discussions/consultations with clients, pre-test and post-test activities, test report / test summary writing, as well as time spent billing and invoicing). This may also have implications for invoicing to clients since all client testing costs are not fully counted. CSA-DFL managers recognize the problems with how the DFL tracks utilization; however, limited financial and human resources have prevented the DFL from developing a more robust utilization measurement regime in a timely manner.

As per DFL data, the number of clients supported by the CSA through the Qualifying and Testing Services Program has been decreasing in recent years. This has important implications for the DFL in terms of its utilization. However, the number of clients supported by the DFL does not provide a full picture because the DFL frequently conducts multiple tests for clients. For example, NEOSSat is identified as a single client in 2012-13 however the DFL undertook 13 separate tests for NEOSSat. Likewise, testing for M3MSat involved six separate tests. The number of clients by fiscal year and type of client are summarized in Table 6 below.

Table 6: Number of Clients Served by Type of Client, 2009-10 to 2013-14

Fiscal Year Number of Clients
OGDs Industry Academia CSA Total
2009-10 1 19 2 6 28
2010-11 2 20 1 3 26
2011-12 2 17 2 2 23
2012-13 1 13 1 5 20
2013-14 0 13 0 7 20

Source: DFL Data.

Although the focus of this evaluation is on the previous five years, a clearer picture of the trend in the number of tests, clients and projects served by the DFL may be obtained by looking at longer-term trends. Figure 4 below indicates that although the numbers of tests, clients and projects undertaken at the DFL have decreased in recent years, the overall trend since 2001-02 indicates that these numbers have leveled off in the past five years to levels seen in 2001-02.

Figure 4: Trends in the Number of Tests, Clients and Projects, 2001-02 to 2013-14

2001-
02
2002-
03
2003-
04
2004-
05
2005-
06
2006-
07
2007-
08
2008-
09
2009-
10
2010-
11
2011-
12
2012-
13
2013-
14
Tests 80 90 72 94 161 186 129 131 107 137 97 104 87
Clients 28 37 23 31 27 30 44 36 28 26 23 20 20
Projects 43 58 39 56 45 48 36 50 41 41 34 27 21

Source: DFL Internal Data

Although the DFL's utilization figures are not robust for the reasons noted previously, it is possible to examine these data for general trends. The DFL's utilization figures indicate that the DFL's average utilization has generally remained constant at around 60% over the time period covering 2001-02 to 2013-14, with a few exceptions. From 2009-10 to 2012-13 the DFL was conducting thermal qualification testing on a large number of spacecraft for one company, and this is reflected in both the utilization of the DFL's TQ area as well as in the DFL's overall utilization rate which remained at over 70% for most of this four-year period. The DFL was shut down for maintenance for six months during the 2013 calendar year, and this is also reflected in the dramatic decrease in the DFL's utilization, which went from 58% in 2012-13 to 28% in 2013-14. The DFL's utilization figures are illustrated in Figure 5.

Figure 5: Utilization of the DFL by Testing Area, 2001-02 to 2013-14Footnote 23

Graph. Data table follows.
(% Utilization)
2001-
02
2002-
03
2003-
04
2004-
05
2005-
06
2006-
07
2007-
08
2008-
09
2009-
10
2010-
11
2011-
12
2012-
13
2013-
14
TQF 54 56 51 55 57 44 45 77 108 100 100 84 42
SQF 65 64 63 60 63 76 83 61 46 62 61 42 13
RFQF 55 55 56 56 57 57 67 60 61 63 65 49 28
AVG 58 58 57 57 59 59 65 66 72 75 75 58 28

Source: DFL Internal Data

The DFL's client satisfaction survey results, which were supported by findings from the key informant interviews with clients conducted for this evaluation, consistently indicate that clients are willing to use the DFL's facilities again, indicating that the decreasing utilization of the DFL is not directly linked to a lack of client satisfaction with the DFL. These survey results are illustrated in Figure 6.

Figure 6: Clients' Likelihood to Use the DFL Again, 2009-10 to 2013-14

Would you use the DFL's test facilities again?

Graph. Data table follows.
Fiscal Year Very likely Likely Unlikely N/A
2009-10 100% 0% 0% 0%
2010-11 82% 12% 6% 0%
2011-12 77% 23% 0% 0%
2012-13 94% 6% 0% 0%
2013-14 60% 40% 0% 0%

Source: DFL Client Satisfaction Survey

In the context of a service delivery organization such as the DFL, utilization has two components – equipment and human resources. The utilization of both must be optimized in order to maximize overall utilization. For example, human resources may be fully utilized but equipment may be underutilized indicating a lack of staff or a lack of staff with the appropriate skill sets to operate the equipment. Utilization could be increased with more staff and/or staff who are transferrable across DFL testing areas. To illustrate, the DFL's thermal qualification testing area currently has seven public servants, supplemented by two contract staff for a total of nine operators. For thermal vacuum chamber testing, health and safety regulations require that three operators be in place for each eight-hour shift, for a total of nine operators over a 24-hour period (this is because a hazardous material, liquid nitrogen, is being used). This effectively utilizes all available operator staff in the thermal qualification area, so other thermal qualification testing cannot be done in parallel because there are no technicians available with the required skill set.

The statistics on FTEs at the DFL reflect a steadily decreasing number of staff with total FTEs dropping from 40.94 in 2009-10 to 32.7 in 2013-14 (a decrease of 20%) and FTEs focused on testing activities falling from 32.39 FTEs to 27.57 (a decrease of 15%). However these figures do not include contractors brought in on an as-needed basis to meet the demands for testing. The CSA's spending on staffing at the DFL has decreased by 10% over the period of the evaluation as indicated in the DFL budget information in Table 1. Table 7 summarizes the DFL's FTEs by category for 2009-10 to 2013-14.

Table 7: DFL Staffing by FTEs Classification, 2009-10 to 2013-14Footnote 24

Fiscal year 2009-10 2010-11 2011-12 2012-13 2013-14
Administrative Services (AS) 2.00 2.00 2.04 1.99 2.00
Clerical and Regulatory (CR) 1.00 1.00 1.00 0.27 0.50
Commerce (O) 1.00 1.00 1.00 1.00 1.00
Computer Systems (CS)Footnote 25 1.55 2.00 2.00 1.12 0.00
Executive (EX) 2.00 1.84 1.00 1.00 1.00
Scientific Research - Manager (REM) 0.00 1.00 1.00 1.00 0.63
Scientific Research – Scientist (RES) 1.00 1.00 1.00 0.82 0.00
Total Management and Administration 8.55 9.84 9.04 7.2 5.13
Electronics (EL) 5.00 5.23 6.00 5.98 5.46
Engineering (ENE) 7.38 7.45 7.84 5.98 5.54
Engineering and Scientific Support (EG) 16.01 14.38 14.50 15.32 13.57
General Technical (GT) 4.00 4.00 3.55 2.99 3.00
Total Testing 32.39 31.06 31.89 30.27 27.57
Total All DFL Staff 40.94 40.9 40.93 37.47 32.7

Source: CSA SAP database

The CSA has implemented a recruitment process which in the context of the DFL means that all new staff will be "cross trained" in a number of areas so that staff within the DFL can more easily be deployed where they are most needed at any time. This could result in an increase in the utilization provided a sufficient number of clients are attracted.

In terms of ways of potentially increasing utilization, CSA-DFL staff report that market share could be increased for some areas, specifically for RF testing. However, the DFL is limited in its ability to do any marketing because staff are limited in their ability to travel to attend conferences, and the DFL does not have a marketing capacity.

According to CSA-DFL representatives, the DFL's current informal marketing plan is heavily dependent on the DFL's ability to participate in and network through attendance at international conferences, working group meetings, workshops and site visits. This allows DFL managers to remain in contact with senior representatives of the international space community and to present any new and upgraded capabilities and technologies available through the DFL to potential users. The curtailment of travel within the federal government has resulted in a significant decrease in the number of networking events attended by DFL representatives. A listing of conferences, workshops and working group meetings attended by DFL staff is presented in Appendix F.

Findings from the benchmarking study indicate that most privately owned or privately run facilities market their capabilities in order to maintain a steady throughput of testing objects (space as well as non-space). For example, Lockheed initiated a marketing campaign over the past few years mainly through seminar and conference attendance, presenting "white papers" and presentations that describe current developments and technical problems solved.

The benchmarking study found that Intespace markets its testing capabilities at conferences as other testing facilities do but also decided to diversify its sector coverage to compensate for the low periods in space testing. Intespace has developed other business lines associated with testing such as engineering in the test domain before and after testing and software development for the correlation of test and predicted results (a spin off called Dyna Works). Intespace revenues are divided as follows: space 70%, non-space 30% (it was 50% last year due to major aerospace testing for Airbus).

Unlike the DFL, ESTEC is operated by a private corporation, European Test Services (ETS), which is jointly owned by IABG and Intespace under a renewable five year contract. ETS has a team of 35 working at ESTEC. Priority for testing is given to ESA projects as there is insufficient ESA business to keep ESTEC busy, ETS is encouraged to do marketing of the facilities to both space and non-space potential users to make better use of ESTEC facilities.

In terms of the total number of potential clients in Canada for the DFL, it was not part of the scope of this evaluation to undertake the research required to produce an estimate. And as noted above, the DFL lacks a marketing department which would normally undertake this sort of research. However the most recent State of the Canadian Space Sector Report (2012) was based on a census approach whereby all 200 organizations identified by Industry Canada as being involved in the space sector were surveyed. However, as explained previously, the sector is dominated by the communications satellite services and applications areas which have no need for AIT. Thus it is reasonable to assume that the number of potential clients for the DFL from the Canadian space sector is considerably less than 200. This does not include the number of potential non-space sector clients for which no estimates are available.

The DFL's utilization could be increased by undertaking additional testing for other industries such as the telecommunications and aerospace industries and by seeking out more international clients. CSA-DFL representatives noted that the CSA does not restrict which industries the DFL serves and it has previously provided testing services to a broad range of clients and industries outside of the space industry. However, CSA-DFL representatives noted that some equipment, such as the thermal vacuum chambers, have very little utility outside of space environmental testing, thus limiting the potential for non-space clients. Also, because the DFL operates in a clean environment, they are unable to test certain "dirty" items because of contamination concerns.

With respect to expanding the DFL's market outside Canada, CSA-DFL managers believe the key markets would be the US and Europe. As is the case for Canadian firms seeking to conduct testing in the US, ITAR restrictions would make it more challenging for US firms to conduct testing at the DFL. CSA-DFL representatives indicated that their European clients are increasingly being pressured to undertake testing in Europe, i.e., there is evidence of increasing protectionism, an opinion confirmed in the Aerospace Review Space Working Group Report (2012).Footnote 26

4.2.1.3 Met Clients' Technical Requirements

Client representatives were asked whether they were satisfied with the documentation provided by the DFL. Most interviewees were satisfied with the documents and specifications provided by the DFL.

The client surveys provide strong evidence of client satisfaction with the DFL's ability to meet clients' technical requirements. The DFL's client satisfaction surveys consistently indicate a high level of satisfaction with the DFL's ability to provide the necessary information to clients and to provide practical information before, during and after the testing process. These survey results are illustrated in Figures 7 and 8.

Figure 7: Clients' Satisfaction with Information Provided by the DFL, 2009-10 to 2013-14

Providing strategic and relevant information to help in the delivery of clients' program

Graph. Data table follows.
Fiscal Year Exceeded Met Not Met N/A
2009-10 43% 48% 0% 9%
2010-11 26% 62% 3% 9%
2011-12 40% 47% 0% 13%
2012-13 47% 41% 0% 12%
2013-14 40% 60% 0% 0%

Source: DFL Client Satisfaction Survey.

Figure 8: Clients' Satisfaction with DFL Technical Recommendations, 2009-10 to 2013-14

Assisting you in your decision making by providing practical recommendations before, during, and after the conduct of your tests?

Graph. Data table follows.
Fiscal Year Exceeded Met Not Met N/A
2009-10 39% 61% 0% 0%
2010-11 29% 65% 3% 3%
2011-12 43% 50% 0% 7%
2012-13 59% 41% 0% 0%
2013-14 60% 20% 0% 20%

Source: DFL Client Satisfaction Survey.

4.2.2 Achievement of Intermediate Outcome – Canada's Autonomy in Space

4.2.2.1 Canadian Clients Use of Other AIT Facilities

Client representatives were asked to describe which elements of their testing needs have been or will be met by facilities other than the DFL. Almost all client representatives, including large and small companies, indicated that they will use the DFL for at least some of their testing. Of those who responded to this question, large firms indicated that they will use mostly their own facilities but will continue to depend on the DFL for tests they are unable to perform in their own facilities, including tests requiring the large vacuum chamber, environmental testing, vibration testing and EMI/EMC.

Some client representatives identified testing facilities they have used in the past, including South West Texas University, York University, IABG in Germany, Litton and Lockheed, to name a few. As noted earlier, the DFL maintains a database of close to 20 facilities in Canada that provide some aspect of AIT testing, although not necessarily for spacecraft. These facilities include private firms, university or university-affiliated laboratories, and federal government testing facilities. However none of those Canadian facilities have the breadth of services available at the DFL nor are they able to accommodate large spacecraft. In addition, none of the facilities are designed to serve external paying clients – they are largely intended to meet the testing needs of the organization that operates the facility.

4.2.2.2 Challenges in Achieving Outcomes

Interviewees representing the CSA identified the following challenges with respect to the DFL achieving its objectives:

Human resources: As described elsewhere in this report, human resources represent a significant challenge to the DFL in maintaining its capacity and ensuring its utilization. There have been numerous retirements at the DFL in recent years and more are expected within the next two years. DFL staff has decreased in size and this, according to CSA-DFL representatives, has directly affected utilization of facilities and ongoing operations. Lack of permanent staff has also increased the DFL's reliance on contract staff who often lack the highly specialized skills required and thus must be trained on the job. Thus the DFL is investing in training individuals who are not permanent employees. The CSA has implemented a recruitment and professional development program (Professional Development Recruitment Program). In the context of the DFL this will ensure that new staff are "cross trained" in a number of areas. In addition, CSA-DFL management would like to hire more engineers (as opposed to technicians) because they are seen as more portable across different areas within the DFL.

Need for facilities upgrades: A few CSA representatives stated that the DFL is in need of upgrades and a modernization. An internal review undertaken by the CSA concluded that the lack of sufficient investment in the DFL in the past 5 to 10 years has resulted in some deterioration and has put its standing as a world-class testing facility in jeopardy. A few interviewees suggested that most (but not all) of the DFL's maintenance issues could be mitigated through better partitioning of the facilities. In fact the CSA is currently exploring solutions such as building a parallel distributed power system consisting of two independent power networks, so that operation can continue during system tests and upgrades (and power outages). Partitioning of testing areas would allow for rotating shutdowns, thus avoiding the need for a complete shutdown of the DFL. This is the approach used by other large national testing facilities.

Consolidated Revenue Fund: Funds collected from external clients go into the CRF and are not transferred back to the CSA. External clients are thus seen by many interviewees as a net drain on the DFL and CSA resources. However, it should be noted that, although the DFL may be able to increase its utilization (a key performance indicator for the DFL) by serving external clients, this represents a net drain on CSA resources only when utilization of DFL testing facilities exceeds its capacity requiring additional staff and other inputs (e.g., chemicals). In addition, one must bear in mind that the CSA is allocated funding in order to deliver testing via the Qualifying and Testing Services Program. If the CSA received back the funds deposited in the CRF, then this could be deemed duplicating or "double-dipping" on the part of the CSA for the Qualifying and Testing Services Program. The CSA would need to obtain Treasury Board approval to get back the CRF money. However the Treasury Board Secretariat would likely cut the CSA's budget by the same amount in order to prevent the CSA from receiving funding for the program twice (once in its budget allocation and once by receiving funds from the CRF).

Fee schedule: CSA-DFL representatives noted that the fee schedule for DFL services is in line with Treasury Board costing guidelines. However, CSA-DFL representatives and a couple of clients noted that the schedule is in need of updating in order to better reflect actual costs. In fact the external fee schedule was last updated in 2009-10 and the internal costing schedule has not been updated since 2010-11. This implies that, allowing for inflation, the CSA may be undercharging clients for DFL services.

Lack of marketing: As described elsewhere in this report, the DFL is unable to increase its utilization through actively marketing its services because the DFL lacks the resources (human and financial) in order to develop and implement a marketing strategy.

In the past the DFL was able to informally market its services through staff attendance at conferences, workshops and other networking events. However the reduction of travel budgets in the federal government has meant that this informal marketing is much more limited.

4.2.2.3 Unintended Outcomes

No unintended outcomes were identified in the key informant interviews, document review or benchmarking study.

4.2.3 Demonstration of Efficiency and Economy

This section assesses whether the Qualifying and Testing Services Program has been able to generate outputs and achieve its outcomes in an economic and efficient manner. The TBS Directive on the Evaluation Function defines the demonstration of efficiency and economy as the "assessment of resource utilization in relation to the production of outputs and progress towards expected outcomes." In general, the analysis of efficiency requires assessing relationships between inputs and outputs and/or outcomes, and the assessment of economy concerns the extent to which best use is made of resource inputs to achieve intended outcomes.

We address economy and efficiency by assessing:

  • Whether the DFL is producing its outputs and outcomes in the most efficient manner;
  • Whether there are opportunities for improving the overall economy of DFL services; and
  • Whether there are alternative, more efficient or effective ways of delivering AIT services for the benefit of the Canadian space sector.

A key limitation to addressing efficiency and economy for the Qualifying and Testing Services Program is the lack of consistent and reliable utilization data and cost data against which to assess the achievement of outputs and outcomes. In the absence of quantitative data, the evaluation has relied largely on qualitative information from interviews and documentation. The 2010 Audit of the DFL also noted challenges related to assessing efficiency and economy. The Audit noted that the DFL does not have an overall strategy for optimizing the use and cost-effectiveness of facilities. Although the DFL has financial indicators such as the contribution to the CRF, these indicators do not make it possible to report on the cost-effectiveness of the DFL's facilities. The 2010 Audit explained that the reason for this is that the revenues generated by external clients are deposited in the CRF whereas the DFL pays the costs relative to implementing the projects of external clients. The 2010 Audit Report concludes that this situation inhibits any analysis of the efficiency of the DFL in producing its outputs and outcomes and therefore is hampering the DFL's business development activities.

Actual expenditures by the Qualifying and Testing Services Program ranged between $11.0M and $13.3M between 2009-10 and 2013-14 including maintenance. However the DFL invoices the private sector for testing services and the funds collected are deposited in the CRF. Thus the program costs the federal government less than $10M per year to operate. Table 8 summarizes expenditures by the Qualifying and Testing Services Program and the amount deposited into the CRF for the past five years.

Table 8: Net Cost of the DFL, 2009-10 to 2013-14

Fiscal year 2009-10 2010-11 2011-12 2012-13 2013-14 Total
Actual Expenditures ($K)Footnote 27 11,131 13,275 11,881 10,985 11,243 58,515
Funds deposited into the CRF ($K) 1,660 3,650 5,311 2,193 1,940 14,754
Net cost of the DFL (including S&F) 9,471 9,625 6,570 8,792 9,303 43,471

The net cost of delivering the Qualifying and Testing Services Program cost Canadians approximately $43.5M over the past five fiscal years. As with any government program, the cost of operating the program should be assessed against its value – for the Qualifying and Testing Services Program its value is reflected in the importance of providing access to reliable AIT facilities to the Canadian space industry (and to the other sectors that have been served), and the economic, employment and social benefits that are generated. The importance of the space sector in terms of its impact on the Canadian economy, employment and social benefits are reflected in the rationale and ongoing need for the program.

In addition, the ability to have testing performed at the DFL means that clients remain in Canada and costs related to testing are spent in Canada and not abroad. Based on findings from all of the research carried out for this evaluation, the majority of clients would most certainly need to travel outside Canada in order to conduct testing. No other facility in Canada is able to meet the sector's testing needs. In addition, when the DFL undertakes testing for foreign clients these clients contribute to the regional economy through spending on hotels, car rentals, restaurant meals, etc. Findings related to the economy and efficiency of the DFL aligned with a number of issues, including human resources; facilities management; pricing; and utilization. Each of these is addressed in the sections below.

4.2.3.1 Human Resources

A few CSA staff stated that there is an insufficient number of staff who are generalists (i.e., able to work in multiple testing areas) which hinders efficiency, particularly since the DFL sometimes lacks sufficient staff to meet demands for testing. A few interviewees, specifically CSA representatives, suggested that the DFL is in need of more engineers who are more portable across testing areas because they have a higher level of education and training. According to CSA-DFL representatives, the DFL is "cross training" new hires to make them more portable across testing areas. In addition, a few representatives from the CSA suggested that other divisions of the Agency could be a source of engineers for the DFL with CSA HQ and DFL sharing engineering staff. There are currently about 200 engineers employed by the CSA. Sharing of engineering staff between the CSA HQ and the DFL could provide the much-needed human resources at the DFL.

Interviewees representing CSA and client representatives suggested that the DFL could increase efficiency and reduce costs by reviewing their human resource allocations across groups and possibly consolidating some groups. A few CSA-DFL representatives also believed that the DFL could review their staff allocations without affecting the quality or level of service provided to clients. It is in fact part of the longer term management plan to combine two qualification and two client service groups at the DFL.

4.2.3.2 Facilities Management

Challenges related to facilities management at the DFL have been described throughout this report. CSA and client representatives made a number of suggestions related to the management of the DFL's facilities which could improve the efficiency and economy of the DFL.

The DFL facilities were built over 40 years ago and have been added to over the years in a somewhat ad hoc fashion. Currently the building cannot be partitioned in order to undertake maintenance in one area while keeping the remaining areas operating. This is inefficient and ultimately costs the CSA more in terms of maintenance costs.

The implications of shutdowns are described throughout the report. Most interviewees noted that the shutdowns, including the long shutdown in 2013 and other regular facility maintenance requirements necessitating testing work to be stopped or interrupted have an adverse impact on client testing at the DFL. Minimizing shutdowns and improving the reliability of access to DFL facilities would improve the effectiveness and efficiency because testing could be conducted with fewer interruptions without the need to stop and restart testing or to completely restart testing because some tests require continuous testing over a long period of time. An example of one improvement that would mitigate some of the shutdowns or interruptions would be the installation of a parallel distributed power system consisting of two independent power networks, something which the CSA is currently exploring.

Improving cooperation between the different sectors of the CSA and its clients in order to ensure optimal coordination of maintenance and testing would also limit the impact of the maintenance shutdowns.

4.2.3.3 Pricing

Given the limitations on obtaining pricing information from other testing facilities, there is no way to benchmark the DFL's prices against other facilities. However a benchmarking study undertaken by Facet Consulting (2008) concluded that in general the DFL rates are not out of line with other test facilities. The DFL's pricing schedule is currently not reviewed on a regular basis – the internal pricing schedule dates from 2010 and the external pricing schedule was last updated in 2009. At minimum one would expect small adjustments to reflect inflation. According to findings from the benchmarking study, Intespace maintains a high level of loading in its facilities which keeps client costs down and enables Intespace to offer competitive pricing. Each service has a price for usage including a ceiling price and a target price depending on the duration of the test, and the behavior of the product in testing. A fixed price is provided for known products. Space science products tend to be unique and test prices are not usually fixed.

4.2.3.4 Utilization

Although the lack of consistently measured utilization prevents a full assessment of whether the DFL is adequately utilized relative to other similar testing facilities, there is general agreement that increasing the DFL's utilization would boost its efficiency and economy.

Increasing business at the DFL through marketing efforts is not possible at present given the DFL's human resource constraints. CSA-DFL staff noted that they lack the capacity and resources to market the DFL effectively.

There is some evidence based on key informant interviews and the benchmarking study that the DFL could increase its client base by providing more testing services to such industries as telecommunications (antenna testing), aerospace and the military, and pursue new markets, such as mining (heavy vehicles) and automotive. This is not to say that the DFL is refusing to serve non-space clients but rather that there may be room to increase utilization by more actively pursuing business in non-space industries – i.e., marketing (which the DFL is not currently in a position to do).

4.2.4 Alternative Models

With respect to an alternative operating model for the DFL, there is evidence that Government-owned Contractor-operated (GoCo) or public–private partnership (PPP) options for operating the DFL merit further investigation. This is currently being done by the CSA in parallel to this evaluation. A few CSA and client representatives noted that there may be GoCo or PPP options for operating the DFL. However they also noted that these alternatives would also require some level of federal government funding, at least in the short term. It was also noted that privatization of the DFL is not a viable option in the short term because it is very complex to negotiate in terms of staffing and ownership. Based on the experience of the French Space Agency's facility located in Toulouse, it can take as long as 20 years to fully privatize such a facility. It was also suggested that the CSA examine the IABG model in Germany which is employee owned.

A few clients made suggestions for ways in which the DFL could better align itself with industry and other stakeholders. Two client representatives suggested that the DFL could become an AIT centre of excellence in Canada. This would situate the DFL as experts in AIT where their expertise is sought out by programs and they would be more closely involved in the AIT design, planning and execution. These interviewees suggested that as a centre of excellence in AIT, the DFL could also encourage space companies to have offices and staff at the DFL.

5. Conclusions and Recommendations

This section summarizes the conclusions of the evaluation study with respect to each evaluation question and presents the study recommendations. Note that these recommendations assume that the DFL continues to operate using the current business model. If the operating model for the DFL changes then some of the recommendations stemming from this report may become irrelevant.

5.1 Relevance/Need

There is evidence of a strong market for satellites with Euroconsult (2014) estimating approximately 115 satellites to be launched each year between 2014 and 2023.Footnote 28 The Aerospace Review Space Working Group Report (2012) also notes that the global space sector is experiencing exponential growth and transformation. All G20 countries have space programs and more than 50 countries have their own domestic satellite system. Given the relatively strong international market for satellites, there is thus evidence of a need for AIT internationally.

With respect to the relevance of the Qualifying and Testing Services Program, the objectives of this program are aligned with federal priorities as contained in the S&T Strategy (2007), the Space Strategy (2003), and Canada's Space Policy Framework (2014). The program aligns with strategic policy issues related to sovereignty and national reputation on the international stage by maintaining Canada's standing as a Tier 2 country in terms of the development and autonomy of its space program and industry. The program also aligns with policy issues related to employment and industrial development, particularly through its role in supporting the space industry, which is an important driver of research and innovation, and which employs some several thousand people, many of whom are engineers, scientists and technicians (i.e., HQPs). Delivery of the program falls within the federal government's jurisdiction via the Canadian Space Agency Act (1990).

In terms of ongoing need, the need for the DFL to be a full-service facility, which can respond to all testing needs related to the launch of a spacecraft, is less evident today because of the evolution of the space industry towards smaller spacecraft and because large companies are building their own facilities, often with federal and/or provincial government financial support. At the same time, it is not known how many projects are in the pipeline at the CSA for which the DFL would be needed in the short term.

Although there is evidence of a continuing need for testing services such as that provided by the DFL, the space industry is evolving towards smaller satellites and increased testing in-house on the part of large companies. There is currently no mechanism within the CSA to identify the testing needs of the space sector (and other sectors) as they evolve. As a result there is a risk of overlap and duplication of testing services and possibly gaps as well.

Recommendation 1: The CSA should set up a process for consulting with stakeholders representing industry (space and other sectors as appropriate), other federal departments and agencies, universities and CSA clients to provide guidance on current and future testing needs. This consultation process would serve to minimize areas of overlap and duplication with other government and private facilities.

5.2 Performance

5.2.1 Achievement of Outcomes

With respect to achieving its immediate and intermediate outcomes, the Qualifying and Testing Services Program has generally been successful. However there have been significant challenges.

This evaluation was, to some extent, hampered by a lack of quantitative data on performance. A PM Strategy was developed for the Qualifying and Testing Services Program and finalized in March 2014. The PM Strategy has not been fully operationalized. However, in trying to identify and collect data for some of the indicators included in the PM Strategy and through discussions with CSA representatives, it became clear to the evaluation team that many of the indicators in the PM Strategy are not relevant to the way the DFL currently operates. The risk is that in the future the data for the PM Strategy will not be collected and/or the data collected will not be relevant for reporting on the success of the DFL or for decision making about the DFL.

Recommendation 2: The CSA should revise the existing Performance Measurement Strategy to make it more viable and improve performance measurement to better reflect how the DFL operates and the environment it operates in. The client surveys should be modified as necessary to align with the Performance Measurement Strategy.

The program has been able to meet the needs of its clients as demonstrated by high levels of client satisfaction which were confirmed by clients interviewed for this evaluation. However government-wide restrictions on funding for travel has meant that program staff are less able to stay abreast of technological and market developments in the space industry through attendance at networking events. This in conjunction with a lack of sufficient investment in DFL infrastructure may be putting at risk the DFL's standing as a world-class facility. Although the DFL may be experiencing challenges with respect to its budget, there is evidence that the DFL is regularly upgrading its facilities and equipment and is, to the extent possible, responsive to client needs. The evaluation found evidence of a number of upgrades.

Ongoing issues with DFL building maintenance pose a significant challenge from the perspective of both the CSA and DFL clients. The DFL building, built in the 1970s, was not designed so that maintenance could be performed on some parts of the building while keeping the rest open and fully operational. Without a complete redesign of the building there are few options for completely resolving the issue of periodic shutdowns. Some solutions such as building a parallel distributed power system are currently being explored by the CSA consisting of two independent power networks, so that operation can continue during system tests and upgrades (and power outages).

The challenges related to building maintenance are compounded by frequent slippage in DFL clients' schedules which make planning shutdowns well in advance difficult. Resolving this issue is challenging because such delays are characteristic of satellite projects. There is thus a need for ongoing collaboration between CSA sectors and DFL clients in order to minimize the impact of closures when they occur. Without a modification to the building, there is little potential to completely resolve this issue beyond continuing to schedule maintenance well in advance and to communicate this to clients – something which the CSA already does.

There is some evidence that the lack of sufficient investment in the DFL's human resources and the relatively large number of recent and pending retirements have resulted in the program being unable to maintain human resources at a level to meet demand for testing during peak periods without reliance on contract employees. This has been compounded by the fact that program staff tend not to be transferrable across DFL testing areas. There is a sense that having access to more engineers rather than technicians would alleviate this challenge because engineers are seen as having a more developed skill set. The CSA has recently put in place a recruitment process and developed a training program for new recruits, in the context of the DFL this will mean that new staff will be "cross trained" and so they will be more readily transferrable across testing areas. There are currently about 200 engineers employed by the CSA. Sharing of engineering staff could serve to meet some of the demand for engineers at the DFL.

Recommendation 3: The CSA should explore a program of sharing staff, particularly engineers, between CSA HQ and the DFL. This option may allow the DFL to overcome some of its staffing issues over the medium term, while providing experience to engineers at the CSA.

There is a sense within the CSA and among clients that the DFL is not sufficiently utilized, and there are concerns that utilization may decline in the coming years due to the uncertainty over future space missions. However this cannot be quantitatively verified because there is no robust, consistently measured utilization data available. Although there is recognition that there are issues with the utilization data and there is intent to do so, the DFL currently does not have the human resources to develop a model for measuring utilization.

Recommendation 4: The CSA needs to develop and put in place a consistent and robust approach to measuring utilization at the DFL which is consistent with the approach used in other similar facilities internationally.

Based on the DFL's utilization statistics from 2001-02 to 2013-14, there is evidence that over this time period, the DFL's utilization has fluctuated somewhat but utilization levels from 2009-10 to 2012-13 were in fact higher than those for earlier periods. Utilization rates plummeted in 2013-14 as a direct result of the 6-month building shutdown in the 2013 calendar year.

Despite the lack of robust utilization data, there is agreement that the utilization of the DFL should be increased. There is a sense that in the absence of increased activity at the CSA which could bring in more testing to the DFL, there is only limited potential to increase testing for the Canadian space sector. Key alternate markets for DFL services are believed to be the EU and the US although attempting to access these two markets would present significant challenges. Another option is for the DFL to expand its testing services to other industries, something which the DFL already does to some extent. To date no market research has been done to assess whether these are viable options.

Recommendation 5: The CSA should undertake a study to identify realistic potential markets for the DFL beyond the space sector. Based on the study, implement a strategy in keeping with its mandate as a federal agency to develop a small number of potential markets for the DFL which have long-term potential. This strategy should be appropriately resourced in order to increase the likelihood of success.

There is evidence of a need for increased marketing of DFL services. The DFL is unable to undertake formal marketing because it lacks the human and financial resources to do so. In the past the DFL was able to undertake informal marketing activities including attendance at conferences, workshops and other events where there are potential clients. This approach to marketing is consistent with the approach of similar organizations such as ESTEC and Intespace.

There is evidence that the DFL is meeting clients' needs with respect to the provision of technical information.

Almost all clients indicated that they have used the DFL for at least some of their testing needs and will continue to do so. However, larger firms indicated that they have been moving away from relying on the DFL for all of their testing needs and now increasingly use the DFL for overflow (when their own facilities are fully booked) or when they require larger, more specialized equipment they do not possess. There was general agreement on the need for a national testing facility such as the DFL in order to maintain national autonomy in testing and to ensure access to reliable testing facilities for all firms in Canada, regardless of size. The need to maintain Canada's standing as a Tier 2 country in space was also linked to the need for a national testing facility – without a national testing facility Canada would, in effect, become a Tier 3 country and thus dependent on other countries for testing space hardware.

5.2.2 Efficiency and Economy

With respect to economy and efficiency, the program has delivered its outputs and outcomes efficiently and has delivered the program economically.

A key limitation to addressing efficiency and economy for the Qualifying and Testing Services Program is the lack of consistent and reliable utilization data and cost data against which to assess the achievement of outputs and outcomes. In the absence of quantitative data, the evaluation has relied largely on qualitative information from interviews and documentation.

The DFL cost Canadian taxpayers an average of less than $10M (expenditures less funds deposited in the CRF) annually over the five-year period of the evaluation. In contrast, the space industry employs several thousand HQPs and generated revenues of $3.3B in 2012. In addition, having a well-equipped state-of-the-art facility available to all Canadian companies, means that companies are not leaving Canada to undertake testing and thus contributing to the economies of other nations. Money spent on testing at the DFL along with the related expenditures remains within Canada. In addition testing for foreign clients results in an influx of spending in the local (i.e., Canadian) economy on the part of clients who spend on things such as hotels, car rentals and restaurant meals.

Key areas that affect the DFL's efficiency and economy include human resources, facilities management, pricing, and utilization.

With respect to human resources, the lack of staff that can be moved from one testing area to another hinders the DFL's efficiency because the DFL sometimes lacks sufficient staff to meet demands for testing. The CSA is working to remedy this by ensuring new staffs are portable across all testing areas within the DFL. The use of more engineers rather than technicians is also seen as a possible solution because engineers are seen as having a more flexible skill set. Another suggestion made related to reviewing the human resources allocation within the DFL through consolidation of groups.

The inability to shut down only some areas for maintenance is seen as inefficient and adds to the cost of maintenance.

Given the limitations on obtaining pricing information from other testing facilities, it was not possible to benchmark the DFL's fee schedule. However there is evidence that the costing model/fee schedule is outdated because neither the internal nor external costing model have been reviewed and updated in recent years.

Recommendation 6: The CSA should review and revise the costing model for the DFL and ensure that it is updated on a regular basis.

Although the lack of consistently measured utilization limits a fair assessment of whether the DFL is adequately utilized relative to other similar testing facilities, there is general agreement that increasing the DFL's utilization would boost efficiency and economy by more effectively using the DFL's facilities and staff.

With respect to an alternative operating model for the DFL, there is evidence that GoCo or PPP options for operating the DFL merit further investigation. This is currently being done by the CSA in parallel to this evaluation.

Management Response and Action Plan

Responsibility Organization / Function Management response Details of action plan Schedule
Recommendation #1
The CSA should set up a process for consulting with stakeholders representing industry (space and other sectors as appropriate), other federal departments and agencies, universities and CSA clients to provide guidance on current and future testing needs. This consulting process would serve to minimize areas of overlap and duplication with other government and private facilities. Director General, Space Science & Technology SS&T agrees that a consultation process with stakeholders from government, industry, and academia should be established. The goal of this consultation process will be to optimize the provision of services in Canada for the benefit of the space sector as a whole, by exploring a variety of options for service delivery, while also advancing the development of a qualified workforce. However, decisions derived from this consultation process may be partly driven by availability of funds from sources other than the CSA and the organizations. Also, the DFL will continue to have a role in development of a qualified workforce by virtue of its expertise. SS&T will establish consultation groups with Canadian stakeholders from government, industry, and academia. It is expected that Assembly, Integration and Testing (AIT) will be a key topic of discussion in this forum, as this is in line with the innovation mandate of the CSA. The consultation process is associated with the pursuit by SS&T of its mandate to promote innovation. June 2015
Recommendation #2
The CSA should revise the existing Performance Measurement Strategy to make it more viable and improve performance measurement to better reflect how the DFL operates and the environment it operates in. The client surveys should be modified as necessary to align with the Performance Measurement Strategy. DFL Director; and Director General, Space Science & Technology The DFL agrees that the Performance Measurement Strategy should be revised. As a service organization, the DFL's performance is best measured by how well its services fulfil the objectives of the clients it serves. In this sense the outcomes of DFL activities are mostly defined and evaluated by client satisfaction. The DFL will explore, with support from CSA experts, options to focus performance indicators and data collection tools on the appropriate areas. The DFL will revise the Performance Measurement Strategy to realign performance indicators and data collection tools with the reality of the DFL. March 2015
Recommendation #3
The CSA should explore a program of sharing staff, particularly engineers, between CSA HQ and the DFL. This option may allow the DFL to overcome some of its staffing issues over the medium term, while providing experience to engineers at the CSA. Director General, Space Science & Technology; and Director, Human Resources Some mechanisms already exist within the CSA to enable such exchanges, and some examples are extant. The situation could be improved by expanding current programs to include the DFL. The DFL intends to participate in the CSA's programs for development of special skills among engineers, by providing an introduction to AIT, thereby developing useful skills for project engineers, while also discovering talent and interest in pursuing a career in AIT. If successful, this will provide flexibility in assignments and agility in meeting changing demands, by offering opportunities for skilled staff to move between positions and further develop their skills. March 2016
Recommendation #4
The CSA needs to develop and put in place a consistent and robust approach to measuring utilization at the DFL which is consistent with the approach used in other similar facilities internationally. DFL Director The DFL agrees that the utilization model should be updated, and is already developing a model of the DFL based upon client services, rather than facility occupancy, linking systems and resources to the services. This utilization model will be analogous to that used to set the internal fee schedule released in 2010. The DFL is currently finalizing the system inventory associated with each of its services. Once this is done, the DFL will develop and implement a suitable method to measure utilization. March 2016
Recommendation #5
The CSA should undertake a study to identify realistic potential markets for the DFL beyond the space sector. Based on the study, implement a strategy in keeping with its mandate as a federal agency to develop a small number of potential markets for the DFL which have long-term potential. This strategy should be appropriately resourced in order to increase the likelihood of success. DG, Space Science & Technology; and DFL Director The CSA agrees that the market could be expanded beyond the space sector and is aware of some potential markets for its services, based upon past enquiries. The CSA is exploring new business models for the DFL to optimize its utilization and to ensure that the DFL will be able to market its services. March 2016
Recommendation #6
The CSA should review and revise the costing model for the DFL and ensure that it is updated on a regular basis. CSA Accounting Manager and DFL Director Finance and DFL agree that the internal and external fee schedules need to be reviewed and revised on a regular basis. Finance, with DFL support, will take the lead to review and revise the costing model. June 2015

Appendices

Appendix A: Evaluation Matrix

Evaluation Issues, Indicators, Data Sources/Methods

Relevance – Does the Program remain consistent with and contribute to the federal government priorities and address actual needs?

Question Indicators Sources/Methods
1. To what extent is there a continued need for the DFL? How are stakeholder needs expected to evolve over the coming years (i.e., 5 to 10 years)?
How does the DFL support the Canadian space industry?
1.1 Current and anticipated needs for AIT services on the part of the CSA and Canadian stakeholders (industry, government departments, universities).
  • Document review (e.g., State of the Canadian Space Sector, Canada's Space Sector, Aerospace Review, RPP, DPR).
  • Key informant interviews:
    • CSA management and Executive Committee
    • DFL management and staff
    • Canadian clients of the DFL
    • Foreign clients of the DFL
    • Canadian non-clients of the DFL.
  • Survey of Canadian clients.
1.2 Extent to which these current and anticipated needs on the part of the CSA and Canadian stakeholders for AIT services are being or will be met by the DFL. Identification of any gaps in AIT services provided by the DFL.
  • Document review (e.g., previous benchmarking studies by Athena and Facet).
  • Key informant interviews:
    • CSA management and Executive Committee
    • DFL management and staff
    • Canadian clients of the DFL
    • Foreign clients of the DFL
    • Canadian non-clients of the DFL.
  • Survey of Canadian clients.
  • Benchmarking study.
1.3 Importance of the DFL in supporting the competitiveness of the Canadian space sector.
  • Key informant interviews:
    • CSA management and Executive Committee.
    • DFL management and staff.
    • Canadian clients of the DFL
    • Canadian non-clients of the DFL.
  • Survey of Canadian clients.
1.4 Comparison with how other countries with space industries meet their domestic needs for AIT.
  • Document review (e.g., previous benchmarking studies by Athena and Facet).
  • Key informant interviews:
    • CSA management and Executive Committee
    • DFL management and staff
    • Foreign clients of the DFL
  • Benchmarking study.
1.5 Potential impact on the Canadian space sector if the DFL were no longer to exist. Availability of other AIT facilities within Canada, the United States and internationally to meet the needs of the Canadian space industry, government departments, universities.
  • Document review (e.g., previous benchmarking studies by Athena and Facet).
  • Key informant interviews:
    • CSA management and Executive Committee
    • DFL management and staff
    • Canadian clients of the DFL
    • Canadian non-clients of the DFL
  • Survey of Canadian clients.
  • Benchmarking study.
2. Is the DFL aligned with federal government priorities? 2.1 Extent to which the DFL aligns with and supports federal priorities as contained in Canada's Space Policy Framework, etc.
  • Document review (e.g., Canada's Space Policy Framework, the Canadian Space Agency Act, Canada's Economic Action Plan, Whole-of-Government Framework (contains Outcomes and Spending Areas) for the Government of Canada.
  • Key informant interviews:
    • CSA management and Executive Committee
    • DFL management and staff
3. Is the DFL consistent with federal roles and responsibilities? 3.1 Extent to which the DFL is consistent with the CSA's roles and responsibilities.
  • Document review (e.g., the Canadian Space Agency Act).
  • Key informant interviews:
    • CSA management and Executive Committee
    • DFL management and staff
3.2 Extent to which the DFL is consistent with federal roles and responsibilities.
  • Key informant interviews:
    • CSA management and Executive Committee
    • DFL management and staff
    • Canadian clients of the DFL
    • Canadian non-clients of the DFL.

Performance – Has the Program achieved its intended outcomes? Are the most appropriate, efficient and economic means being used to achieve outcomes?

Question Indicators Sources/Methods
4. To what extent has the DFL achieved its immediate outcomes?
4a. Maintained access to reliable space AIT facilities. (Oc1) 4a.1 Maintenance of the DFL's certification and conformance to ISO 9001:2008 standard
  • Performance Measurement Plan (Annual ISO Audits).
4a.2 % of time in the year when DFL facilities are available.
  • Performance Measurement Plan (Quarterly Reports).
4a.3 % of tests not completed due to schedule conflicts.
  • Performance Measurement Plan (Quarterly Reports).
4a.4 % of clients satisfied with the extent to which facilities meet their needs.
  • Performance Measurement Plan (Client Surveys B2, Q1, Q4, Q9).
4a.5 % of clients satisfied with DFL staff skills and attitude.
  • Performance Measurement Plan (Client Surveys Q6, Q7, Q8, A2, B1, B3, B4, C2)
  • Major Program Client Satisfaction Survey Summaries
4a.6 Extent to which the DFL has successfully maintained and upgraded facilities to meet the needs of clients.
  • Document review (e.g., previous benchmarking studies by Athena and Facet).
  • Key informant interviews:
    • CSA management and Executive Committee
    • DFL management and staff
    • Canadian clients of the DFL
    • Foreign clients of the DFL
    • Canadian non-clients of the DFL.
  • Survey of Canadian clients.
4a.7 Extent to which the DFL has developed new services to meet clients' needs.
  • Document review (e.g., previous benchmarking studies by Athena and Facet).
  • Key informant interviews:
    • CSA management and Executive Committee
    • DFL management and staff
    • Canadian clients of the DFL
    • Foreign clients of the DFL
    • Canadian non-clients of the DFL.
  • Survey of Canadian clients.
4b. Maintained utilization of facilities. (Oc2) 4b.1 Number of clients served by type of clients (OGDs, industry, academia, CSA programs).
  • Performance Measurement Plan (Quarterly Reports).
4b.2 Utilization as a function of service area (Environmental, RF and AI).
  • Performance Measurement Plan (Quarterly Reports).
4b.3 % utilization by client type (OGDs, industry, academia, CSA programs).
  • Performance Measurement Plan (Quarterly Reports).
4b.4 % of utilization by Canadian projects versus others.
  • Performance Measurement Plan (Quarterly Reports).
4b.5 % of clients who will use the DFL again or recommend its services.
  • Performance Measurement Plan (Client Surveys B5, B6).
4b.6 Number of projects supported.
  • Performance Measurement Plan (Quarterly Reports).
4b.7 Number of individual tests performed.
  • Performance Measurement Plan (Quarterly Reports).
4b.7 Total number of potential Canadian and foreign clients for DFL services (i.e., market potential of the DFL).
  • Document review (Industry Canada/CSA list of firms listed as space companies).
  • Key informant interviews
    • CSA management and Executive Committee
    • DFL management and staff
    • Canadian clients of the DFL
    • Foreign clients of the DFL
    • Canadian non-clients of the DFL.
4c. Maintained or increased availability of reliable information and services to assist clients in meeting project technical requirements. (Oc3) 4c.1 % satisfaction with information and services.
  • Performance Measurement Plan (Client Surveys A1, A2, A3, C1, Q3, Q5).
4c.2 Availability of equipment specifications and other documents such as test request forms, ROM quotes, client orientation packages (i.e., ability of the DFL to readily provide specifications and other documents to clients/potential clients when needed).
  • Document review (review of specifications documents available through the DFL).
  • Key informant interviews:
    • Canadian clients of the DFL
    • Foreign clients of the DFL
    • Canadian non-clients of the DFL.
5. To what extent has the DFL achieved its intermediate outcome of maintaining Canada's autonomy in space AIT? (Oc4) 5.1 Number/% of Canadian projects using non-DFL AIT facilities (within Canada and outside of Canada).
  • Key informant interviews:
    • Canadian clients of the DFL
    • Canadian non-clients of the DFL
    • Foreign clients of the DFL.
  • Survey of Canadian clients.
5.2 Extent to which the maintenance of Canada's autonomy in space AIT is perceived to be important by Canadian stakeholders.
  • Key informant interviews:
    • CSA management and Executive Committee
    • DFL management and staff
    • Canadian clients of the DFL
    • Canadian non-clients of the DFL.
  • Survey of Canadian clients.
6. What, if any, challenges has the DFL experienced in meeting its objectives? How has the DFL responded to these challenges (e.g., human resources, fee schedule, retention of revenues)? 6.1 Challenges experienced by the DFL and responses to these challenges.
  • Document review (e.g., DPR, RPP, internal DFL documents).
  • Key informant interviews:
    • CSA management and Executive Committee
    • DFL management and staff
  • Survey of Canadian clients.
7. Have there been any unexpected outcomes (positive or negative) as a result of DFL services (Qualifying and Testing Services Program)? 7.1 Positive and/or negative unexpected outcomes as a result of the Qualifying and Testing Services Program.
  • Document review (e.g., internal DFL documents).
  • Key informant interviews:
    • CSA management and Executive Committee
    • DFL management and staff
    • Canadian clients of the DFL
    • Foreign clients of the DFL
    • Canadian non-clients of the DFL.
  • Survey of Canadian clients.
8a) Is the DFL producing its outputs and outcomes (immediate) in the most efficient manner?

8a1 Trends in efficiency and identification of any improvement opportunities, as measured by the following indicators:

  • Planned vs. actual client project costs.
  • Accessibility of facilities.
  • Reliability of facilities.
  • Utilization of facilities.
  • Timeliness of service delivery.
  • Client satisfaction.

8a2 Profile of resources consumed by the DFL, broken down by year, facility, resource type (salary, O&M), and by client type (internal fee for service, external fee for service, other). Also include revenues generated for the Consolidated Revenue Fund.

  • Document review (e.g., previous studies of costs, utilization, etc.).
  • Analysis of file data on project costs, utilization, etc.
  • Key informant interviews:
    • CSA management and Executive Committee.
    • DFL management and staff.
    • Canadian clients of the DFL
    • Foreign clients of the DFL.
  • Survey of Canadian clients.
8b) Are there alternative, more efficient or effective ways of delivering AIT services for the benefit of the Canadian space sector? 8b1 Identification and assessment of possible alternatives for the delivery of DFL services and DFL's resource consumption approaches, and impacts on efficiency and effectiveness.
  • Document review (e.g., previous benchmarking reports by Athena and Facet).
  • Key informant interviews:
    • CSA management and Executive Committee
    • DFL management and staff
    • Canadian clients of the DFL
    • Foreign clients of the DFL
    • Canadian non-clients of the DFL.
  • Survey of Canadian clients.
9. Are there any opportunities for improving the overall economy of DFL services? 9.1 Analysis of overhead and input costs for producing DFL outputs (client projects) and identification of any opportunities for reducing costs.
  • Document review (e.g., internal DFL documents).
  • Analysis of file data on costs.
  • Key informant interviews
    • CSA management and Executive Committee
    • DFL management and staff.

Appendix B: Documents Reviewed

External/Publicly Available Documentation

  1. Aerospace Review, Space Working Group Report, September 2012. Accessed online at http://aerospacereview.ca/eic/site/060.nsf/vwapj/5-Space_Working_Group_Report-Sept12-Final-eng.pdf/$file/5-Space_Working_Group_Report-Sept12-Final-eng.pdf.
  2. Athena Global. Establishing the Market Value of Services Comparable to those Provided by the David Florida Laboratories. July 2006.
  3. Canadian Space Agency, Evaluation Report Template, May 16, 2013.
  4. Canadian Space Agency, Quality Assurance Checklist for Evaluation Reports, July 16, 2012.
  5. Canadian Space Agency, BI PAMF Policy, August 2, 2005.
  6. Canadian Space Agency, Memorandum – Results of the Cassiope Program Level AI&T Survey, December 6, 2009.
  7. Canadian Space Agency, Memorandum – Results of the Sapphire Program Level AI&T Survey, April 15, 2013.
  8. Canadian Space Agency, Memorandum – Results of the M3MSat Program Level AI&T Survey, September 18, 2013.
  9. The Economist, June 7, 2014. Accessed online at www.economist.com/news/technology-quarterly/21603240-small-satellites-taking-advantage-smartphones-and-other-consumer-technologies.
  10. Euroconsult, Profiles of Government Space Programs. February 2014.
  11. Euroconsult, Satellites to be Built and Launched by 2023: World Market Survey. July 2014.
  12. Facet Consulting. A Comparative Analysis of Facilities, Services and Capabilities Provided by the David Florida Laboratories with other Similar Facilities across Canada. August 2008.
  13. Government of Canada. Minister of Industry and Minister of State (Agriculture). Canadian Space Agency. Departmental Performance Report (DPR) 2011-2012. Web. www.asc-csa.gc.ca/eng/publications/pr-2012.asp.
  14. Government of Canada. Minister of Industry and Minister of State (Agriculture). Canadian Space Agency. Departmental Performance Report (DPR) 2010-2011. Web. www.tbs-sct.gc.ca/dpr-rmr/2010-2011/inst/csa/csa00-eng.asp.
  15. Government of Canada. Minister of Industry. Canadian Space Agency. Departmental Performance Report (DPR) 2009-2010. Web. www.tbs-sct.gc.ca/dpr-rmr/2009-2010/inst/csa/csa-eng.pdf.
  16. Government of Canada. Minister of Industry. Canadian Space Agency. Departmental Performance Report (DPR) 2008-2009. Web. www.tbs-sct.gc.ca/dpr-rmr/2008-2009/inst/csa/csa-eng.pdf.
  17. Government of Canada. Minister of Industry and Minister of State (Agriculture). Canadian Space Agency, CSA Report on Plans and Priorities 2013-14. Web. www.asc-csa.gc.ca/pdf/eng/publications/rpp-2013-eng.pdf.
  18. Government of Canada. Minister of Industry and Minister of State (Agriculture). Canadian Space Agency. CSA Report on Plans and Priorities 2012-2013. Web. www.asc-csa.gc.ca/pdf/rpp-2012-eng.pdf.
  19. Government of Canada. Minister of Industry and Minister of State (Agriculture). Canadian Space Agency. CSA Report on Plans and Priorities 2011-2012. Web. http://tbs-sct.gc.ca/rpp/2011-2012/inst/csa/csa-eng.pdf.
  20. Government of Canada. Minister of Industry and Minister of State (Agriculture). Canadian Space Agency. CSA Report on Plans and Priorities 2010-2011. Web. http://tbs-sct.gc.ca/rpp/2010-2011/inst/csa/csa-eng.pdf.
  21. Government of Canada. Minister of Industry. Canadian Space Agency. CSA Report on Plans and Priorities 2009-2010. Web. http://tbs-sct.gc.ca/rpp/2009-2010/inst/csa/csa-eng.pdf.
  22. Government of Canada. Minister of Industry. Canadian Space Agency. CSA Report on Plans and Priorities 2008-2009. Web. http://tbs-sct.gc.ca/rpp/2008-2009/inst/csa/csa-eng.pdf.
  23. Government of Canada. Minister of Industry. Canadian Space Agency. Canada's Space Sector. March 31, 2009. Web. www.asc-csa.gc.ca/pdf/space_espace_eng.pdf.
  24. Government of Canada. Minister of Industry. Canadian Space Agency. Canada's Space Sector. March 31, 2012. www.asc-csa.gc.ca/pdf/eng/industry/state-2012.pdf.
  25. Government of Canada. Canadian Space Agency. Canada's Space Policy Framework, March 2014. Web. www.asc-csa.gc.ca/eng/publications/space-policy/.
  26. Government of Canada. Minister of Industry and Minister of State (Agriculture). Canadian Space Agency. State of the Canadian Space Sector 2012. Web. www.asc-csa.gc.ca/pdf/eng/industry/state-2012.pdf.
  27. Government of Canada. Minister of Industry. Science and Innovation Sector. Mobilizing Science and Technology to Canada's Advantage. 2007. Web. www.ic.gc.ca/eic/site/icgc.nsf/vwapj/SandTstrategy.pdf/$file/SandTstrategy.pdf.
  28. Government of Canada. Minister of Finance. Canada's Economic Action Plan 2013. Web. http://actionplan.gc.ca/en/page/economic-action-plan-2013.
  29. Government of Canada. Minister of Justice. The Canadian Space Agency Act. 2012. Web. http://laws-lois.justice.gc.ca/PDF/C-23.2.pdf.
  30. Government of Canada. Minister of Industry. Aerospace Defence and Marine Branch. Reaching Higher: Canada's Interests and Future in Space, Aerospace Review, V.2. November 2012. Web. http://aerospacereview.ca/eic/site/060.nsf/eng/h_00003.html.
  31. Government of Canada. Minister of Industry and Minister of State (Agriculture). Canadian Space Agency. Canada's Space Policy Framework: Launching the Next Generation. 2014. Web. www.asc-csa.gc.ca/eng/publications/space-policy/default.asp.
  32. Government of Canada. Treasury Board Secretariat (2011) / Whole-of-Government Framework: Descriptions for Government of Canada Outcome Areas. www.tbs-sct.gc.ca/ppg-cpr/descript-eng.aspx.
  33. Government of Canada. Treasury Board Secretariat (2011) / Directive on the Management of Expenditures on Travel, Hospitality and Conferences. www.budget.gc.ca/2013/doc/plan/chap4-1-eng.html.
  34. Government of Quebec. Quebec Premier's Website, Press Release December 13, 2010. "Création de plus de 200 emplois de qualité au sein de l'entreprise Corporation Macdonald, Dettwiler et associés (MDA)" (French only). https://www.premier-ministre.gouv.qc.ca/actualites/communiques/details-en.asp?idCommunique=274.
  35. National Space Agency of Malaysia. Setting-Up The Assembly, Integration And Test Centre In Malaysia, Elena Woo Lai Leng, Maszlan Ismail, Mustafa Din Subari, Malaysia Space Centre (ANGKASA) 2009.
  36. Space Insider, October 17, 2011. Accessed online at www.space.com/13283-small-satellites-cubesats-research-technology.html.

DFL Program Documentation

  • Six year survey feedback statistics
  • Current status of DFL studies 3
  • DFL Audit 2005
  • DFL Audit 2010
  • DFL Mandate
  • DFL Study Final Report
  • DFL Terms and Conditions 2010-02-16
  • DFL Info Package
  • Form No 5 vA 7 Test Request Form
  • Latest Fee Schedule Feb 2011
  • New Equipment Worksheet
  • Revenue vs. Cost 1140403
  • Visio DFL Cost Model Driver
  • CSA-DFL Costing Model Aug 13
  • CSA Final Report Aug 13
  • CSA Good Faith Review

CSA Administrative Documentation

  • CSA-DFL Costing Model August 13
  • Presentation Intespace vs. DFL pages pairs
  • Space Summit 2011
  • Evaluation Report Template May 16, 2013
  • Quality Assurance Checklist for Evaluation Reports July 16, 2012
  • BI PAMF Policy August 2, 2005

Appendix C: Key Informant Interview Guides

Evaluation of the Qualifying and Testing Services Program of the CSA – Interview Guide – Interviews with Canadian clients

A. Background

Thank you for agreeing to be interviewed as part of the Evaluation of the Qualifying and Testing Services Program comprised of the David Florida Laboratories. The Canadian Space Agency (CSA) has engaged a team of outside evaluation specialists to conduct this independent evaluation.

As part of the CSA the DFL provides facilities and environmental testing services to both private and public sector clients on a fee-for-service basis. All systems built by the CSA and its contractors must undergo final space qualification at the DFL.

The mandate of the DFL in delivering the Qualifying and Testing Services Program on a fee-for-service basis is to:

  • Provide ongoing support to meet the objectives of the Canadian Space Program through the qualification of flight hardware;
  • Provide environmental and radio frequency testing at the system and subsystem level to a range of domestic and offshore clients on space-based and terrestrial programs;
  • Manage the ongoing schedule of test support to maximize the quality and integrity of the test environment, and minimize any risk to people, facilities, and the test article's safety;
  • Acquire and develop test technology to provide high-value test services and facilities to effectively support the Canadian Space Program;
  • Provide a mechanism for the transfer to industry of appropriate environmental and radio frequency test technology;
  • Conduct, in conjunction with industry, activities aimed at increasing opportunities for DFL use for Canadian industry and academia; and
  • Ensure the cost-effective operation of the DFL by establishing usage priorities and procedures consistent with optimum use of the facilities. This includes the provision of test support for non-space-related work to achieve uniform facility loading.

The objectives of the evaluation study are to assess the continued relevance of the objectives of the DFL and its performance in achieving its objectives. The Canadian Space Agency must complete the evaluation of the Qualifying and Testing Services Program by the end of September 2014 in order to respond to Treasury Board requirements.

The evaluation involves a number of data collection activities, including interviews within the CSA and the DFL as well as a broad range of clients including industry, other federal departments and agencies, and universities.

Your views will be kept strictly confidential by the evaluation team, and only aggregated results will be included in the evaluation report. Although we may use quotes in reporting they will not be attributed. The names of those interviewed for this evaluation will not be included in the evaluation report. Once approved, the final evaluation report will be made public by the CSA in accordance with Treasury Board policy.

Your interview is expected to last up to one hour. With your permission, the interview may be recorded.

Please review the following questions in advance of your interview. If you have no opinion on a particular question, feel free to skip it.

B. Relevance/Need for the DFL
  1. What are your organization's current needs with respect to AIT? How well does the DFL meet these needs? Are there any gaps in the AIT services provided by the DFL?
    How do you anticipate your organization's needs evolving over the next few years? How well do you anticipate DLF meeting these needs?
  2. In your opinion, what role does the DFL play in supporting the competitiveness of the Canadian space sector? How important is the DFL in maintaining Canadian competitiveness?
    If the DFL were (hypothetically) to no longer exist, are sufficient AIT services available elsewhere (i.e., within Canada, internationally) to meet Canada's space needs?
  3. In your opinion, is there a continued need for the Canadian government to support the availability of AIT services for space?
C. Achievement of Objectives
  1. In your opinion, to what extent has the DFL maintained access to reliable AIT facilities? In general, do DFL facilities meet the needs of clients?
  2. To what extent has the DFL been successful at keeping pace with the evolving technological needs of clients? Has the DFL implemented new services to meet the needs of clients?
  3. As a current client of the DFL, how likely are you to continue to use the DFL to meet your organization's AIT needs? What are the key factors in your organization's decision to use (or not use) DFL facilities?
  4. In your opinion has the DFL successfully maintained utilization of its facilities? What, if any, challenges has the DFL faced in maintaining utilization of its facilities? Is there room for the DFL to increase its market share for AIT services?
  5. Overall, how satisfied are you with the ability of the DFL to provide equipment specifications and other documents such as test request forms, ROM quotes, and client orientation packages? Has the DFL been responsive to your organization's requests for documentation?
    Does the documentation provided by the DFL meet your needs?
  6. Which elements of your organization's AIT needs have been or will be met by facilities other than the DFL? (Please identify the facilities and services other than the DFL that you use or plan to use in Canada and/or outside Canada.)
  7. To what extent do you feel it is important to maintain Canada's autonomy in space AIT? Is the DFL the best (most appropriate) mechanism through which to achieve Canadian autonomy in space AIT? What, if any, alternatives exist?
  8. Have there been any unintended outcomes as a result of DFL activities (either positive or negative)?
D. Economy and Efficiency
  1. Do you have any suggestions as to how the DFL could improve its operational efficiency? Does the DFL need to do any of the following:
    1. Increase its productivity, e.g., reduce the amount of time taken to perform a test so that more tests can be performed?
    2. Increase the reliability of its facilities, e.g., is there any issues with facilities not being available when needed or tests needing to be re-done?
    3. Reduce the inputs/costs of performing tests?
    4. Increase the accessibility of its facilities?
    5. Increase the utilization of its facilities, e.g., by increasing business with existing clients or attracting new clients?
    6. Upgrade or add new facilities to meet client demands?
    7. Increase client satisfaction, in order to attract more business from existing clients and increase referrals to new clients?
  2. In your opinion, from the perspective of the Canadian taxpayer, are there possible alternatives to delivering DFL services to Canadian space sector clients)?
    From the perspectives of both the Canadian taxpayer and Government of Canada, how else, as an alternative(s) to the current DFL, could AIT services be delivered more efficiently or more effectively to Canadian space sector clients? Please explain the value-for-money rationale of any such alternatives.
  3. Do you have any other comments?

Evaluation of the Qualifying and Testing Services Program of the CSA – Interview Guide – Interviews with CSA Management and Executive Committee

A. Background

Thank you for agreeing to be interviewed as part of the Evaluation of the Qualifying and Testing Services Program comprised of the David Florida Laboratories. The Canadian Space Agency (CSA) has engaged a team of outside evaluation specialists to conduct this independent evaluation.

As part of the CSA the DFL provides facilities and environmental testing services to both private and public sector clients on a fee-for-service basis. All systems built by the CSA and its contractors must undergo final space qualification at the DFL.

The mandate of the DFL in delivering the Qualifying and Testing Services Program on a fee-for-service basis is to:

  • Provide ongoing support to meet the objectives of the Canadian Space Program through the qualification of flight hardware;
  • Provide environmental and radio frequency testing at the system and subsystem level to a range of domestic and offshore clients on space-based and terrestrial programs;
  • Manage the ongoing schedule of test support to maximize the quality and integrity of the test environment, and minimize any risk to people, facilities, and the test article's safety;
  • Acquire and develop test technology to provide high-value test services and facilities to effectively support the Canadian Space Program;
  • Provide a mechanism for the transfer to industry of appropriate environmental and radio frequency test technology;
  • Conduct, in conjunction with industry, activities aimed at increasing opportunities for DFL use for Canadian industry and academia; and
  • Ensure the cost-effective operation of the DFL by establishing usage priorities and procedures consistent with optimum use of the facilities. This includes the provision of test support for non-space-related work to achieve uniform facility loading.

The objectives of the evaluation study are to assess the continued relevance of the objectives of the DFL and its performance in achieving its objectives. The Canadian Space Agency must complete the evaluation of the Qualifying and Testing Services Program by the end of September 2014 in order to respond to Treasury Board requirements.

The evaluation involves a number of data collection activities, including interviews within the CSA and the DFL as well as a broad range of clients including industry, other federal departments and agencies, and universities.

Your views will be kept strictly confidential by the evaluation team, and only aggregated results will be included in the evaluation report. Although we may use quotes in reporting they will not be attributed. The names of those interviewed for this evaluation will not be included in the evaluation report. Once approved, the final evaluation report will be made public by the CSA in accordance with Treasury Board policy.

Your interview is expected to last up to one hour. With your permission, the interview may be recorded.

Please review the following questions in advance of your interview. If you have no opinion on a particular question, feel free to skip it.

B. Relevance/Need for the DFL
  1. What are your organization's current needs with respect to AIT? How well does the DFL meet these needs? Are there any gaps in the AIT services provided by the DFL?
    How do you anticipate your organization's needs evolving over the next few years? How well do you anticipate DLF meeting these needs?
  2. In your opinion, what role does the DFL play in supporting the competitiveness of the Canadian space sector? How important is the DFL in maintaining Canadian competitiveness?
    If the DFL were (hypothetically) to no longer exist, are sufficient AIT services available elsewhere (i.e., within Canada, internationally) to meet Canada's space needs?
  3. Are you able to describe how other countries meet their domestic needs for AIT services? Please give examples.
  4. How does the mandate of the DFL currently align with CSA and broader federal government priorities as described in the new Space Policy Framework? Are there any documents or other evidence you can point to that would help to demonstrate this alignment?
  5. In your opinion, is there a continued need for the Canadian government to support the availability of AIT services for space?
  6. Is the DFL well aligned with the role and mandate of the CSA? Would the DFL be a better fit within another federal department or agency?
C. Achievement of Objectives
  1. In your opinion, to what extent has the DFL maintained access to reliable AIT facilities? In general, do DFL facilities meet the needs of clients?
    To what extent has the DFL been successful at keeping pace with the evolving technological needs of clients? Has the DFL implemented new services to meet the needs of clients?
  2. In your opinion has the DFL successfully maintained utilization of its facilities? What, if any, challenges has the DFL faced in maintaining utilization of its facilities? Is there room for the DFL to increase its market share for AIT services?
  3. To what extent do you feel it is important to maintain Canada's autonomy in space AIT? Is the DFL the best (most appropriate) mechanism through which to achieve Canadian autonomy in space AIT? What, if any, alternatives exist?
  4. What, if any, challenges has the DFL encountered in meeting its objectives? How has the DFL responded?
  5. Have there been any unintended outcomes as a result of DFL activities (either positive or negative)?
D. Economy and Efficiency
  1. Do you have any suggestions as to how the DFL could improve its operational efficiency? Does the DFL need to do any of the following:
    1. Increase its productivity, e.g., reduce the amount of time taken to perform a test so that more tests can be performed?
    2. Increase the reliability of its facilities, e.g., is there any issues with facilities not being available when needed or tests needing to be re-done?
    3. Reduce the inputs/costs of performing tests?
    4. Increase the accessibility of its facilities?
    5. Increase the utilization of its facilities, e.g., by increasing business with existing clients or attracting new clients?
    6. Upgrade or add new facilities to meet client demands?
    7. Increase client satisfaction, in order to attract more business from existing clients and increase referrals to new clients?
  2. In your opinion, from the perspective of the Canadian taxpayer, are there possible alternatives to delivering DFL services to Canadian space sector clients)?
    From the perspectives of both the Canadian taxpayer and Government of Canada, how else, as an alternative(s) to the current DFL, could AIT services be delivered more efficiently or more effectively to Canadian space sector clients? Please explain the value-for-money rationale of any such alternatives.
  3. In your opinion, is there any potential for the DFL to further decrease its overhead and input costs without negatively affecting services? Or, are there ways of improving services within the current amount of input costs?
  4. Do you have any other comments?

Thank you.

Evaluation of the Qualifying and Testing Services Program of the CSA – Interview Guide – Interviews with DFL Management

A. Background

Thank you for agreeing to be interviewed as part of the Evaluation of the Qualifying and Testing Services Program comprised of the David Florida Laboratories. The Canadian Space Agency (CSA) has engaged a team of outside evaluation specialists to conduct this independent evaluation.

As part of the CSA the DFL provides facilities and environmental testing services to both private and public sector clients on a fee-for-service basis. All systems built by the CSA and its contractors must undergo final space qualification at the DFL.

The mandate of the DFL in delivering the Qualifying and Testing Services Program on a fee-for-service basis is to:

  • Provide ongoing support to meet the objectives of the Canadian Space Program through the qualification of flight hardware;
  • Provide environmental and radio frequency testing at the system and subsystem level to a range of domestic and offshore clients on space-based and terrestrial programs;
  • Manage the ongoing schedule of test support to maximize the quality and integrity of the test environment, and minimize any risk to people, facilities, and the test article's safety;
  • Acquire and develop test technology to provide high-value test services and facilities to effectively support the Canadian Space Program;
  • Provide a mechanism for the transfer to industry of appropriate environmental and radio frequency test technology;
  • Conduct, in conjunction with industry, activities aimed at increasing opportunities for DFL use for Canadian industry and academia; and
  • Ensure the cost-effective operation of the DFL by establishing usage priorities and procedures consistent with optimum use of the facilities. This includes the provision of test support for non-space-related work to achieve uniform facility loading.

The objectives of the evaluation study are to assess the continued relevance of the objectives of the DFL and its performance in achieving its objectives. The Canadian Space Agency must complete the evaluation of the Qualifying and Testing Services Program by the end of September 2014 in order to respond to Treasury Board requirements.

The evaluation involves a number of data collection activities, including interviews within the CSA and the DFL as well as a broad range of clients including industry, other federal departments and agencies, and universities.

Your views will be kept strictly confidential by the evaluation team, and only aggregated results will be included in the evaluation report. Although we may use quotes in reporting they will not be attributed. The names of those interviewed for this evaluation will not be included in the evaluation report. Once approved, the final evaluation report will be made public by the CSA in accordance with Treasury Board policy.

Your interview is expected to last up to one hour. With your permission, the interview may be recorded.

Please review the following questions in advance of your interview. If you have no opinion on a particular question, feel free to skip it.

B. Relevance/Need for the DFL
  1. Broadly speaking, what are the Canadian space industry's current needs with respect to AIT? How well does the DFL meet these needs? Are there any gaps in the AIT services provided by the DFL?
    How do you anticipate the Canadian space industry's needs evolving over the next few years? How well do you anticipate DLF meeting these needs?
  2. In your opinion, what role does the DFL play in supporting the competitiveness of the Canadian space sector? How important is the DFL in maintaining Canadian competitiveness?
    If the DFL were (hypothetically) to no longer exist, are sufficient AIT services available elsewhere (i.e., within Canada, internationally) to meet Canada's space needs?
  3. Are you able to describe how other countries meet their domestic needs for AIT services? Please give examples.
  4. How does the mandate of the DFL currently align with CSA and broader federal government priorities as described in the new Space Policy Framework? Are there any documents or other evidence you can point to that would help to demonstrate this alignment?
  5. In your opinion, is there a continued need for the Canadian government to support the availability of AIT services for space?
  6. Is the DFL well aligned with the role and mandate of the CSA? Would the DFL be a better fit within another federal department or agency?
C. Achievement of Objectives
  1. In your opinion, to what extent has the DFL maintained access to reliable AIT facilities? In general, do DFL facilities meet the needs of clients?
  2. To what extent has the DFL been successful at keeping pace with the evolving technological needs of clients? Has the DFL implemented new services to meet the needs of clients?
  3. What have been the implications of upgrades implemented by Security and Facilities (S&F) in recent years on client services?
  4. In your opinion has the DFL successfully maintained utilization of its facilities? What, if any, challenges has the DFL faced in maintaining utilization of its facilities? Is there room for the DFL to increase its market share for AIT services?
  5. To what extent do you feel it is important to maintain Canada's autonomy in space AIT? Is the DFL the best (most appropriate) mechanism through which to achieve Canadian autonomy in space AIT? What, if any, alternatives exist?
  6. What, if any, challenges has the DFL encountered in meeting its objectives? How has the DFL responded?
  7. Have there been any unintended outcomes as a result of DFL activities (either positive or negative)?
D. Economy and Efficiency
  1. Do you have any suggestions as to how the DFL could improve its operational efficiency? Does the DFL need to do any of the following:
    1. Increase its productivity, e.g., reduce the amount of time taken to perform a test so that more tests can be performed?
    2. Increase the reliability of its facilities, e.g., is there any issues with facilities not being available when needed or tests needing to be re-done?
    3. Reduce the inputs/costs of performing tests?
    4. Increase the accessibility of its facilities?
    5. Increase the utilization of its facilities, e.g., by increasing business with existing clients or attracting new clients?
    6. Upgrade or add new facilities to meet client demands?
    7. Increase client satisfaction, in order to attract more business from existing clients and increase referrals to new clients?
  2. In your opinion, from the perspective of the Canadian taxpayer, are there possible alternatives to delivering DFL services to Canadian space sector clients?
    From the perspectives of both the Canadian taxpayer and Government of Canada, how else, as an alternative(s) to the current DFL, could AIT services be delivered more efficiently or more effectively to Canadian space sector clients? Please explain the value-for-money rationale of any such alternatives.
  3. In your opinion, is there any potential for the DFL to further decrease its overhead and input costs without negatively affecting services? Or, are there ways of improving services within the current amount of input costs?
  4. Do you have any other comments?

Thank you.

Evaluation of the Qualifying and Testing Services Program of the CSA – Interview Guide – Interviews with Foreign Clients

A. Background

Thank you for agreeing to be interviewed as part of the Evaluation of the Qualifying and Testing Services Program comprised of the David Florida Laboratories. The Canadian Space Agency (CSA) has engaged a team of outside evaluation specialists to conduct this independent evaluation.

As part of the CSA the DFL provides facilities and environmental testing services to both private and public sector clients on a fee-for-service basis. All systems built by the CSA and its contractors must undergo final space qualification at the DFL.

The mandate of the DFL in delivering the Qualifying and Testing Services Program on a fee-for-service basis is to:

  • Provide ongoing support to meet the objectives of the Canadian Space Program through the qualification of flight hardware;
  • Provide environmental and radio frequency testing at the system and subsystem level to a range of domestic and offshore clients on space-based and terrestrial programs;
  • Manage the ongoing schedule of test support to maximize the quality and integrity of the test environment, and minimize any risk to people, facilities, and the test article's safety;
  • Acquire and develop test technology to provide high-value test services and facilities to effectively support the Canadian Space Program;
  • Provide a mechanism for the transfer to industry of appropriate environmental and radio frequency test technology;
  • Conduct, in conjunction with industry, activities aimed at increasing opportunities for DFL use for Canadian industry and academia; and
  • Ensure the cost-effective operation of the DFL by establishing usage priorities and procedures consistent with optimum use of the facilities. This includes the provision of test support for non-space-related work to achieve uniform facility loading.

The objectives of the evaluation study are to assess the continued relevance of the objectives of the DFL and its performance in achieving its objectives. The Canadian Space Agency must complete the evaluation of the Qualifying and Testing Services Program by the end of September 2014 in order to respond to Treasury Board requirements.

The evaluation involves a number of data collection activities, including interviews within the CSA and the DFL as well as a broad range of clients including industry, other federal departments and agencies, and universities.

Your views will be kept strictly confidential by the evaluation team, and only aggregated results will be included in the evaluation report. Although we may use quotes in reporting they will not be attributed. The names of those interviewed for this evaluation will not be included in the evaluation report. Once approved, the final evaluation report will be made public by the CSA in accordance with Treasury Board policy.

Your interview is expected to last up to one hour. With your permission, the interview may be recorded.

Please review the following questions in advance of your interview. If you have no opinion on a particular question, feel free to skip it.

B. Relevance/Need for the DFL
  1. What are your organization's current needs with respect to AIT? How well does the DFL meet these needs? Are there any gaps in the AIT services provided by the DFL?
    How do you anticipate your organization's needs evolving over the next few years? How well do you anticipate DLF meeting these needs?
  2. Are you able to describe how other countries meet their domestic needs for AIT services? Please give examples.
C. Achievement of Objectives
  1. In your opinion, to what extent has the DFL maintained access to reliable AIT facilities? In general, do DFL facilities meet the needs of clients?
  2. To what extent has the DFL been successful at keeping pace with the evolving technological needs of clients? Has the DFL implemented new services to meet the needs of clients?
  3. As a current client of the DFL, how likely are you to continue to use the DFL to meet your organization's AIT needs? What are the key factors in your organization's decision to use (or not use) DFL facilities?
  4. In your opinion has the DFL successfully maintained utilization of its facilities? What, if any, challenges has the DFL faced in maintaining utilization of its facilities? Is there room for the DFL to increase its market share for AIT services?
  5. Overall, how satisfied are you with the ability of the DFL to provide equipment specifications and other documents such as test request forms, ROM quotes, and client orientation packages? Has the DFL been responsive to your organization's request for documentation?
    Does the documentation provided by the DFL meet your needs?
  6. Which elements of your organization's AIT needs have been or will be met by facilities other than the DFL? (Please identify the facilities and services other than the DFL that you use or plan to use in Canada and/or outside Canada.)
  7. Have there been any unintended outcomes as a result of DFL activities (either positive or negative)?
D. Economy and Efficiency
  1. Do you have any suggestions as to how the DFL could improve its operational efficiency? Does the DFL need to do any of the following:
    1. Increase its productivity, e.g., reduce the amount of time taken to perform a test so that more tests can be performed?
    2. Increase the reliability of its facilities, e.g., is there any issues with facilities not being available when needed or tests needing to be re-done?
    3. Reduce the inputs/costs of performing tests?
    4. Increase the accessibility of its facilities?
    5. Increase the utilization of its facilities, e.g., by increasing business with existing clients or attracting new clients?
    6. Upgrade or add new facilities to meet client demands?
    7. Increase client satisfaction, in order to attract more business from existing clients and increase referrals to new clients?
  2. From the perspectives of both the Canadian taxpayer and Government of Canada, how else, as an alternative(s) to the current DFL, could AIT services be delivered more efficiently or more effectively to Canadian space sector clients? Please explain the value-for-money rationale of any such alternatives.
  3. Do you have any additional comments?

Thank you.

Evaluation of the Qualifying and Testing Services Program of the CSA – Interview Guide – Interviews with Non-Clients (Canadian)

A. Background

Thank you for agreeing to be interviewed as part of the Evaluation of the Qualifying and Testing Services Program comprised of the David Florida Laboratories. The Canadian Space Agency (CSA) has engaged a team of outside evaluation specialists to conduct this independent evaluation.

As part of the CSA the DFL provides facilities and environmental testing services to both private and public sector clients on a fee-for-service basis. All systems built by the CSA and its contractors must undergo final space qualification at the DFL.

The mandate of the DFL in delivering the Qualifying and Testing Services Program on a fee-for-service basis is to:

  • Provide ongoing support to meet the objectives of the Canadian Space Program through the qualification of flight hardware;
  • Provide environmental and radio frequency testing at the system and subsystem level to a range of domestic and offshore clients on space-based and terrestrial programs;
  • Manage the ongoing schedule of test support to maximize the quality and integrity of the test environment, and minimize any risk to people, facilities, and the test article's safety;
  • Acquire and develop test technology to provide high-value test services and facilities to effectively support the Canadian Space Program;
  • Provide a mechanism for the transfer to industry of appropriate environmental and radio frequency test technology;
  • Conduct, in conjunction with industry, activities aimed at increasing opportunities for DFL use for Canadian industry and academia; and
  • Ensure the cost-effective operation of the DFL by establishing usage priorities and procedures consistent with optimum use of the facilities. This includes the provision of test support for non-space-related work to achieve uniform facility loading.

The objectives of the evaluation study are to assess the continued relevance of the objectives of the DFL and its performance in achieving its objectives. The Canadian Space Agency must complete the evaluation of the Qualifying and Testing Services Program by the end of September 2014 in order to respond to Treasury Board requirements.

The evaluation involves a number of data collection activities, including interviews within the CSA and the DFL as well as a broad range of clients including industry, other federal departments and agencies, and universities.

Your views will be kept strictly confidential by the evaluation team, and only aggregated results will be included in the evaluation report. Although we may use quotes in reporting they will not be attributed. The names of those interviewed for this evaluation will not be included in the evaluation report. Once approved, the final evaluation report will be made public by the CSA in accordance with Treasury Board policy.

Your interview is expected to last up to one hour. With your permission, the interview may be recorded.

Please review the following questions in advance of your interview. If you have no opinion on a particular question, feel free to skip it.

B. Relevance/Need for the DFL
  1. What are your organization's current needs with respect to AIT? How well does the DFL meet/not meet these needs? Are there any gaps in the AIT services provided by the DFL?
    How do you anticipate your organization's needs evolving over the next few years? Do you anticipate DLF meeting these needs?
  2. In your opinion, what role does the DFL play in supporting the competitiveness of the Canadian space sector? How important is the DFL in maintaining Canadian competitiveness?
    If the DFL were (hypothetically) to no longer exist, are sufficient AIT services available elsewhere (i.e., within Canada, internationally) to meet Canada's space needs?
  3. In your opinion, is there a continued need for the Canadian government to support the availability of AIT services for space?
C. Achievement of Objectives
  1. In your opinion, to what extent has the DFL maintained access to reliable AIT facilities? In general, do DFL facilities meet the needs of clients?
  2. To what extent has the DFL been successful at keeping pace with the evolving technological needs of clients? Has the DFL implemented new services to meet the needs of clients?
  3. As a current non-client of the DFL, are you able to explain the reasons why you have not used the DFL for your AIT needs? What facilities has your organization used for it AIT needs in recent years? Why were these facilities used rather than the DFL? What is the likelihood that you will use the DFL in the future?
  4. In your opinion has the DFL successfully maintained utilization of its facilities? What, if any, challenges has the DFL faced in maintaining utilization of its facilities? Is there room for the DFL to increase its market share for AIT services?
  5. Overall, how satisfied are you with the ability of the DFL to provide equipment specifications and other documents such as test request forms, ROM quotes, and client orientation packages? Has the DFL been responsive to your organization's requests for documentation?
    Does the documentation provided by the DFL meet your needs?
  6. Which elements of your organization's AIT needs have been met by facilities other than the DFL? (Please identify the facilities and services other than the DFL that you use in Canada and/or outside Canada.)
  7. To what extent do you feel it is important to maintain Canada's autonomy in space AIT? Is the DFL the best (most appropriate) mechanism through which to achieve Canadian autonomy in space AIT? What, if any, alternatives exist?
  8. Have there been any unintended outcomes as a result of DFL activities (either positive or negative)?
D. Economy and Efficiency
  1. In your opinion, from the perspective of the Canadian taxpayer, are there possible alternatives to delivering DFL services to Canadian space sector clients)?
    From the perspectives of both the Canadian taxpayer and Government of Canada, how else, as an alternative(s) to the current DFL, could AIT services be delivered more efficiently or more effectively to Canadian space sector clients? Please explain the value-for-money rationale of any such alternatives.
  2. Do you have any additional comments?

Thank you.

Appendix D: Benchmarking Interview Guide

Evaluation of the David Florida Laboratory of the Canadian Space Agency

Benchmarking Interview Guide
A. Background

Thank you for agreeing to be interviewed as part of the Evaluation of the David Florida Laboratory (DFL). The Canadian Space Agency (CSA) has engaged a team of outside evaluation specialists to conduct this independent evaluation.

The David Florida Laboratory (DFL), located in Ottawa, is Canada's main spacecraft assembly, integration and testing centre and is maintained and operated by the Canadian Space Agency (CSA). The purpose of the DFL is to provide, in a cost-effective manner, an environmental test facility capable of meeting current and emerging needs of Canada's space community and the nation's space-related objectives. On a fee-for-service basis, the DFL is available for use by Canadian and foreign aerospace and telecommunication companies and organizations for qualifying their space hardware.

The DFL has unique facilities and capabilities for Canada such as a 7 m (diameter) x 10 metre (length) thermal vacuum chamber, a 178 kN vibration shaker and a large RF Anechoic Chamber suitable to test large satellites such as Radarsat-2. The RF Anechoic Chambers have access to the longest far-field range (370 metres) in Canada.

As part of the evaluation, a benchmarking of similar satellite testing facilities in Canada and in other countries is being conducted to enable a comparative analysis of the testing services and costs of these other facilities in relation to the services provided by the DFL. Interviews with key officers in the facilities is being undertaken to assist this analysis.

The views expressed in the interviews in response to the questions given below will be kept confidential by the evaluation team, and only aggregated results will be included in the evaluation report. Once approved, the final evaluation report will be made public by the CSA in accordance with government policy.

Your interview is expected to last up to one hour. With your permission, the interview may be recorded.

B. Questions for Canadian testing facilities
  1. Comparison of your firm's testing facilities to DFL facilities: How do your firm's facilities compare with the facilities at the DFL in terms of the types of services provided? Are your testing facilities available to users outside your firm? What proportion of the facilities' users is from outside your firm? What are the charge rates for each of your firm's facilities? Are there any documents you can provide that would provide further details in answer to these questions?
  2. Reasons for using your firm's testing facilities or DFL facilities: Why would your firm or other firms or organizations use your firm's facilities for satellite testing and not use DFL facilities? Why would a firm choose to use the DFL? Please give examples to illustrate your answer.
  3. Costing for use of your firm's testing facilities: Are your firm's testing facilities run on a self sustaining basis through user fees that recover capital as well as operations and maintenance costs? If not, what items in the operations and capital costs are borne by your firm?
  4. Need for the DFL: If the DFL were no longer to exist, what facilities would your firm use in Canada or in another country to meet the satellite testing needs of your firm that cannot be met by your own facilities? What, in your view, would be the impact of not having the DFL facility on the Canadian space sector overall?
B. Questions for Foreign testing facilities
  1. Comparison of your testing facilities to DFL facilities: How do your organization's facilities compare with the facilities at the DFL in terms of the types of services provided? Do you see any gaps in the testing services provided by the DFL? Are your facilities available to all potential users from your country or another country, or are there restrictions on what organizations or firms can use the facilities and on what types of satellites or satellite components can be tested? If there are restrictions, please explain. Are there any documents you can provide that would provide further details in answer to these questions?
  2. Reasons for using your organization's or firm's testing facilities: What are the factors that influence users to choose to use your organization's or firm's testing facilities? Does your organization or firm market its testing facilities in order to maintain a flow of users or is this not necessary? Please elaborate on your responses to these questions.
  3. Meeting your country's AIT needs: In your view, is there sufficient satellite testing capacity in your country to meet your country's needs for satellite testing services? If not, what impact does this have on your country's space industry? Are there any documents you can provide that would provide further details in answer to these questions?
  4. Need for the DFL: If the DFL were no longer to exist, would your organization's or firm's testing facilities be available to Canadian users? If so, would there be any restrictions on the types of satellites or satellite components to be tested or whether users are in the public or private sector?

Appendix E: DFL Client Satisfaction Survey Results: 2009-10 to 2013-14

Response Rate for DFL Client Satisfaction Survey, 2009-10 to 2013-14

Fiscal Year Number of Surveys Sent Number of Responses Response Rate (%)Footnote 29
2009-10 85 37 44
2010-11 95 36 38
2011-12 82 28 34
2012-13 56 23 41
2013-14 33 7 21

Overall Satisfaction

Graph. Data table follows.
Fiscal Year
2009-10 2010-11 2011-12 2012-13 2013-14
Exceeded 59% 45% 50% 64% 60%
Met 40% 51% 47% 35% 36%
Not Met 0% 3% 0% 0% 2%
N/A 1% 1% 3% 1% 2%

Source: DFL Client Satisfaction

Providing strategic and relevant information to help in the delivery of clients' program

Graph. Data table follows.
Fiscal Year
2009-10 2010-11 2011-12 2012-13 2013-14
Exceeded 43% 26% 40% 47% 40%
Met 48% 62% 47% 41% 60%
Not Met 0% 3% 0% 0% 0%
N/A 9% 9% 13% 12% 0%

Source: DFL Client Satisfaction Survey.

Being proactive by identifying potential problems before they occur?

Graph. Data table follows.
Fiscal Year
2009-10 2010-11 2011-12 2012-13 2013-14
Exceeded 39% 21% 33% 47% 40%
Met 57% 74% 60% 53% 60%
Not Met 0% 3% 0% 0% 0%
N/A 4% 3% 7% 0% 0%

Source: DFL Client Satisfaction Survey.

Assisting you in your decision making by providing practical recommendations before, during, and after the conduct of your tests?

Graph. Data table follows.
Fiscal Year
2009-10 2010-11 2011-12 2012-13 2013-14
Exceeded 39% 29% 43% 59% 60%
Met 61% 65% 50% 41% 20%
Not Met 0% 3% 0% 0% 0%
N/A 0% 3% 7% 0% 20%

Source: DFL Client Satisfaction Survey.

Demonstrating a solid understanding of your needs and priorities

Graph. Data table follows.
Fiscal Year
2009-10 2010-11 2011-12 2012-13 2013-14
Exceeded 52% 50% 43% 59% 60%
Met 48% 47% 57% 41% 40%
Not Met 0% 3% 0% 0% 0%
N/A 0% 0% 0% 0% 0%

Source: DFL Client Satisfaction Survey.

Delivering quality, cost effective, and timely services i.e., as scheduled?

Graph. Data table follows.
Fiscal Year
2009-10 2010-11 2011-12 2012-13 2013-14
Exceeded 48% 41% 57% 59% 80%
Met 48% 53% 43% 41% 20%
Not Met 4% 6% 0% 0% 0%
N/A 0% 0% 0% 0% 0%

Source: DFL Client Satisfaction Survey.

Being service-oriented by being constructive and helpful?

Graph. Data table follows.
Fiscal Year
2009-10 2010-11 2011-12 2012-13 2013-14
Exceeded 57% 38% 50% 65% 80%
Met 43% 59% 50% 35% 20%
Not Met 0% 3% 0% 0% 0%
N/A 0% 0% 0% 0% 0%

Source: DFL Client Satisfaction Survey.

Providing technical expertise when needed?

Graph. Data table follows.
Fiscal Year
2009-10 2010-11 2011-12 2012-13 2013-14
Exceeded 61% 29% 37% 59% 60%
Met 39% 71% 60% 41% 40%
Not Met 0% 0% 0% 0% 0%
N/A 0% 0% 3% 0% 0%

Source: DFL Client Satisfaction Survey.

Would you use the DFL's test facilities again?

Graph. Data table follows.
Fiscal Year
2009-10 2010-11 2011-12 2012-13 2013-14
Very likely 100% 82% 77% 94% 60%
Likely 0% 12% 23% 6% 40%
Unlikely 0% 6% 0% 0% 0%
N/A 0% 0% 0% 0% 0%

Source: DFL Client Satisfaction Survey.

Would recommend the DFL's test facilities to your colleagues?

Graph. Data table follows.
Fiscal Year
2009-10 2010-11 2011-12 2012-13 2013-14
Very likely 87% 82% 73% 94% 80%
Likely 13% 15% 23% 6% 0%
Unlikely 0% 3% 0% 0% 20%
N/A 0% 0% 3% 0% 0%

Source: DFL Client Satisfaction Survey.

Responding in an effective way to your needs throughout the project?

Graph. Data table follows.
Fiscal Year
2009-10 2010-11 2011-12 2012-13 2013-14
Exceeded 52% 44% 43% 59% 40%
Met 48% 53% 57% 41% 60%
Not Met 0% 3% 0% 0% 0%
N/A 0% 0% 0% 0% 0%

Source: DFL Client Satisfaction Survey.

Using courteous, competent, and professionnal staff?

Graph. Data table follows.
Fiscal Year
2009-10 2010-11 2011-12 2012-13 2013-14
Exceeded 70% 47% 57% 65% 60%
Met 30% 53% 43% 35% 40%
Not Met 0% 0% 0% 0% 0%
N/A 0% 0% 0% 0% 0%

Source: DFL Client Satisfaction Survey.

Appendix F: Networking Events Attended by DFL Staff – 2009-10 to 2013-14

2014

  • Aerospace Testing Seminar in Los Angeles (networking and co-chair)
  • American Institute of Aeronautics and Astronautics (AIAA), Boulder, Colorado, Annual Working Group on Dynamic Space Simulation to provide an opportunity for the exchange of ideas, new technologies and information pertaining to structural testing of space hardware
  • Society For Experimental Mechanics, International Modal Analysis Conference, Orlando, FL, IMAC is an annual conference that has become the largest forum on technologies related to structural dynamics. IMAC Technical Program Co-Chair and Paper reviewer

2013

  • 35th European Space Agency (ESA) Antenna workshop on Antennas and Free Space RF Measurement; visit to EMC and Compact Range Test facility – ESA-ESTEC, Noordwijk, the Netherlands
  • 4th International Workshop on Verification and Testing of Space Systems in Torino, Italy, at Thales Alenia (networking and visit to the Alenia Rome facilities to discuss requested testing to be performed at the DFL)
  • Visited the NASA JPL facilities in Los Angeles as part of the AIAA WG on Space Simulation
  • NATO AVT-210 Specialists' Meeting on Risk and Reliability Assessment and Validation for Small Spacecrafts in Karlstad, Sweden (presentation on recent advances in AIT at the DFL and associated risks, networking with different experts from defence space areas, participation in important discussion on risk and reliability assessment for small S/C's)
  • Technical University of Cracow, Poland, May 2013 (presentation with an overview of DFL AIT capabilities)
  • 4th International Workshop on Verification and Testing of Space Systems, Matera, Italy, acquisition of knowledge about new verification technologies and methodologies
  • IMAC XXXI Conference and Exposition on Structural Dynamics, Garden Grove, CA, allow the DFL to stay current with modal testing trends and technology in order to support modal testing of space hardware, Technical Co-chair
  • 27th Aerospace Testing Seminar, Los Angeles, CA, communicate and exchange knowledge about the improvement and implementation of aerospace testing technology, Technical Co-chair

2012

  • 15th International Symposium of Antenna Technology and Applied Electromagnetic (ANTEM), Toulouse Space Show 2012 – Toulouse, France
  • 2012 Antenna Measurement Techniques Association Symposium (AMTA) and Boeing tour – Seattle, WA
  • Antennas for Space Applications Short Course and ESTEC tour – ESA-ESTEC, Noordwijk, the Netherlands
  • General chair for the International AIAA Space Simulation conference (2012) in Baltimore
  • Attended the Aerospace Testing Seminar in Los Angeles (networking and co-chair)
  • visited the KARI facilities in South Korea
  • AIAA SPACE 2012 Conference and Exhibition, Pasadena, CA (networking with American companies, NASA, DoD, USAF Labs, participated in the AIAA TC Committee)
  • Site Visit to NASA JPL, Pasadena, CA (visit included integration, structural testing and thermal qualification facilities and special Mars analogue yard for testing of the engineering model of Curiosity Mars Rover)
  • Site Visit to ITL, Markham, ON (visit included presentation on DFL AIT capabilities, discussions and tour of their facility)
  • Site Visit to MPB Communications, Montreal, QC (presentation on DFL capabilities, discussions and tour of their facility)
  • IMAC-XXX: Conference & Exposition on Structural Dynamics, Jacksonville, FL, allow the DFL to stay current with modal testing trends and technology in order to support modal testing of space hardware, Technical Co-chair
  • Canadian Space Summit, Calgary, AB, Canada's leading conference devoted to space science, research and technology
  • 12th European Conference on Spacecraft Structures, Materials & Mechanical Testing 2012, Noordwijk, the Netherlands, Conference on Spacecraft Structures, Materials & Mechanical Testing covering the structural engineering aspects of scientific, earth observation and telecommunication satellites

2011

  • 2011 Antenna Measurement Techniques Association Symposium (AMTA) – Denver, CO
  • 2011 European Conference on Antennas and Propagation (EuCAP); Thales Alenia Space Rome Facility Tour, EMC facility and Nearfield test ranges – Rome
  • International Workshop on Multipactor, Corona and Passive Intermodulation (Mulcopim) – Valencia, Spain
  • Visited the Orbital facilities in Dulles, Washington as part of the AIAA WG on Space Simulation
  • ICES 2011, the AIAA International Conference on Environmental Systems, Portland, OR (networking with American and international companies)
  • AIAA Space 2011 Conference, Long Beach, CA (networking with American major space companies, NASA, DoD, USAF and Missile Centre, etc., participated in the AIAA TC Committee)
  • 33rd ESA Antenna Workshop, ESTEC, Nordwijik, the Netherlands (Dr. Shantnu Mishra participated in this Workshop, networking, two side meetings – one on MATED program with ESA, and other with ESA and SG Microwave on the feasibility of DFL participation in the PIM localization technique development using spherical near field measurements (in relation also to ARTES program))
  • Site Visit to DRDC Valcartier, QC, Hyper Velocity Impact (HVIT) facilities (presentation on DFL capabilities, discussions and consultations, tour of their facilities)
  • Site Visit to ABB Bomem Group, Quebec City, QC (presentation on DFL capabilities, discussion and consultations, tour of their facilities)
  • Site Visit to TRIUMF - Canada's National Laboratory for Particle and Nuclear Physics, Vancouver, BC (presentation on DFL capabilities, discussions and consultations, tour of their facilities)
  • Site Visit to Simon Fraser University, Department of Electrical Engineering (RF Group), Vancouver, BC (presentation on the DFL, discussion and tour of the facilities)
  • Site Visit to University of Calgary, Department of Physics (Institute for Space Imaging Science), and Department of Electrical and Computer Engineering, Calgary, AB (two presentations on DFL testing capabilities, discussions and tour of their facilities)
  • 2011 LMS Americas Aerospace Engineering Conference, Huntington Beach, CA, allow the DFL to stay current with latest technological evolution in test-based engineering
  • 26th Aerospace Testing Seminar (ATS), Manhattan Beach, CA, Development of improved test techniques for CSA space programs
  • IMAC XXiX Conference & Exposition on Structural Dynamics, Jacksonville, FL, development of improved test techniques for CSA space programs, Technical program Co-chair

2010

  • 2010 Antenna Measurement Techniques Association Symposium (AMTA) – Atlanta, GA
  • 2010 International Symposium on Antenna Technology and Applied Electromagnetics and American Electromagnetics Conference (ANTEM/AMEREM)
  • 2010 IEEE Antennas and Propagation Society (APS) and CNC/USNC/Union Radio Science International (URSI) Symposium – Toronto, ON
  • 2010 European Conference on Antennas and Propagation (EuCAP) – Barcelona, Spain
  • ASTRO 2010 Conference, Toronto, ON (networking with Canadian space companies, two papers presentations)
  • AIAA SPACE 2010 Conference, Anaheim, CA (networking with American major space companies, NASA, DoD, one paper presented, participated in the AIAA TC Committee)
  • Site Visit to University of Waterloo, Dept. of Electrical and Computer Engineering, Waterloo, ON, to see newly built antenna testing chamber and range (presentation on the DFL and consultations were provided for University of Waterloo by RFQF)
  • Site Visit to COM DEV facilities in Cambridge, ON (presentation on the DFL, tour of their facilities, discussions and consultations). Two facilities were visited: COM DEV International Products, 155 Sheldon Drive; and COM DEV Canada, Mission Development Group, 60 Struck Court.
  • Site Visit to Space Flight Laboratory (SFL), UTIAS, Toronto, ON (presentation on the DFL and consultations, discussions, tour of their facilities)
  • Site Visit to York University, The Centre for Research in Earth and Space Science (CRESS), Toronto, ON (presentation on the DFL and consultations, tour of their facility)
  • 26th Space Simulation Conference, Annapolis, Maryland, unique opportunity to network and exchange information and ideas for environmental testing of space mission hardware, Technical Program Chair
  • Satellite 2012, Washington, D.C., high level meetings with international partners
  • IMAC XXVIII Conference & Exposition on Structural Dynamics, Jacksonville, FL, development of improved test techniques for CSA space programs, Technical program Co-chair

Footnotes

Footnote 1

However many of these clients are under contract to the CSA.

Return to footnote 1 referrer

Footnote 2

DFL is certified to ISO 9001:2008: radio frequency, structural and thermal qualification testing of space-bound and terrestrial hardware.

Return to footnote 2 referrer

Footnote 3

Represents amounts invoiced to external, i.e., private sector clients.

Return to footnote 3 referrer

Footnote 4

Additional foreign clients were interviewed for the Benchmarking study.

Return to footnote 4 referrer

Footnote 5

Establishing the Market Value of Services Comparable to those provided by DFL; Athena Global; July 2006
A Comparative Analysis of the Facilities, Services and Capabilities provided by DFL with Other Similar Facilities across Canada; Facet consulting Ltd., August 12, 2008.

Return to footnote 5 referrer

Footnote 6

Reaching Higher: Canada's Interest in Space, Volume 2, Aerospace Review, November 2012.

Return to footnote 6 referrer

Footnote 7

Euroconsult, Satellites to be Built and Launched by 2023: World Market Survey, July 2014.

Return to footnote 7 referrer

Footnote 8

The Economist, June 7, 2014. Accessed online at www.economist.com/news/technology-quarterly/21603240-small-satellites-taking-advantage-smartphones-and-other-consumer-technologies.

Return to footnote 8 referrer

Footnote 9

The Economist, June 7, 2014. Accessed online at www.economist.com/news/technology-quarterly/21603240-small-satellites-taking-advantage-smartphones-and-other-consumer-technologies.

Return to footnote 9 referrer

Footnote 10

Space Insider, October 17, 2011. Accessed online at www.space.com/13283-small-satellites-cubesats-research-technology.html.

Return to footnote 10 referrer

Footnote 11

Euroconsult, Profiles of Government Space Programs. February 2014.

Return to footnote 11 referrer

Footnote 12

Canadian Space Agency, State of the Canadian Space Sector, 2012. Accessed online at www.asc-csa.gc.ca/pdf/eng/industry/state-2012.pdf.

Return to footnote 12 referrer

Footnote 13

Aerospace Review, Space Working Group Report, September 2012. Accessed online at http://aerospacereview.ca/eic/site/060.nsf/vwapj/5-Space_Working_Group_Report-Sept12-Final-eng.pdf/$file/5-Space_Working_Group_Report-Sept12-Final-eng.pdf.

Return to footnote 13 referrer

Footnote 14

Quebec Premier's Website, Press Release, December 13, 2010. "Création de plus de 200 emplois de qualité au sein de l'entreprise Corporation Macdonald, Dettwiler et associés (MDA)" (French only). https://www.premier-ministre.gouv.qc.ca/actualites/communiques/details-en.asp?idCommunique=274. The Quebec provincial government's investment in the facility is expected to result in the creation of 200 jobs and the maintenance of 100 existing jobs. It is also expected that the MDA facility will attract other companies to locate in the region.

Return to footnote 14 referrer

Footnote 15

Facet Consulting Ltd. A Comparative Analysis of the Facilities, Services and Capabilities Provided by the David Florida Laboratory and other Similar Facilities across Canada, Final Report 2008.

Return to footnote 15 referrer

Footnote 16

Canadian Space Agency. Canada's Space Policy Framework, March 2014.

Return to footnote 16 referrer

Footnote 17

Euroconsult, Profiles of Government Space Programs. February 2014.

Return to footnote 17 referrer

Footnote 18

Bill C-16, the Canadian Space Agency Act.

Return to footnote 18 referrer

Footnote 19

Although the response rates for DFL surveys are in line with those for client satisfaction surveys without a response incentive a few factors explain the response rate. Many DFL clients are return clients and once they complete one survey they are likely less inclined to respond to subsequent surveys. In addition, DFL sends a survey for each test program completed for a client and so if there are multiple tests to be performed the client will receive a survey multiple times for the test program. In these situations once the client completes one survey they are likely less inclined to respond again.

Return to footnote 19 referrer

Footnote 20

CSA, Memorandum - Results of the Cassiope Program Level AI&T Survey, December 6, 2009; CSA, Memorandum - Results of the Sapphire Program Level AI&T Survey, April 15, 2013; CSA, Memorandum – Results of the M3MSat Program Level AI&T Survey, September 18, 2013.

Return to footnote 20 referrer

Footnote 21

Travel budgets within the CSA are based on the recommendations of a working group which are then reviewed and approved by the CSA's Executive for each sector within the CSA. Each Director General is then responsible for allocating a travel budget within their sector.

Return to footnote 21 referrer

Footnote 22

In addition to previous measures dating back to 2008, Directive on the Management of Expenditures on Travel, Hospitality and Conferences was introduced on January 1, 2011, to ensure that travel expenditures are managed in a more effective and economical manner. The Directive was strengthened in October 2012 by requiring increased oversight of all department-related events where total planned costs exceed $25,000. The Economic Action Plan 2013 further strengthened the Directive by requiring additional oversight of travel authorization and promoting alternatives to travel. To complement these efforts, departmental spending on public service travel was reduced by 5 per cent, or $42.7 million, on an ongoing basis, beginning in fiscal year 2013–14. www.budget.gc.ca/2013/doc/plan/chap4-1-eng.html

Return to footnote 22 referrer

Footnote 23

The DFL does not have an organization-wide method for measuring utilization. The approach to measuring utilization varies across testing facility. As a result, these data are not a reliable measure of the DFL's utilization.

Return to footnote 23 referrer

Footnote 24

Number presented include term employees (i.e., not permanent staff) but not those on short-term contract, total number of FTEs thus underestimates the DFL's workforce.

Return to footnote 24 referrer

Footnote 25

Computer Services (CS) have been centralized federal government-wide and are now part of Shared Services.

Return to footnote 25 referrer

Footnote 26

Aerospace Review, Space Working Group Report, September 2012. Accessed online at http://aerospacereview.ca/eic/site/060.nsf/vwapj/5-Space_Working_Group_Report-Sept12-Final-eng.pdf/$file/5-Space_Working_Group_Report-Sept12-Final-eng.pdf.

Return to footnote 26 referrer

Footnote 27

Actual expenditures are slightly higher because the values in the table exclude some costs such as EBP (Employee Benefit Plan) and internal services such as human resources, finances, legal services, etc. The CSA is unable to extract these costs specifically for the DFL.

Return to footnote 27 referrer

Footnote 28

Euroconsult, Satellites to be Built and Launched by 2023: World Market Survey, July 2014.

Return to footnote 28 referrer

Footnote 29

Although the response rates for DFL surveys are in line with those for client satisfaction surveys without a response incentive a few factors explain the response rate. Many DFL clients are return clients and once they complete one survey they are likely less inclined to respond to subsequent surveys. In addition, DFL sends a survey for each test program completed for a client and so if there are multiple tests to be performed the client will receive a survey multiple times for the test program. In these situations once the client completes one survey they are likely less inclined to respond again.

Return to footnote 29 referrer