Audit of the Management Framework for Safety and Mission Assurance - Audit Report

Download the PDF version (745 KB)

Project No. 14/15 01-02

Prepared by
the Audit and Evaluation Directorate

June 2015

1.0 Summary

1.1 Audit objective

The objective of the audit project was to determine whether the Canadian Space Agency (CSA) has a management framework in place that enables it to achieve its safety and mission assurance (S&MA) objectives.

1.2 Audit opinion

In our opinion, the management framework in place enables the CSA to achieve its safety and mission assurance (S&MA) objectives. However, we identified certain deficiencies which represent moderate issues that require management's attention.

1.3 Statement of assurance

It is my opinion, as Chief Audit Executive, that sufficient and appropriate audit procedures have been conducted and evidence gathered to support the accuracy of the opinion provided in this report. This opinion is based on a comparison of the conditions, as they existed at the time of the audit, against pre-established criteria that were agreed on with management. The opinion is only applicable to the entity examined. The evidence was gathered in compliance with the Treasury Board's internal audit policy, guidelines and standards. The procedures followed comply with the professional standards of the Institute of Internal Auditors. The evidence gathered is sufficient to convince senior management of the validity of the opinion derived from the internal audit.

1.4 Summary of recommendations

The CSA developed its Safety and Mission Assurance Policy and related practices around 1999-2000. The aim of this policy was to meet NASA safety and mission assurance requirements for Canada's participation in the International Space Station project, and also to enable the organization to provide products and services that meet or exceed requirements related to safety, performance, quality, cost and schedules. The policy applies to all of the organization's programs and projects, except for infrastructure and facility projects and certain experimental equipment projects. Contractors who do business with the CSA are also required to apply S&MA practices and methods that meet the requirements of the policy.

In this audit, we selected and examined the S&MA activities for a sample of 5 projects from a total of 19. We noted that the policy was applied in a structured and consistent manner for the major projects that were examined. The quality and quality assurance requirements for these projects are clearly defined, and work plans have been developed and implemented. However, we found some deficiencies in the application of the policy for certain smaller projects such as microsatellite projects. In particular, with respect to the establishment of quality and quality assurance requirements for contractors and in the development and implementation of work plans.

We also noted that some key elements of the quality assurance process are subject to limited guidelines. While objectives and general principles have been developed and are used to guide S&MA work, they are somewhat vague and do not provide the level of detail required for the structured application of the policy. For example, a simplified quality assurance approach is sometimes used in order to minimize project costs. However, there are no established guidelines to assist managers with such an approach. In addition, most of the activities related to the design and construction of CSA's space projects are handled by contractors. Furthermore, guidelines to assist stakeholders addressing quality assurance matters during the contract award and management process are limited.

Whereas in terms of the S&MA function's role within the CSA, the policy emphasizes oversight, the S&MA function actually mainly performs quality assurance tasks in a matrix support role. In addition, the concept of independence of the function is not clearly defined.

These findings lead us to formulate the following recommendations:

  1. Update the CSA's Safety and Mission Assurance Policy in order to clarify the roles and responsibilities of stakeholders.
  2. Develop clear guidelines on the application of the policy, including the following:
    • The establishment of quality and quality assurance requirements based on specific criteria such as the nature, the complexity and the phase of the project;
    • The communication of requirements to contractors doing business with the CSA;
    • The evaluation of the contractors' plans and their capacity to meet the CSA's S&MA requirements;
    • The development of an internal methodology for applying and monitoring the requirements and the policy as a whole;
    • Follow-up on deficiencies identified during quality assurance reviews and implementation of action plans;
    • Reports to be produced and accountability for S&MA activities.
  3. Develop standard processes and tools for planning, carrying out and documenting S&MA activities.

Signature of the Chief Audit Executive

Audit team member

  • Dany Fortin
  • Johanna Gailer

2.0 Audit report

2.1 Background and risk

The mandate of the Canadian Space Agency is to promote the peaceful use and development of space, to advance the knowledge of space through science and to ensure that space science and technology provide social and economic benefits for Canadians.

When designing, carrying out and managing space programs and/or projects, the CSA aims to:

  • protect human life, space products and services, investments and the environment;
  • provide assurance that space products meet or exceed customer requirements for safety, performance, quality, cost and schedules;
  • ensure that space products accomplish their defined mission objectives reliably and safely.

To achieve these objectives, the CSA applies its S&MA policy to programs and projects involving space-based equipment, ground segment equipment and test equipment. All CSA organizational units, along with subcontractors performing work under contracts, are responsible for compliance with the policy.

To oversee the application of the policy, the CSA created an internal function called Safety and Mission Assurance (S&MA). Its mandate is to ensure that CSA programs and projects comply with safety and mission assurance strategies, policies, agreements and standards. The S&MA function carries out activities in the following areas: safety, reliability, quality assurance, assurance of software products, electronic, electrical and electromechanical parts, materials and process, and certification of flight readiness.

The S&MA function reports to the Space Science and Technology Directorate, which itself reports to the President of the CSA. The S&MA team consists of 12 employees, including one manager, three senior engineers and eight engineers. The budget allocated to this function for fiscal year 2014–2015 is $1,397,000, that is, $1,333 000 in salaries and $64,000 in operating expenses.

2.2 Audit objective, scope and approach

Objective

The objective of the audit project was to determine whether the CSA has a management framework in place that enables it to achieve its safety and mission assurance (S&MA) objectives.

Scope

The internal audit project focused on:

  • The management and governance framework for S&MA activities at the CSA.
  • S&MA practices at the CSA.

A sample of five projects was selected and examined from a total of 19 projects that were active between 2012–2013 and 2013–2014.

Approach

The audit criteria were established in accordance with best management practices. The audit criteria and sub-criteria are presented in a table appended hereto. The audit involved various audit procedures such as interviews and document reviews.

It should be noted that the audit objective and audit criteria were discussed with management.

2.3 Findings, recommendations and management responses

In order to determine whether the CSA has a management framework in place that enables it to achieve its safety and mission assurance objectives, we expected to find the following elements:

  • A management and governance framework for S&MA activities at the CSA, including:
    • A clear policy on S&MA that is specific to the CSA;
    • Guidelines describing S&MA practices;
    • Systematic planning of activities;
    • Standardized processes, management tools and controls to ensure the quality of S&MA work;
    • Accountability for S&MA activities.
  • Effective S&MA practices that are applied to CSA projects, including:
    • Planning of S&MA activities is carried out before a project begins and is updated as needed;
    • S&MA policy and guidelines are applied to all projects covered by the policy;
    • Work performed by the S&MA function is properly documented;
    • Identified deficiencies in relation to S&MA activities are subject to a documented follow-up.

2.3.1 Management and Governance Framework

Audit objective: To determine whether the Canadian Space Agency (CSA) has a management framework in place that enables it to achieve its safety and mission assurance objectives.

Findings
Criterion 1

A management and governance framework for S&MA activities is in place at the CSA.

Condition

A clear S&MA policy specific to the CSA is in place.

A Safety and Mission Assurance Policy was established at the CSA around 1999-2000. The aim of the policy was to meet NASA safety and mission assurance requirements for Canada's participation in the International Space Station project, and also to enable the organization to provide products and services that meet or exceed requirements related to safety, performance, quality, cost and schedules. The policy applies to all of the organization's programs and projects, except for infrastructure and facility projects and certain experimental equipment projects. Contractors who do business with the CSA are also required to apply S&MA practices and methods that meet the requirements of the policy.

In terms of responsibility, the policy states that managers of programs and projects must comply with the policy and apply it in coordination with the internal Safety and Mission Assurance function, which consists of a group of CSA engineers specializing in quality assurance.

According to the policy, the main responsibilities of this internal function are to:

  • ensure that CSA programs and projects comply with safety and mission assurance strategies, policies, agreements and standards;
  • oversee the application of the safety and the quality assurance of missions and conduct independent evaluations to ensure that appropriate risk management practices are used in order to identify, document, evaluate and eliminate S&MA risks in all programs, projects and operations.

For the five projects examined, we reconciled the policy requirements and the S&MA activities carried out by program managers and S&MA staff. We found that, most of the time, S&MA staff work directly with program staff in a matrix support role. Their work involves providing the quality assurance expertise needed to determine whether deliverables received from contractors meet requirements. For certain programs, independent certification of flight readiness is conducted when required.

A matrix support role for programs and projects differs in several aspects from the role of oversight and independent assessment of practices that is outlined in the policy. Indeed, matrix support can be likened to an in-house support function, whereas oversight relates to a "checks and balances" role providing independent assurance of S&MA practices. The concept of independent oversight is difficult to apply in a case where S&MA staff are directly involved in project quality assurance work.

In our view, since the Safety and Mission Assurance Policy was implemented more than 15 years ago, it should be updated in order to clarify the roles and responsibilities of the stakeholders. This should include clarification on when S&MA staff should provide matrix support and when they should ensure oversight of S&MA activities. An update of the policy began some time ago but has still not been completed.

Guidelines describing S&MA practices at the CSA are in place.

Guidelines outlining the S&MA objectives, principles and management approach have been developed and are applicable to CSA projects and programs. They are set out in a guideline document entitled "Safety and Mission Assurance Practice", which was developed concurrently with the CSA's S&MA policy around 1999–2000.

The S&MA guidelines indicate that the S&MA discipline covers S&MA management, safety, reliability, software mission assurance, quality assurance, electronic, electrical, and electromagnetic parts, flight readiness certification, materials, mechanical parts and processes. They define the corresponding objectives and principles with the aim of achieving the overall S&MA objectives throughout the complete lifecycle of the products. The requirements set forth in the guidelines are to be tailored to the needs and classes of specific projects.

The guidelines require that the S&MA policy be applied to all phases of a project and managed in such a way that:

  • The organization, requirements, methods, standards, tools and resources for S&MA work are well defined before the project development phase and fully implemented;
  • Problems are identified and adverse consequences are prevented;
  • The end product conforms to the specifications, and non-conformances are addressed in an appropriate manner.

The guidelines document sets out the basic principles governing the organization's S&MA activities, the main ones being:

  • Define project activities in an S&MA plan, including the objectives, requirements, critical elements and constraints;
  • Allocate adequate resources, staff and facilities to carry out the required S&MA tasks;
  • Identify, define and document the responsibility/interfaces of each project organization in the S&MA plan;
  • Ensure that contractors, subcontractors and suppliers perform proper S&MA monitoring and control;
  • Monitor progress and report on all S&MA matters, in particular those related to alerts, critical items and non-conformances.

These objectives and general principles help to provide a framework for the S&MA work done within the organization. However, based on a review of the selected files and based on interviews with project leaders and S&MA staff, we identified key aspects of the quality assurance process that are not covered or are barely touched upon in the guidelines. Indeed, a simplified quality assurance approach is sometimes used to minimize project costs; however, there are no guidelines to help managers who use such an approach. In addition, most of the activities related to the design and construction of CSA's space projects are handled by contractors. However, guidelines to assist stakeholders addressing quality assurance matter during the contract award and management process are limited.

The guidelines would provide a more effective framework for quality assurance activities if they included the CSA's expectations with respect to the following:

  • The establishment of quality and quality assurance requirements based on specific criteria such as the nature, the complexity and the phase of the project;
  • The communication of requirements to contractors doing business with the CSA;
  • The evaluation of contractors' plans and their capacity to meet the CSA's S&MA requirements;
  • The development of an internal methodology for applying and monitoring the requirements and the policy as a whole (e.g. S&MA plan, quality control, process audit, acceptance of deliverables, flight readiness certification);
  • Follow-up on observations and implementation of action plans;
  • Reports to be produced and accountability for S&MA activities.

Systematic planning of S&MA activities is carried out periodically.

We noted that planning of S&MA activities is carried out periodically through the Agency's work plans, operational plans and human resource allocation system. S&MA activities as well as resources are identified for individual projects and are periodically reviewed. In addition, there are regular discussions between the S&MA manager and program and project managers about the activities.

Standardized processes and management tools are defined to ensure the quality of S&MA work.

During the file review, we noted that many processes and tools are used in order to plan, report on and monitor the results of S&MA work. However, they differ from one project and work team to the next. Although the specific environment and requirements differ, certain aspects of projects are similar and would benefit from standardization. This would ensure the quality of the tools used, the information gathered and the processes implemented. It would also ensure the continuity of activities and best practices in the event of the departure of employees with the expertise.

There is accountability for S&MA activities.

The French version of the S&MA guidelines stipulates that all aspects of S&MA will be monitored and that reports will be prepared. However, the English version of the document is less specific and simply sets forth the requirement to "Monitor progress and report on all S&MA matters." The English version appears to suggest that accountability requirements can be met in a less formal manner, for example, through working documents, meetings or emailed notices. Our file review showed that these requirements are addressed in a less formal manner. Clarification should be provided regarding the intent of the guidelines, in order to eliminate any confusion.

Accountability for S&MA activities at the CSA is integrated with corporate accountability for the different programs. The planned activities are identified at the start of the year as part of the CSA's integrated management cycle. The S&MA function submits detailed operational plans describing the activities to be carried out during the coming year, the resources that will be allocated to them, the desired outputs and the performance targets. Follow-up is done at mid-year and at year-end.

In addition to meeting corporate accountability requirements, the S&MA function sometimes produces formal reports, such as audit reports and reports on lessons learned. Ideally, the guidelines should specify under what circumstances such reports must be produced.

Finally, each work team in the S&MA function prepares a weekly progress report on activities associated with programs and projects under their responsibility. The manager of the S&MA function then compiles a summary and presents it to his or her supervisor.

Causes

Policy and guidelines

The S&MA policy and guidelines were developed a number of year ago and need to be updated to reflect the CSA's current activities.

Standardized processes and tools

Developing standardized processes and tools for S&MA activities would require an appreciable time investment by the most experienced S&MA employees. Assigning the required staff to such an undertaking presents a challenge, because up until now priority has been given to programs and projects.

Effect

Policy and guidelines

Lack of clarity related to the roles of stakeholders as well as lack of clear guidelines on the application of the policy.

Standardized processes and tools

There are no established standards for ensuring the quality of the tools used, the information gathered and the processes implemented for the organization's S&MA activities. In addition, the departure of employees with the requisite expertise may pose a risk in relation to the continuity of activities and best practices.

Recommendation

Policy and guidelines

  1. Update the CSA's Safety and Mission Assurance Policy in order to clarify the roles and responsibilities of stakeholders. This would include indicating when S&MA employees should provide matrix support and when they should provide independent oversight of activities.
  2. Develop clear guidelines related to the application of the policy. The guidelines should include the following:
    • The establishment of quality and quality assurance requirements based on specific criteria such as the nature, the complexity and the phase the project;
    • The communication of requirements to contractors doing business with the CSA;
    • The evaluation of contractors' plans and their capacity to meet the CSA's S&MA requirements;
    • The development of an internal methodology for applying and monitoring the requirements and the policy as a whole (e.g. S&MA plan, quality control, process audit, acceptance of deliverables, certification of flight readiness);
    • Follow-up on observations and implementation of action plans;
    • Reports to be produced and accountability for S&MA activities.

Standardized processes and tools

  1. Develop standardized processes and tools for planning, carrying out and documenting S&MA activities.
Responsibility identified
Organization
  • Space Science and Technology Directorate
  • Programs and Integrated Planning Directorate
Function
  • Director General, Space Science and Technology Directorate
  • Director General, Programs and Integrated Planning
Management response

We concur with the audit report recommendations and in fact have been attempting to publish a policy and develop processes and procedures over the last few years, this has been difficult due to the higher priority project support tasks.

Action plan
Details of the action plan

Recommendation 1

The S&MA Policy will be revised, clarifying roles and responsibilities within a new business model that will be applied consistently across the organization.

Deadline: March 31, 2016

Recommendation 2

S&MA guidelines for implementation of the updated policy will be developed to address the issues identified.

Deadline: June 30, 2016

Recommendation 3

Standardized S&MA planning, execution and documentation processes and tools will be developed and implemented in line with the updated policy and guidelines.

Deadline: September 30, 2016

2.3.2 S&MA practices

Audit objective: To determine whether the Canadian Space Agency (CSA) has a management framework in place that enables it to achieve its safety and mission assurance objectives.

Findings
Criterion 2

Effective S&MA practices are applied to CSA projects.

Condition

As already mentioned, we selected and examined a sample of five projects from a total of 19 active projects during the period covered by the audit. The selection was made based on the auditors' professional judgement and was aimed at identifying S&MA practices applied to projects of different types and sizes. The composition of the sample was as follows:

  • 2 highly complex large projects with a substantial budget
  • 2 microsatellite projects
  • 1 project related to the design and operation of aerostatsFootnote 1

Planning of S&MA activities is carried out before the start of projects and updated as needed

To determine whether planning of S&MA activities was carried out before the start of projects and updated as needed, we expected to find a description of project-specific S&MA roles and responsibilities, along with a plan including the requirements, standards, methods and planned resources. In addition, we expected to see that the plan was updated as needed.

We found that planning of S&MA activities was well detailed and documented in the case of the major projects and for the aerostat design and operation project. The quality and quality assurance requirements for these projects were clearly defined and work plans were developed and implemented. However, we noted that for the two microsatellite projects, the S&MA requirements were not clearly defined, and this created difficulties for one project when the time came to recommend acceptance of the deliverables. For the other project, we did not find a single planning document describing the in-house work required or the S&MA roles and responsibilities for the project.

Recommendation No. 2, as set out above, will help to improve communication of S&MA expectations for contractors and clarify expectations related to the internal planning process for S&MA activities.

The S&MA policy and guidelines are applied to projects

For each project selected, we noted that a multitude of working documents existed. Our review enabled us to conclude that, overall, S&MA matters are monitored, issues are identified and adverse consequences and non-conformances are prevented. We also noted that the policy and guidelines were applied in a structured and consistent manner for the major projects and for the aerostat design and operation project that we examined. However, the S&MA plan for one of the microsatellite projects was not implemented and it was not updated to reflect the changes. In fact, the "CSA Project Approval Document" for that project detailed several S&MA tasks that were to be performed on an ongoing basis by S&MA staff. However, owing to a reallocation of resources, the tasks were not performed as planned.

Recommendation No. 2, mentioned above, will help to enhance the in-house methodology used in the planning process for S&MA activities, including the requirements related to updating plans.

Work performed by the S&MA function is properly documented.

As already mentioned, many working documents were generated for each project that was examined. The documents describe the various tests, reviews and follow-ups performed. It is difficult for the internal audit team to determine whether all S&MA work is properly documented, since the result of this work is dispersed among all the documentation that exists for a project. Several advices from S&MA staff can be found among the emails and minutes of meetings. Nonetheless, in our file review, we noted that S&MA work was well documented and contained evidence of the activities undertaken. At the same time, we noted that configuration management was used more sparingly in some projects and to a great extent in others.

In our opinion, the establishment of standardized practices with respect to organizing documentation of S&MA work would clarify the related expectations and ensure consistency from project to project.

Recommendation No. 3 (see above) which is aimed at the development of standard processes and tools also applies to the organization of file documentation.

Documented follow-up is performed for deficiencies identified in S&MA activities.

For each project examined as part of the present audit, when S&MA deficiencies were identified, we found that documented follow-up was performed. Follow-up tools were developed and allow the S&MA function to follow the progress of implementation of the required actions.

Cause

Documentation

There are no specific guidelines covering the organization of documentation on S&MA activities.

Effect

Documentation

It can be difficult to find documentation on S&MA work since the organization of documentation varies from project to project.

Recommendation

N/A

Responsibility identified
Organization

N/A

Function

N/A

Management response

N/A

Management action plan
Details of action plan

N/A

Appendix – Terms of reference

Audit objective: To determine whether the Canadian Space Agency (CSA) has a management framework in place that enables it to achieve its safety and mission assurance objectives.

Audit criteria Audit sub-criteria Legend:
  • Sub-criterion met: 
  • Sub-criterion partially met: 
  • Sub-criterion not met: 
Criterion No. 1:
A management and governance framework for S&MA activities is in place at the CSA
Sub-criterion 1.1: A clear S&MA policy is in place and enables the Agency to meet the requirements of various projects Sub-criterion partially met
See recommendation 1
Sub-criterion 1.2: Guidelines describing S&MA practices at the CSA are in place Sub-criterion partially met
See recommendation 2
Sub-criterion 1.3: Systematic planning of S&MA activities is carried out periodically Sub-criterion met
Sub-criterion 1.4: Processes, management tools and controls are defined to ensure the quality of S&MA work Sub-criterion partially met
See recommendation 3
Sub-criterion 1.5: There is accountability for S&MA activities Sub-criterion partially met
See recommendation 2
Criterion no. 2:
Effective S&MA practices are applied to CSA projects
Sub-criterion 2.1: Planning of S&MA activities is carried out before the start of a project and updated as needed Sub-criterion partially met
See recommendation 2
Sub-criterion 2.2: The S&MA policy and guidelines are applied to all projects covered by the policy Sub-criterion partially met
See recommendation 2
Sub-criterion 2.3: The work done by the S&MA function is properly documented Sub-criterion partially met
See recommendation 3
Sub-criterion 2.4: Documented follow-up is performed on deficiencies identified in S&MA activities Sub-criterion met

Footnotes

Footnote 1

An aerostat is a device that remains aloft through the use of lighter-than-air gases (e.g. weather balloon)

Return to footnote 1 referrer