Internal Audit of the Management of Testing Facilities

Download the PDF version (5.8 MB)

Audit Report

Project #09/10 01-07

prepared by
Audit and Evaluation Directorate

November 2010

Table of Contents

1.0 Summary

1.1 Audit objective

The objective of the audit was to assess whether test facility planning and management processes are such as to fulfil internal and external clients' requirements, and so attain the program's expected objectives and outcomes (re David Florida Laboratory).

1.2 Auditor's opinion

In our opinion, the planning and management processes of the testing facilities makes it possible to meet the requirements of internal and external clients efficiently and effectively. However, this audit has identified some deficiencies with moderate impact that require special attention from management.

1.3 Statement of assurance

As Chief Audit Executive, I am of the opinion that sufficient and appropriate audit procedures were conducted and evidence gathered to support the opinion put forward in this report. This opinion is based on a comparison of existing conditions with audit criteria previously determined and approved by management. The opinion applies solely to the audited organization.

1.4 Summary of recommendations

We found in our audit that the DFL has adopted procedures for managing the quality of its services that take the requirements of clients into account. These procedures are ISO-certified and monitored on a quarterly basis. We found that these procedures ensure that the quality of services is monitored.

We also found that the DFL has an HR plan that includes a strategy for replacing technical employees who are expected to retire in greater numbers in the future.

We also found that the DFL has an HR plan that includes a strategy for replacing technical employees who are expected to retire in greater numbers in the future.

  1. Formalize and document the planning process between DFL and internal testing facilities users.
  2. The DFL should draft a document outlining its overall strategy for use of facilities and draw up an associated implementation plan.
  3. The DFL should review the output indicators and results indicators to ensure that they are relevant and adequately measure DFL performance.

Chief Audit Executive's signature

Audit team member

  • Frédéric Parent, CMA, CIA

2.0 Audit Report

2.1 Background

The mission of the Canadian Space Agency (CSA) is to lead the development and application of space knowledge for the benefit of Canadians and humanity. The David Florida Laboratory (DFL) is an integral part of the CSA, is an essential component of the national space infrastructure, and contributes to the fulfilment of the CSA's mission.

The DFL facilities offer a range of services such as thermal vacuum; thermal balance; vibration, modal, and radio frequency testing; and mass properties measurements, to name but a few. The DFL is one of only a few facilities of its kind in the world capable of simulating both the enormous stresses experienced during a rocket launch and the temperature extremes associated with space flight. The DFL provides governments and the industry with the specialized rooms, equipment and support personnel required for the assembly and qualification of spacecraft and their subsystems and major components.

The DFL's mandate includes the following:

  • Providing environmental and radio frequency testing for space-based and terrestrial programs at the system and subsystem level to a range of domestic and off-shore clients on a fee-for-service basis;
  • Managing the ongoing schedule of test support to maximize the quality and integrity of the test environment, and minimize any risk to people, facilities and the test article's safety;
  • Conducting, in conjunction with the industry, marketing activities to increase opportunities for DFL use; and
  • Ensuring the cost-effective operation of the DFL by establishing usage priorities and procedures consistent with optimum use of the facilities. This includes the provision of test support for non-space-related work to achieve uniform facility loading.

2.2 Audit purpose, scope and method

Purpose of Audit

To assess whether test facility planning and management processes are such as to fulfil internal and external clients' requirements, and so attain the program's expected objectives and outcomes (re David Florida Laboratory).

Scope

The audit project focused on DFL test facility planning and management activities for the period from April 1, 2009, to July 31, 2010. The scope of the audit included the following:

  • Test facility planning procedures, including procedures for identifying clients' requirements and selecting performance indicators and targets;
  • Procedures for managing follow-up activities for test facility planning; and
  • Long-term planning procedures for ensuring the sustainability of DFL activities.

Method

Various audit procedures were used, including staff interviews and reviews and analyses of documents and records.

The audit criteria are established in accordance with relevant best management practices and applicable laws and policies.

2.3 Findings, recommendations and management response

2.3.1 Objective 1

To provide assurance that the test facilities planning processes are such as to effectively and efficiently meet the requirements of internal and external clients, and so attain the program's expected objectives and outcomes.

Criterion 1: Current management practices make it possible to effectively meet the requirements of internal and external users.

Findings Sub-criterion 1 There is a formal procedure for taking (internal and external) user feedback into account.
Condition

Procedures have been developed and implemented to ensure that feedback is taken into account. These procedures provide a framework for management's planning, monitoring and ongoing improvement practices and result from the DFL's ISO 9001:2008 certification.

To maintain this certification, the DFL must undergo a compliance audit twice a year. No non-conformities were reported in the most recent audit.

  1. Planning phase

    External users

    The DFL's planning procedures stipulate that a working group must be set up to determine client requirements and how to fulfil them with existing DFL facilities. Templates have also been developed to provide a framework for key stages in the planning process, particularly in the area of determining requirements.

    Internal users

    The planning process for internal users requires interaction between the Project Management Office (PMO) and DFL. The PMO communicates specification requirements and DFL submits a proposal in order to respond to the identified needs. An agreement is thereafter signed between both CSA's sectors. These procedures are relatively new and undocumented. DFL's Director mentioned to us that documentation is currently being developed to formalize processes and forms to be used.

  2. Monitoring phase

    The DFL has developed monitoring procedures for assessing its capacity to meet client requirements. These procedures are followed in order to review information received from clients and check the effectiveness of the quality control system.

    The DFL monitors with the help of a satisfaction questionnaire that is given to clients. Output indicators are used to measure the percentage of returned questionnaires filled out by clients and the percentage of projects in which requirements were met (satisfaction rate). In the 2009-2010 fiscal year, the client satisfaction rate was 95% with a participation rate of 36%.

    When complaints are entered on the questionnaire, they are recorded in a non-compliance report. An investigation may be undertaken to identify the causes of the problem and a meeting may be held with the client to settle the dispute.

Cause N/A
Effect The absence of a formal planning process between DFL and internal testing facilities users doesn't allow for optimal planning for the use of facilities.
Recommendation 1 Formalize and document the planning process between DFL and internal testing facilities users.
Identified Responsibility Organization David Florida Laboratory
Duties Director, David Florida Laboratory
Management Response I agree with this recommendation. DFL is currently conducting a dialogue with csa pmos regarding the method of documenting DFLs relationship with projects, and have achieved agreement in principle.
Management Action Plan Details of action plan

DFL has proposed to Program Management in Space Utilization and Space Exploration branches to describe its role in a project in a Project Implementation Plan (PIP). DFLs role in such projects is to provide facilities and expertise in Assembly, Integration and Test (AIT). This document will specify that a Space AIT Facility Requirement Specification (SAFRS) is delivered to DFL, in response to which DFL will submit a proposal outlining how and if it can meet the requirements. Should DFL be unable to meet a requirement then CSA management will decide what resources can be applied to resolve the issue, and what impact this will have on the project. These documents will be formally approved by Program Management in both branches for use as living documents, to manage the DFL role in the project throughout its life cycle.

Timetable: March 2011 TBC pending on-going discussions with program management of RCM and ESM programs

Criterion 1: Current management practices make it possible to effectively meet the requirements of internal and external users.

Findings Sub-criterion 2 Human resources and financial planning and existing infrastructure take requirements expressed by users into account.
Condition The DFL's capacity to provide services is determined on the basis of existing infrastructure. The DFL has limited human and financial resources and thus cannot develop or acquire new testing technologies other than those provided with the existing facilities.

However, DFL management has a capital budget that it uses to make temporary modifications to facilities and provide clients with some additional services. For the current fiscal year, this budget amounted to $1.38M. For example, we noticed that temporary facilities had been installed to provide clients with better quality air during the James Webb telescope project.

Because human resources are limited, it is difficult to conduct tests over periods of several days. To solve this problem, DFL management has a budget for hiring consultants temporarily to meet client requirements.

Cause N/A
Effect N/A
Recommendation 1 N/A
Identified Responsibility Organization N/A
Duties N/A
Management Response N/A
Management Action Plan Details of action plan

N/A

Timetable:

Criterion 1: Current management practices make it possible to effectively meet the requirements of internal and external users.

Findings Sub-criterion 3 Planning ensures optimum use of facilities.
  • Promotion of services (marketing)
  • Utilization rate
  • Cost-effectiveness of facilities
Condition The DFL does not have a document outlining an overall strategy for optimizing the use and cost-effectiveness of facilities. Although the DFL has financial indicators, such as the contribution to the Consolidated Revenue Fund, the financial indicators do not make it possible to report on the cost-effectiveness of facilities.
Cause Revenues generated by external clients are deposited in the Consolidated Revenue Fund, whereas the DFL pays the costs relative to implementing the projects of external clients. This situation is hampering the DFL's business development activities.
Effect The lack of a clearly documented strategy for optimizing the use of and making facilities cost-effective increases the risk of the DFL being underutilized.
Recommendation 2 The DFL should draft a document outlining its overall strategy for the use of facilities and develop an associated implementation plan.
Identified Responsibility Organization David Florida Laboratory
Duties Director, David Florida Laboratory
Management Response I agree with this recommendation. During a recent off-site meeting among managers, the future of DFL was discussed and it was agreed that such a plan is necessary. Consequently, a DFL Strategic Plan is currently being written, based upon planning undertaken by DFL management during the past year. This will address upgrades to facilities, equipment, human resources, and interface management (including clients) to meet projected requirements in the 2011-2015 time frame. Key in this plan will be the balance of commercial vs non- commercial business and the level of marketing appropriate to a government service organisation such as DFL, especially in the context of the new mandate of capacity-building
Management Action Plan Details of action plan
Submit for approval to XC a Strategic Plan for DFL as input to the 2011/12 work plan exercise.

Timetable: Feb 2011

Criterion 2: Appropriate performance indicators and targets have been determined in order to achieve objectives.

Findings Sub-criterion 1 Selected indicators are clear, quantifiable and in line with objectives.
Condition

The DFL has output indicators to serve as a framework for its activities and performance indicators to measure outcomes. The DFL's expected outcomes are as follows: development, provision of expertise and supply of space qualification services and of functional and environmental testing of space hardware primarily for CSA sponsored programs and projects, and subsequently for the Canadian space industry and other private- and public-sector clients. We found in our audit that the only indicator used to measure the achievement of outcomes was the percentage of satisfied clients.

The DFL's sub-objectives are also described in the operational plans, as follows: Operate and maintain facilities and develop new testing technologies.

We noted that two of these sub-objectives were not associated with an output indicator, i.e., maintain facilities and develop new testing technologies.

Cause N/A
Effect The lack of matching of objectives with performance targets limits the effectiveness of the performance measurement mechanism.
Recommendation 3 The DFL should review the output and outcome indicators to ensure that they are relevant and adequately measure DFL performance.
Identified Responsibility Organization David Florida Laboratory
Duties Director, David Florida Laboratory
Management Response I agree with this recommendation. Based upon strategic plan discussions, DFL is currently developing new performance measures for the 2011/12 work plan period. This will be an integral part of the Strategic Plan outlined above. Discussions are being held wit corporate Planning to get guidelines and concurrence.
Management Action Plan Details of action plan

Develop new performance measures in concert with CSA corporate planning experts. The key new performance measure will be related to development, maintenance and implementation of the specific elements of the strategic plan.

Timetable: Feb 2011

Criterion 2: Appropriate performance indicators and targets have been determined in order to achieve objectives.

Findings Sub-criterion 2 Selected targets are reasonable and facilitate achievement of objectives.
Condition

Management, with support from the Governance, Planning and Performance team, sets the targets. The Director General exercises control by authorizing targets and indicators. We found in our audit that the DFL targets are systematically achieved. In some cases, the set targets were below the DFL's outcomes achieved in the past.

In addition, many targets do not have clear success criteria. For example, the number of programs to be supported by the DFL is approximately 12. The use of the words "approximately" or "estimated" does not help to clearly report on the achievement of a target.

Cause N/A
Effect The risk arising from a lack of clarity in the targets to be achieved is that reporting may become ambiguous.
Recommendation See Recommendation 3
Identified Responsibility Organization David Florida Laboratory
Duties Director, David Florida Laboratory
Management Response I agree with the recommendation, and propose a response as in Recommendation 3
Management Action Plan Details of action plan
The action plan is as outlined under Recommendation 3

Timetable: Feb 2011

Criterion 3: Management has planned and adopted a strategy to ensure sustainability of DFL activities.

Findings Sub-criterion 1

There is an HR plan and it includes the following:

  • Analysis of current and future human resource and skill requirements;
  • Analysis of senior positions and succession planning;
  • Training and professional development plan.
Condition

The DFL has an HR plan that includes an analysis of the organization's current and future requirements. An analysis of positions requiring succession planning has been done and there are training and professional development plans.

The DFL will have to replace a large portion of its staff in the next few years. This situation was documented in the DFL risk analysis. A workforce renewal framework document has been drafted, but workforce renewal is a lengthy process and subject to CSA budget constraints.

It was found in the audit that the sub-criterion had been met. However, we identified a major risk in terms of the retention of technical knowledge for carrying out DFL tests. The loss of skilled technicians is an operational risk that may affect the DFL's regular operations.

Cause N/A
Effect N/A
Recommendation N/A
Identified Responsibility Organization N/A
Duties N/A
Management Response  
Management Action Plan Details of action plan

 

Timetable:

Criterion 3: Management has planned and adopted a strategy to ensure sustainability of DFL activities.

Findings Sub-criterion 2

There is an infrastructure plan and it includes the following:

  • Analysis of current and future requirements relative to physical facilities;
  • Investment plan to help meet client requirements.
Condition The DFL has begun assessing the sustainability of its activities. A presentation outlining investment scenarios was given to the Executive Committee. The information concerning investments to be made comes solely from the discussion with the managers in charge of DFL testing. No formal market analysis has been carried out to identify client requirements. We did not find any documented strategy in support of long-term DFL investments.
Cause Historically, technology is developed when CSA programs require it.
Effect The lack of a document outlining a long-term vision increases the risk of the DFL not being able to meet client requirements in the future.
Recommendation See Recommendation 2
Identified Responsibility Organization David Florida Laboratory
Duties Director, David Florida Laboratory
Management Response I agree with this recommendation and propose the same response as under Recommendation 2
Management Action Plan Details of action plan
The action plan is as outlined under recommendation 2.

Timetable: Feb 2011

2.3.2 Objective 2

Provide assurances that the testing facility management processes make it possible to meet the requirements of internal and external clients effectively and efficiently so as to achieve the program objectives and expected results.

Criterion 1: Periodic monitoring is done to ensure that objectives are achieved.

Findings Sub-criterion 1 Performance measurement results associated with Operational Plans are recorded and disseminated to the appropriate managers and taken into consideration for decision-making purposes.
Condition Performance indicators are monitored twice a year. The governance, planning and performance team analyses variances and contacts directors to obtain explanations for the variances (positive or negative).
Cause N/A
Effect N/A
Recommendation N/A
Identified Responsibility Organization N/A
Duties N/A
Management Response N/A
Management Action Plan Details of action plan

N/A

Timetable:

Criterion 1: Periodic monitoring is done to ensure that objectives are achieved.

Findings Sub-criterion 2 Management compares results with expectations at regular intervals.
Condition

Senior management does a follow-up mid-way through the year and at the end of the fiscal year.

DFL management must submit a written report every two weeks to the Director General to brief him on the situation and problems.

Cause N/A
Effect N/A
Recommendation N/A
Identified Responsibility Organization N/A
Duties N/A
Management Response N/A
Management Action Plan Details of action plan

N/A

Timetable:

Criterion 2: Corrective measures are taken when adjustments are necessary.

Findings Sub-criterion 1 Active monitoring is done by taking appropriate action at the right time.
Condition No situation requiring the implementation of corrective measures was identified.
Cause N/A
Effect N/A
Recommendation N/A
Identified Responsibility Organization N/A
Duties N/A
Management Response N/A
Management Action Plan Details of action plan

N/A

Timetable: