Agence spatiale canadienne
Symbol of the Government of Canada

Table of Contents

Audit report

Project Management Processes and Practices
PROJECT #06/07 01-03

Prepared by the
Audit and Evaluation Directorate

October 2007

Top of page

Executive Summary

The goal of this audit project was to evaluate the extent to which Canadian Space Agency (CSA) project management processes and practices (Phases 0 to E, inclusively) enable it to make informed decisions on project and initiative funding; appropriately track projects; implement approved initiatives in an effective, efficient and economical manner; attain the anticipated outcomes established in key planning documents; comply with the applicable policies, regulations and guidelines set by the Agency and by central agencies; and report on how it uses resources.

We used an outcome-based approach, one designed to determine whether the Agency exercises due diligence in how it selects projects and initiatives for funding. We wanted to evaluate the extent to which the project management process (Phases 0 to E) allows the Agency to make appropriate and informed choices, use reliable and available information in a timely manner, and optimize its use of public funds.

Our study revealed that problems in cost-benefit performance, inabilities to meet established schedules and rising costs are features inherent to projects managed by the Agency.

Space Program activities are characterized by complex, high-risk projects. We are well aware that it would be unreasonable to assume that these characteristics could be eliminated. However, by applying the appropriate project management methods (Phases 0 to E), it should be possible to minimize their impacts. Key aspects of project management have been described in CSA and Treasury Board policies and manuals.

CSA has developed good project and risk management frameworks, but does not make judicious use of them in its day-to-day management practices. The problem begins with not very strategic decisions that fund too many projects. With limited funding to meet its obligations, CSA keeps project cost estimates too low and underestimates the risk reserves required for the identified risks and for its use of immature technologies.

Project costs at CSA increase as tasks are deferred to later phases, tasks important to project advancement are cancelled, and the risks associated with partnerships, technologies and transferring projects to successive phases with immature technologies are underestimated. These problems are compounded by a lack of systems for allocating actual direct labour costs and a tendency to undertake development activities in the operations phase (Phase E).

CSA must introduce important changes to correct its project management processes and practices (Phases 0 to E) rather than the management frameworks it has developed.

CSA must review its governance processes, including performance measurement systems and how it reports expenditures.

CSA must perform more thorough analyses and compare budgeted amounts with actual costs, taking into account comprehensive project life-cycle costs.

CSA must learn from past experience. Above all, it must disseminate this knowledge and apply it in day-to-day operations.

This internal audit was performed in accordance with the Treasury Board Secretariat's Policy on Internal Audit and Standards for the Professional Practice of Internal Auditing, published by the Institute of Internal Auditors (IIA). In our professional judgment, sufficient and appropriate audit procedures have been conducted and evidence gathered to support the accuracy of the conclusions reached in this report. Our conclusions were based on a comparison of the situations as they existed at the time against the audit criteria.

Since the audit was completed, changes have been made to CSA's decision-making and management structure that represent a partial response to the findings of this audit report.

Top of page

Description of the Mandate

1.0 Introduction

1.1 Rationale for the Audit

This audit was launched in response to a request from the Acting President of the Canadian Space Agency (CSA) to examine and propose improvements to the management and financial planning processes used in CSA project management. Project management was to be understood in the broadest sense of the term, i.e., from Phase 0 to Phase E, inclusive. This audit was not included in the 2006-2007 audit plan approved by the Audit Committee.

Top of page

1.2 Purpose of the Audit

The purpose of this audit was to determine if CSA's project and initiative management processes and practices allow it to:

  • make informed decisions on the funding of projects and initiatives;
  • appropriately monitor its financial resources following sound management practices;
  • implement approved initiatives in an effective, efficient and cost-effective manner;
  • attain the anticipated outcomes set forth in key planning documents;
  • comply with the applicable policies, regulations and guidelines set by the Agency and by central agencies; and
  • account for how CSA resources are used.

Appendix A provides a more detailed description of the audit objectives and criteria.

Top of page

1.3 Scope

The audit project covered systems and procedures currently used at the Agency. These systems and procedures facilitate:

  • operations planning, including the identification, selection, approval, justification and allocation of resources for a range of program elements;
  • execution;
  • operations monitoring and performance assessments; and
  • accountability.

As described in Section 2, the audit covered various aspects of project management taken in the broadest sense of the term, from the selection of projects for funding to operations. We also compared the Agency's practices and processes with best practices.

Top of page

1.4 Methodology

This audit engagement was carried out in accordance with audit standards set forth in the Treasury Board Secretariat's Policy on Internal Audit as well as Standards for the Professional Practice of Internal Auditing, published by the Institute of Internal Auditors. These standards require that audit objectives be set on the basis of audit criteria.

Audit standards also require that the audit engagement be conducted in a methodical manner according to a process that includes:

  • a planning and preliminary review phase,
  • an execution phase, and
  • the reporting and disclosure of audit results.

Various audit procedures were used, including interviewing employees and reviewing and analysing documents, records and reports.

Top of page

2.0 Background Information on Project Management Processes and Practices

Space projects funded by the Canadian Space Agency carry a high level of risk because they are often one-of-a-kind and employ cutting-edge technologies. Since 1999, CSA has managed an annual budget of approximately $300 million.

In 2000, CSA was delegated financial authority for project funding in amounts up to $5 million on the condition that it develop and implement a Project Approval and Management Framework (PAMF) and a Risk Management Framework. In 2003, CSA introduced changes to its PAMF in order to better address situations faced in project management.

As project leader, the Chief Financial Officer is responsible for applying the PAMF. The project manager and the team responsible for applying the PAMF report to the Director General of Space Programs, who is responsible for managing space systems projects through Phases B, C and D.

In 2003, CSA developed a Canadian Space Strategy following a recommendation made by the Auditor General of Canada. CSA then developed three strategies, approved by the Executive Committee (EC), in the following program areas: space science and exploration, satellite communications and earth observation.

The distribution of expenditures described in the 2007-2008 Report on Plans and Priorities (RPP) is: 35% for earth observation, 42% for space science and exploration and 8% for satellite communications. According to the 2006-2007 Performance Report (DPR), actual expenditures were distributed as follows: 38.6% for earth observation, 48.4% for space science and exploration, and 11% for satellite communications.

In the past, CSA has funded projects on a one-by-one basis, since most were major Crown projects (MCPs). Now its projects are smaller in scope, both in terms of their timetables and funding. Most project budgets are under $100 million. There are three MCPs still in development or operation: the Canadian Space Station Program (CSSP), Radarsat-1 and Radarsat-2. More recently, the JWST project has become an MCP, and the cost of CHINOOK, which was under study at the time of our audit, has been estimated at over $100 million. These two projects carry many significant technological and scheduling risks. JWST is being developed in partnership with NASA and ESA, while CHINOOK is an entirely Canadian project that would be operated by CSA. The project's scope and funding requirements are still under study.

In 2006-2007, CSA began developing its Long-Term Capital Plan (LTCP), which includes capital asset projects.

Top of page

2.1 CSA's Environment1

CSA runs several of its projects as national or international co-operation efforts. In several cases, it can influence but cannot control a project's technical requirements, schedules or other parameters. Several of these projects are governed by letters of agreement (LOAs) and/or memoranda of understanding (MOUs) between space agencies or governments.

Decisions on participation in these co-operation projects and approvals to proceed with each phase are influenced not only by how project performance compares with objectives, but also by national and international commitments and decisions concerning the Canadian Space Program (CSP).

1Based on the Project Management Policy, 2003.

Top of page

2.2 Authorities Involved in the Decision-Making Process

In the process of obtaining project approvals, the Core Function Control Panel (CFCP), the Program Review Advisory Board (PRAB) and the Executive Committee (EC) each may play a role depending on the phase for which approval is being sought, the estimated project cost and whether matrix management is required to support its implementation. The roles and responsibilities related to obtaining the approvals and signatures required to proceed are as follows:

  • Core Function Control Panel (CFCP)
    The panel is made up of the directors general of the core functions. Until 2004, members took turns as chair for one-year periods upon approval of the Executive Committee. Since 2004, the Panel has been chaired by the Vice-President of Science, Technologies and Programs. The panel was dissolved in May 2007.

    The CFCP's main objective was to facilitate a successful implementation of the Matrix Management Policy in an environment characterized by multiple projects and, consequently, to ensure that the Agency was making optimal use of its resources for the benefit of all the functions and meeting the needs of all the Agency's projects.

  • Program Review Advisory Board (PRAB)
    The Board is the highest authority at CSA on program development and the program management process. It is responsible for managing, monitoring and controlling CSA program content and priorities.

    The Board is composed of CSA's president; vice-presidents; the directors general and directors of the Canadian Astronaut Office and Policy, Planning and Relations; the Chief Financial Officer; and the co-ordinators of the Agency's core thrusts.

    The Board was dissolved in May 2007.

  • Executive Committee (EC)
    The Executive Committee is the Agency's official decision-making body and approves projects with cost estimates under $5 million. The Committee is responsible for delegating spending authority within the Agency and carries responsibility for overall management, monitoring and control of CSA activities.

    The Committee is composed of CSA's president; vice-presidents; the directors general and the directors of the Canadian Astronaut Office; Policy, Planning and Exterior Relations; Finance; Human Resources; Communications and Public Affairs; Security and Facilities; and the directors general of Space Technologies, Space Science, Space Operations and Space Programs; Audit, Evaluation and Review; and Legal Services.

    Changes were made to the committee in 2007.

  • Treasury Board (TB)
    The Treasury Board approves projects with cost estimates that exceed $5 million, the approval level accorded to CSA, as well as all major changes to CSA projects. The Board reviews projects to ensure that they represent effective and efficient responses to the operational needs as set out in CSA's defined priorities and in the government's long-term investment plan.

Top of page

2.3 Project Authorization Process

2.3.1 Agency Approvals

CSA has signing authority for its projects when the cost of Phases B, C and D, including risks and GST, do not exceed $5 million.

When the project value exceeds $500,000, a project brief must be drafted at the end of Phase A in order to obtain approvals from the PRAB, CFCP and Executive Committee before it can be transferred to Phase B. A project brief is also required if the costs for Phases 0/A exceed $500,000. Before approvals are requested, the project brief must be approved by the Senior Financial Officer and the appropriate service line authority.

Annual work plans are used to authorize budgets for Phases 0/A of projects supervised and funded by the Space Technologies and Space Science branches.

According to the PAMF, a project approval request is filed at each phase to obtain the required approvals. The PRAB authorizes projects with estimates between $500,000 and $5 million as long as the project is included in a work plan. If the project has not been mentioned in the work plan or the project cost estimate has increased, it must be referred to the EC for authorization.

Top of page

2.3.2 TB Approvals

For projects with cost estimates for Phases B, C and D in excess of $5 million, a project brief must be prepared at the end of Phase A for TB approval before transfer to Phase B. The project brief will be included in the submission to Treasury Board once the EC has recommended approval.

The preliminary project approval (PPA) is TB's authorization to proceed with the project definition phase (Phases B and C for space projects and Phase II for non-space projects). Occasionally, effective project approval can be obtained for Phases B, C and D if CSA is relatively certain of its cost estimates for these phases.

Effective project approval (EPA) provides authorization to implement the project (generally Phase D for space projects and Phase III for non-space projects). CSA determines project objectives (project reference), including the cost objective for the project execution phase, and TB authorizes the funding so that CSA can execute the project. CSA submits an EPA once the content of the entire project has been defined and assessments have become highly detailed (high quality).

After a project approval, CSA becomes accountable to TB for meeting the objectives and all other directives set out in the approval letter.

Depending on the project, EC and TB approve individual project phases (either Phases B and C or Phases B, C and D) in accordance with the degree of certainty attained in project cost estimates.

Top of page

2.4 Definition of a Project1

A project consists of Phases A, B, C and D, i.e., the planning, preliminary design, detailed design and implementation phases, respectively. In addition, all projects are managed within the larger framework of a life-cycle approach to goods and services management. A project therefore also consists of the activities of Phases E and F, i.e., implementation and termination.1

1 From the CSA Project Approval and Management Policy, 2005.

The following graph shows the stages in a project life cycle.

  • 1 Phase 0: Opportunity evaluation and advanced studies. This phase is used to develop project concepts and ideas. Phase 0 is not considered part of the project, but it may go through the same stages as subsequent phases. These activities are mentioned in CSA's annual work plan.
  • Phase A: Concept development and feasibility. Mission conditions and design studies are developed. This phase includes arriving at a decision on the project's technical and economic feasibility and defining its scientific requirements. The phase ends with a system requirement review (SRR).
  • Phase B: Preliminary design. This phase establishes the preliminary concept and design according to the technical requirements. It generally ends with a preliminary design review (PDR).
  • Phase C: Detailed design. This phase is used to finalize the design, and concludes with a critical design review (CDR).
  • Phase D: Manufacturing and acceptance. In the manufacturing phase, the design is transformed into equipment and software. An acceptance review (AR) is conducted in this phase. Other reviews may also be required. This phase usually includes the launch, preliminary operations and project close-out activities.
  • Phase E: Operations. Activities in this phase include system operation and maintenance, and may include project warranty and development activities that have been formally transferred to operations and maintenance officers. In some cases, this phase may include launch and acceptance in orbit, if these stages are not executed as part of
    Phase D.
  • Phase F: Termination. Disposal of a system at the end of its useful life, ground segment and space segment. This phase implies product or system disposal, but not project close-out. The disposal may be a project in itself if the work is complex and a phase-out review (RD) is conducted.

1 From the PAMF Project Management Guide, 2005

Top of page

2.5 Scope of Project Management

Sound practices require that the project management process include the following:

  • Project integration
  • Project content
  • Project timelines
  • Project costs
  • Project quality
  • Human resources committed to the project
  • Project communications
  • Project risks
  • Project procurement

Top of page

2.6 Project Process

Projects and initiatives begin in the Space Technologies and Space Science branches, which manage a large portion of CSA's budget. Their mandate is to undertake projects and manage them through Phases 0/A. Cost projections are made at the end of Phase A, when the projects are handed over to the Space Programs Branch for Phases B, C and D. At the end of Phase D, the project is transferred to the Space Operations Branch when it requires CSA involvement in operations. In this case, CSA will play a larger role when the project is entirely Canadian.

For projects with a scientific component, the work of developing and implementing a project selection strategy begins long before the project reaches Phases B, C and D. The Space Science Branch conducts a series of studies and consults with the scientific community on several occasions to enquire about the various areas of interest that could lead to important scientific projects. Such projects may or may not be carried out in partnership with other international agencies. Several scientific committees examine these issues.

The Space Technologies Branch has developed a long-term technological plan and conducts a series of studies before these technologies take the form of actual projects to be conducted in international partnerships or independently by CSA. The Branch also uses opportunity calls to generate ideas on new technologies. The two branches then analyse the proposals and select subjects for research.

Some of CSA's project funding choices are the result of other space agencies selecting Canadian contractors. As a result, CSA feels obliged to support the projects.

The following structure describes how CSA human resources are assigned to the management of specific projects.

All CSA projects must be carried on according to the PAMF. As the PAMF project leader, the Chief Financial Officer is responsible for its application.

Top of page

3.0 Audit Results

3.1 Introduction

Decisions on project and initiative funding have an impact on the Agency's effectiveness in achieving its mandate and implementing its space strategy.

The quantity and types of projects and initiatives funded have a direct impact on the Agency's capacity to fulfill its mandate in the space industry.

The Agency is responsible for ensuring that projects and initiatives reflect user needs, support its strategy and provide benefits for Canadians. It must ensure that these projects and initiatives are managed following sound management practices and comply with the policies of various central agencies in addition to its own policies.

The Agency delegates project monitoring activities to a project management team that informs the Agency's senior management of any major problems that could pose risks to the successful completion of a project or initiative.

In order to identify the underlying causes of rising project costs at CSA, we reviewed all the project stages, from approval to performance.

The following have direct or indirect effects on project cost increases: the time required for decision-making, funding authorizations, project planning, funding levels, human resources assignments, change management, the accuracy of cost estimates and their underlying assumptions, the maturity of the technology used, project progress and development tracking, risk management and project performance monitoring methods.

Top of page

3.2 Governance – Decision-Making Process

We expected that we would find a strategic plan and an implementation plan to help CSA fulfill its mandate. We expected that there would be a solid and effective decision-making structure in place for selecting projects and initiatives so that the Agency can appropriately fulfill its mandate under its strategy.

In 2003, the Agency developed a comprehensive strategy and identified core thrusts. It approved a technology plan, developed an integrated planning system (IPS) that shows approved funding per activity and implemented a 10-year plan that reflects how funds will be allocated to projects and annual work plans. The Agency also needs a strategic plan that would identify where it should be 10 years from now and an implementation plan highlighting the activities that will allow it to reach the objectives set forth in the strategic plan. CSA began a governance review in 2005.

According to the Project Management Policy, the EC's role is not to approve project phases, but rather to formally authorize the execution of a project, including its scope, costs, schedule and technological and operational constraints. We found that although the EC is responsible for overall management, monitoring and control of CSA activities, it tends to focus on project phases rather than on project selection strategies.

In our interviews, we found that there is a consensus that the EC should authorize a strategic plan that would guide project selection and allow its leaders to implement the space strategy. As it stands, projects are approved phase by phase, without necessarily taking into account the strategic direction or priorities that have been defined and communicated to all CSA managers. Rather than approving project phases, CSA should approve a strategic plan and implement an operational plan in which projects serve as a means to achieving the Agency's mandate. Our analyses have shown that the branches operate independently and compete with one another, rather than aligning their efforts on the basis of a defined, comprehensive strategic objective that is understood and accepted by all CSA branches.

The Agency has neither a plan for implementing its strategy nor priorities associated with its various projects. It selects project phases for funding one at a time, without placing these decisions in a broader strategic context. It does not determine how funding a specific project will contribute to reaching its 10-year positioning objective. Under this approach, CSA is unable to make a strategic selection of the projects to fund in order to reach a long-term positioning objective in the space industry.

In our review of the 10-year plan, we found it difficult to determine CSA's degree of financial flexibility. We also found that the financial information in the IPS and the 10-year plan are based on cost estimates submitted in project approval documents (PADs). Our review revealed that these estimates do not represent the actual project schedules, costs per phase or project life-cycle costs (which include operating costs).

The lack of precise estimates of project life-cycle costs results in the CSA receiving flawed information on its financial obligations. As a result, the Agency does not have an accurate measure of its financial commitments in all its projects from Phases A to E, including operating costs.

Some of the Agency's projects have been funded because another space agency selected the beneficiary (a company or a university) to carry out a portion of one of its projects following a competitive bidding process, and not because they figured in work plans or supported the Agency's strategy. CSA's senior management told us that they have no choice but to approve funding in these cases. They represent unsolicited proposals from other space agencies, and we estimate that there have been five such cases since 2003, for an estimated total of $135 million in CSA funding over several years.

The 10-year plan developed by CSA reflects its allocation of funds to projects and annual work plans. The plan does not clearly show that adequate funds are available for upcoming activities. The Policy, Planning and Exterior Relations Branch, which must implement the strategy, has very limited involvement in project selection and the redistribution of project funding to projects that arise unexpectedly.

The Agency must have a clear series of objectives and a strategic plan that takes into account its capacities, the requirements of its operational environment and the needs and priorities of stakeholders. The Agency must formally define its short-term and long-term priorities and develop a strategic and implementation plan. The strategic direction and priorities must be shared with everyone responsible for initiating new projects. Given CSA's limited financial resources, the Agency must initiate an effective process for establishing its priorities in order to be able to efficiently carry out activities and make informed choices.

The EC should conduct a review of the operations of its various committees involved in how project phases are selected and approved and implement the governance review begun in 2005. The EC should approve program objectives rather than individual projects. The PRAB should exercise more vigilance in its program review role.

Recommendations

President

1) Implement the governance review begun in 2005 and review the method used to approve projects.

2) Set clear objectives, develop a strategic and implementation plan and share these objectives and the strategic and implementation plan with CSA staff.

Top of page

3.3 Obtaining Financial Approbal from TB

We expected that the Agency would apply clear procedures for obtaining financial approval from TB, that the Agency's staff, regardless of reporting level, would be aware of the procedures, and that they would be complying with procedures. We expected that controls would be in place to verify the application of procedures and that measures were being taken to correct procedural errors.

Before commencing projects and project definition phases (B and C), CSA must obtain the appropriate TB financial approvals when project costs exceed its financial authority ($5 million). This is a time-consuming process. The following diagram shows the required process under project management policies and the PAMF.

Decision Tree for Compliance with TB Policies


Source: TB Web site, Supplementary Guidance - Major Crown Projects

Projects that could be designated MCPs must be discussed as soon as possible, since MCPs are subject to a special management regime that carries more benefits when it is implemented early in the planning process. This includes determining whether it is necessary to obtain approval-in-principle from Cabinet.

Decision Tree for Compliance with TB Policies on MCPS


Source: TB Web site, Supplementary Guidance - Major Crown Projects

All of this requires time and good planning. This crucial stage in the approval of project phases has an impact on costs, since the longer CSA waits, the more it has to postpone the conclusion of the current phase so that project activities can be maintained until the required authorities are obtained to proceed with the next phase. Some project managers admitted that they did not know the internal mechanisms required for obtaining TB spending approval. In addition, some project leaders and managers told us that they were not acquainted with TB policies on project management and MCPs.

Our analysis of the approval documents and our interviews demonstrated that the Agency could have obtained more rapid TB financial approval for some projects. Cost assessments for some projects fell below the $5 million threshold, so the projects could have been approved internally. For other projects in excess of $100 million, cost assessments below this threshold were based on unrealistic estimates, so CSA did not comply with MCP management rules.

In May 2006, a submission was made to the EC concerning a process review used at CSA for submissions to Treasury Board. The analysis highlighted a need to upgrade submission preparation skills and meet time constraints. The following recommendations were made: standardize the process, clarify responsibilities, consult as required throughout the submission development and preparation period, upgrade CSA's knowledge and skills in drafting submissions to Treasury Board and streamline the process. The Corporate Secretariat was to fully document the recommended standard process through an advisory working group. At the time of this writing, the Secretariat had not begun this work, and the Executive Committee had not adequately followed up on this request. Following our requests for more information on the subject, the Government Liaison Office reclaimed responsibility for this mandate.

Poor knowledge of these processes and policies has had a significant impact on project management, which has taken more time and therefore been more costly. Either the industry is obliged to stop project-related activities while awaiting approval, or CSA decides to extend the current phase to maintain the activities, which is inefficient and drives up project costs.

The following example demonstrates what can result when the processes and rules governing the project approval process are misunderstood. In the SCISAT project, the Agency funded Phases C and D without obtaining TB approval. It based its decision on approvals obtained in 1995 for a program to develop small scientific satellites for which R&D studies and activities were to have led to the development of two satellites. This program underwent significant changes in the years that followed. In 1999, the year that CSA secured a base budget, its signing authority for projects was $1 million. CSA did not make submissions to TB for additional funding for the SCISAT project for Phases C and D, so it did not comply with central agency policies and overstepped its funding authority on this project.

The SCISAT project began in 1998, and project costs grew as follows:

  • $34.0 million in 1999/2000
  • $34.5 million in 2000/2001
  • $50.3 million in 2001/2002
  • $62.1 million in 2002/2003

The CDR for this project (a review conducted at the end of Phase C) was performed in June 2001.

Recommendations

Branches: Space Technologies, Space Science, Space Programs, Operations and Assets, Directorate, Safety and Program Assurance

3) Ensure that all employees are acquainted and are complying with the processes used to obtain TB financial approval and the PAMF.

Branches: Space Technologies, Space Science, Space Programs, Operations and Assets

4) Ensure that CSA obtains TB approval earlier in its project planning and that no project is exempt.

Top of page

3.4 Transparent, Clear and Complete Information

We were expecting that project funding decisions would be based on clear, transparent, complete and timely information.

All good decision-making depends on obtaining clear, transparent and reliable information. Failure to satisfy one of these criteria will distort the conclusions reached in decisions made by CSA's Executive Committee.

Our analysis revealed that the Planning and Financial Analysis division still does not have all the information it requires to fulfill its mandate, and that the information required to complete and support TB submissions is not received in a timely manner.

We reviewed the information submitted to the Executive Committee for decisions on certain projects, and found that it did not always provide the key elements required for decision-making. Some important points were overlooked in submissions to the Committee.

The following two examples provide good illustrations of this situation.

Top of page

3.4.1 HI-FI project

This was the case in the HI-FI project, where approval was sought for $104,000 in funding to cover the occurrence of a risk. According to information collected in this audit, the funding was not in fact to cover a risk, but rather represented the additional cost of work already performed by the contractor.

In its submission to the EC, managers stated that if the Committee did not approve the funding, CSA would not be able to meet its contractual obligations. We learned of an e-mail that shows that the Agency was under no such obligation. This information was not shared with the EC, even though it was known to certain individuals managing the project.

We believe that one of the causes of this lack of transparency is the fact that some employees are uncomfortable disagreeing with senior management. This situation does not reflect the values and ethics of the federal public service.

Top of page

3.4.2 Orbital project

In reviewing information on the ORBITAL project, we found it difficult to identify all the information required to establish total project funding and funding sources. After reviewing the supporting documents, we found that the project's total funding was not indicated. CSA was unable to identify the sources of funding for this project, and this information was not available in the documents or in the submission to the EC. Incomplete information interferes with informed decision-making and, in this case, CSA has not acted effectively and efficiently.

CSA must ensure that it makes well-informed decisions based on relevant, clear, transparent and complete information. Members of the Executive Committee must fulfill their roles and responsibilities appropriately. To this end, they must be able to access any information they consider necessary.

Our review has demonstrated that the PAD does not highlight all the information required for decision-making. Managers who make submissions to the EC for approvals do not always highlight key information. This makes it difficult to evaluate different projects on the same basis and make informed decisions.

Recommendations

Chief Financial Officer

5) Standardize and submit important information to the EC.

Branches: Space Technologies, Space Science, Space Programs, and Operations and Assets

6) Provide all the information required for timely decision-making and ensure that key project elements can be clearly understood.

Top of page

3.5 Project Approval and Management Framework (PAMF)

In 2000, CSA was delegated financial authority up to $5 million for projects, conditional on the implementation of a project management framework. CSA therefore developed the PAMF and assigned its management to a team of project managers reporting to the Space Programs Branch. The Chief Financial Officer was responsible for the PAMF as team leader during its development and application.

The PAMF covers all aspects of project management, including risk. It complies with central agency policies and reflects sound project management practices. The Framework's goals include:

  • Ensuring that the projects proposed for approval are subject to an informed and effective review, and
  • Attaining effective and cost-effective project management with clearly identified project managers.

CSA has also developed a Risk Management Framework.

We expected that the Agency would follow the principles stated in the two management frameworks it had developed, and that its managers would apply these principles in their work in all stages of their projects. We expected that the Agency would have the means to control how the PAMF is applied.

The project management team charged with applying the framework also has to manage projects. This dual responsibility (applying the PAMF and managing projects) clearly places members in a situation where they lack objectivity and independence. Their peers perceive them as not being impartial enough to properly apply the PAMF, given that they also manage projects, and the team does not have the authority needed to apply the Framework to all CSA projects. Task separation is a basic principle underlying the proper control of how directives and policies are applied.

Some individuals believe that this management team should verify budget estimates, but team members say that this does not fall within their responsibilities. The roles and responsibilities of the team charged with applying the PAMF do not appear to be clear and understood by all.

In order for this team to remain objective and impartial, it must be free of any influence exercised by those in charge of project management. It should therefore report to the Chief Financial Officer (CFO), to whom it can provide the expertise required to perform its functions. This would ensure the team's impartiality and its experts' support in applying the PAMF. In addition, the CFO would be able to appropriately fulfill the following responsibilities using evidence on costs, risks and schedules:

  • Attesting to compliance with all aspects of the Financial Administration Act and related regulations;
  • Attesting to the availability of funds under the Agency's multi-year risk assessment and commitment control plan and the effect on the plan.

Recommendations

President

7) Relocate the team responsible for applying the PAMF to ensure that it has the independence and impartiality needed to exercise proper oversight, and provide the team with the expertise required to fulfill its role certifying compliance with legislation, policies and the PAMF.

Senior Vice-President

8) Clarify the roles of the team responsible for applying the PAMF.

Top of page

3.6 Elements Influencing Project Costs

Several elements can influence project costs: planning, changes made during any project phase, risk identification and management, budget estimates, the maturity of the technology, project tracking, technology performance and outcomes for Canadians.

We reviewed each of these elements, analysed their effects on project costs and made recommendations on how to minimize the impact of cost increases on current or future projects.

Top of page

3.6.1 Project Planning Phase

Space projects involve new technologies and by their very nature take extended periods of time to complete. Classification as an MCP will make a project run even longer. If the projects are carried out in partnership, changes made by the partners will also have an impact on technological requirements and schedules. Many unknowns may arise, and the Agency cannot exercise much control over this element.

To counter these elements and minimize the impact that they can have as occurrences of risk, the Agency must pay particular attention to project planning by defining project scope, the required resources and the impacts for Canadians.

We expected that the project planning phase would receive all the attention it requires, that sound project management principles would be applied. We expected that a precise determination of the human and financial resources would be required for all projects.

We found that CSA does not apply project management principles efficiently. From the outset, there is a significant lack of project management, most importantly in terms of planning the required human and financial resources and giving due consideration to the maturity of the technology.

Managers are working under time constraints and are pressed to meet deadlines for obtaining TB approval, and this has an impact on the quality of project planning. The need to quickly undertake different project phases to keep the industry and project scientists actively involved results in inaccurate planning of essential project elements. This lack of precision can be explained in part by the fact that projects are managed without the technological and science resources that should be involved from the outset. CSA has many projects that are inadequately staffed and funded at the time of implementation.

The Space Technologies Branch is participating in 24 different missions. It estimates that it needs 4 people to manage each mission it funds, for a total of 96 persons, but it only has a staff of 80 to 90 people.

The Agency has many projects in progress, and it is limited in the number of human resources that can provide the required technical and scientific support. This situation affects its capacity to manage projects efficiently and effectively. In addition, it has inherited financial obligations, and its commitments under some of these obligations have already begun. This limits the Agency's financial capacity to undertake new initiatives. Despite this situation, the Agency is undertaking new projects. The Agency has never stopped a project, despite the withdrawal of various international partners. Our analysis of CSA's financial capacity shows that it is unable to fully support projects throughout their entire life cycles. The ORBITAL and SWIFT–CHINOOK projects are good examples. In the former, funding sources have not been identified, and funding needs for the latter outstrip what is available through CSA.

This approach limits the Agency's ability to fund new initiatives and may compromise the implementation of all the scientific activities currently planned under the CSP. This may create an imbalance between the Agency's obligations and its funding capacity, a situation that may well deteriorate over the next few years. In her 2002 report, the Auditor General of Canada had already notified CSA's senior management of this problem.

In order to avoid budget overruns, the Agency must cancel or postpone activities currently planned under the CSP. The current imbalance between its budget resources and its obligations under the CSP could even deteriorate in the medium term. It is therefore clear that the Agency must address this imbalance, either by obtaining an increase in CSA funding levels or by setting priorities so that it can scale back current activities under the CSP.

Ninety-five percent of the people we met pointed out that the Agency could be more efficient if it spent more time planning project resources and objectives. The number of projects currently under way cannot possibly be managed according to sound management practices.

Project managers should spend more time defining the deliverables of each project phase. For example, the deliverables in Phase A are the operational concept, the system's technical requirements, a project plan and a high-level plan for system audits. Once deliverables have been well defined, the project's required resources can be estimated (in particular, the required internal resources). Effective planning of human resources for effective operational management will allow project leaders to make optimal decisions in a timely manner.

Some employees suggested that the Agency should prioritize projects in order to concentrate its efforts and resources on Priority 1 projects, then on Priority 2 projects, etc. This would require the development of precise classification criteria. Such priorities would allow the Agency, which has limited funds, to meet its obligations more appropriately and prudently.

Based on our analyses and discussions, the Agency undertakes initiatives-on its own or in partnership—without having fully analysed all their impact on required and available resources. This runs counter to sound management practices. This situation has been discussed in the past, and the solutions that have been implemented do not appear to have worked.

Recommendations

Branches: Space Technologies, Space Science, Space Programs, and Operations and Assets

9) Spend more time preparing and planning projects and make better assessments of the financial and human resources required to carry out projects and attain outcomes effectively and efficiently.

President

10) Establish priorities and reduce the number of active projects in order to ensure that projects are adequately funded, attain approved objectives and are managed efficiently and economically.

Top of page

3.6.2 Changes in Project Scope

The PAMF includes a section on changes that describes the systems and information required for responsible management of the kind of changes that are inevitably required in the course of a project. We expected that the Agency would apply sound management principles to monitor these changes.

In our review of selected projects, we found that the changes made to the initial plans resulted in significant project delays and increased costs. Different managers were found to have different approaches to risk and project management, and the principles of project management were not being applied uniformly across the Agency. In our interviews, some managers mentioned that they rarely consulted the PAMF or central agency project management policies.

Change management principles are not being applied uniformly. When the Agency makes changes to its projects, it is very difficult to track these changes and know their actual impacts on the project, on technology development and on risks. We had considerable difficulty following the changes made to the scope of tasks in the JWST project or determining what impact these changes would have on the project's benefits to Canadians. When changes are made to project requirements and scope, little documentation is produced from impact analyses (if they are made at all). Impact analyses would show that CSA ensures that changes will not affect the project's scientific and technological benefits and its outcomes for Canadians. No cost-benefit or cost-effectiveness analyses are performed to confirm whether production of the scientific or technological tool will still be efficient, given an increased project cost or reduced project scope. Such cost-benefit analyses are also overlooked in the initial decision-making. We found no documents confirming that such analyses are made. Several managers confirmed that cost-benefit analyses are not made, stating that they consider building a payload and having access to the scientific data better than funding the purchase of this data through a grant program.

Recommendations

Director, Safety and Program Assurance

11) Standardize basic project management principles; ensure that project management and PAMF policies are known to all project managers, directors and directors general and to senior management. Ensure that these principles and policies are uniformly and rigorously applied by all managers.

Branches: Space Technologies, Space Science, Space Programs, and Operations and Assets

12) Perform a cost-benefit analysis at the start of each project and when the scope of a project is changed or project costs increase.

13) Review the new cost estimate in light of the benefits identified at the beginning of the project, and ensure that the project still represents CSA's best means to obtain the desired outcomes for Canadians.

Top of page

3.6.3 Changes in Cost Estimates

We expected that project managers would be able to question the costs provided by industry and that CSA would make decisions based on realistic cost projections for the complete life cycles of its projects. We expected to find databases of historical project costs that CSA would have maintained in order to profit from past experience. In 2003, CSA set up a team to develop tools and methods that would foster an ability to make better cost projections. We expected that the team would have a clear mandate, well-defined responsibilities and an appropriate budget, and that it would provide project managers with support.

This Cost Estimating Team (CET), consisting of representatives of the Space Science, Space Technologies, Space Programs and Space Operations branches as well as the Finance Directorate, has short-, medium- and long-term objectives.

The CET's short-term objective is to perform an independent and detailed review of the estimated costs of selected space projects and create and maintain a database with the preliminary information. The database will provide information on costs for previous, current and future programs and provide access to costing methods and cost estimation data for complete project life cycles.

The team's medium- and long-term objective is to ensure that CSA masters the techniques and system used to estimate project life-cycle costs so that effective cost assessments can be made of proposals submitted to senior CSA managers and the government (PWGSC). The CET has also been formed to reduce the costs associated with preparing and evaluating proposals and related negotiations as well as the length of the contracting cycle.

No budget was allocated to the CET for it to acquire appropriate tools for carrying out its mandate. As at this writing, the committee has only met once, and there is still no database of data from previous estimates with which CSA could develop historical costs and expertise. This means that the Agency cannot profit from prior experience to improve its cost estimating capacity. Without the appropriate resources and clearly defined responsibilities, the team cannot possibly carry out its work.

Large amounts have already been spent on training the people involved in CSA project management, yet the people charged with making cost estimates have not been provided with the tools required. The result is that they cannot carry out the appropriate work.

Our analyses reveal that CSA is unable to obtain realistic cost and risk estimates for its projects, and this is true commencing with the project planning phase. When we examined changes in project costs, we found a constant need for additional funds or a reduction in the scope of tasks, which leads us to conclude that CSA is very optimistic about what it can accomplish with the financial resources originally allocated to projects.

When we reviewed the reasons given for project budget increases, we identified three main causes: impacts of the technology used, impacts of decisions made by partners and insufficient human resources. In addition to creating a need for additional funding, these three factors very often affected project length and extended schedules. In order for the Agency to make informed decisions, it would be well advised to perform better financial assessments of the risks associated with international partnerships and the development of technology, and provide appropriate resources for the projects it authorizes.

CSA's procedures for determining the funding needs of its projects as well as their human resources, schedules and risks generate additional costs when tasks need to be discussed and revised, schedules adjusted, and the originally defined tasks scaled down or cancelled.

When we analysed cost changes, we found that costs were continually rising in almost all CSA projects except the projects receiving grants and contributions, such as the Cassiope mission, and those under fixed-cost contracts (see table, page 28).

Our examination of changing project costs raised questions about the Agency's ability to make the required changes to its project management practices. More specifically, we expected that the Agency would have made improvements to how it prepares and validates cost estimates. The Agency's record in cost, schedule and risk assessments reveals that it has not been able to learn from experience and apply this knowledge to its project management or, more specifically, to how it validates cost projections.

Cost estimates are based on certain working assumptions related to the technology, technology developed in prior projects that can be used in other projects, a project's stability, etc. CSA must learn from its experience and apply the knowledge when estimating project costs. It must draw on its experience and make adjustments to its assumptions about the technologies used, its international partnerships and the impacts of changes. Our analysis of CSA projects and the causes cited for cost increases reveals that such adjustments are not being made.

The table on the next page shows various assumptions that went unadjusted, only to be re-used in several projects. This supports our contention that lessons are not being applied and demonstrates that CSA cost estimates are not realistic.

The cost estimate performed at the start of Phase A is less precise than the estimate made at the end of the phase. The same is true of adjusted cost estimates made at the end of Phases B and C. As a project advances, cost estimates for the phase and the overall project should become more precise. It would be useful to know the uncertainty level in cost estimates submitted for decisions on overall project funding levels.

According to the table on Page 28, the average change in project costs is a 77% increase. Removing extreme cases from the data reduces the variance in cost increases to 39%. A similar analysis made in 2004 found a 42% variance in the form of project cost increases. For projects begun in 2003 and after eliminating extreme values, costs increased at a rate of 26%. It should be noted that these rates are calculated on the basis of budgets for Phases B, C and D. If data on all projects had been available, we also could have validated these rates with data for phases A
and E.

The 3% reduction in the project cost variance rate is minimal, given that the employees working on projects at all reporting levels should have learned from their experience and been able to manage projects better, particularly in terms of the impact of an immature technology and the interface with international partners.

Overly Optimistic Assumptions Used by CSA in Cost Estimates, Based on an Analysis of Cost Increases

Optimistic Assumption MVIS JWST UVIT NEOSSAT CHINOOK RADARSAT-
CONSTELLATION
Prior projects and past experience will provide leverage X X
Requirements will not change X X X X X
Funding will not be affected by any work delays X X X
The technology is mature enough to be used, so the project will be able to move on to the next phase X X
The impacts of an international partnership will be limited X X
Factors that explain cost stability Fixed cost Unawarded contract Project in Phase A only
Inflation CSA does not account for inflation in its cost estimates. The only exception is salary forecasts, which are adjusted at a rate of 3%.

Our findings with respect to the Agency's application of the Project Approval and Management Framework (PAMF) match those made by the Auditor General of Canada in 2002. Managers are still not willing to attribute appropriate risk levels to their projects when it risks raising project costs above the $5-million financial authority threshold. Documentation, cost projections and risk assessments still do not comply with PAMF requirements and sound management practices.

Our discussions and review of the correspondence between the private sector and CSA lead us to conclude that almost all the projects have cost estimates that fall far below the actual costs. In some cases, the Agency asked industry to lower its project cost projections to make it easier to secure approval. According to CSA's correspondence with industry, CSA requests projections on the basis of available funds, information that is provided to its contacts in the industry to be used when they prepare their projections.

Changing Project Costs

Project Initial Cost Estimates
Phases B, C & D
in Thousands of Dollars (Year)
Current Cost Estimates
Phases B, C & D
in Thousands of Dollars (Year)
Cost
Variance
Earth Observation
CSSP (MCP) 697.0 (1986) 1,396.5 (2003) 100.36%
Radarsat-1 (MCP) 541.0 (1989) 620.4 (1995) 14.60%
Radarsat-2 (MCP) (1996) (2006)1 81.22%
CLOUDSAT 13.9 (2000) 15.3 (2005) 10.07%
SCISAT 32.5 (2000) 60.3 (2004) 85.54%
SWIFT 32.5 (2001) 42.8 (2003)2 31.69%
CHINOOK (2006) (2007)1,3 19.98%
Radarsat- Constellation (2005) (2006)3 8.23%
Space Science
MVIS 1.1 (1998) 10.0 (2005)4 809%
MOST 5.3 (1999) 9.8 (2004) 84.91%
Insect Habitat 10.4 (2000) 10.1 (2005) (2.88%)
Mars PHOENIX 23.9 (2003) 29.8 (2007) 24.69%
MiMBU 6.3 (2003) 6.3 (2006)5 0 %
Herschel HIFI 8.7 (2003) 11.0 (2006) 26.44%
JWST (2003) (2006)1 46.43%
e-Osteo (2004) (2006) 17.65%
UVIT (2003) (2006) 65.79%
NEOSSAT (2005) (2006) 27.08%
APXS (2005) (2006) 24.05%
Source: RPP/DPR, long-term capital plan and PAD - These variances do not include variances for Phases A and E, which are part of the total life-cycle cost.

1 Based on past experience, the costs may continue to rise since the project is not complete, and there have been suggestions that these are not the final costs.
2 The project was stopped when partners withdrew their participation. After the international partners withdrew, SWIFT was modified, becoming CHINOOK. CHINOOK has not been approved at these cost levels.
3 Currently in Phase A only.
4 The actual direct labour costs have not all been allocated to the project.
5 Project not completed.

This approach is used to ensure the implementation of a new project that CSA management wants and that would probably not be considered viable if this method was not employed. The approach distorts the entire project tracking process and complicates project management by requiring new, more realistic estimates later in the process. It results in employees and industry wasting a considerable amount of time and effort justifying these new estimates or explaining why costs have increased since the original estimates. They must then conduct a review of the original project specifications, looking for ways to stay within the project's funding limits. Without a realistic budget, the EC arrives at a flawed decision, and the risk of cost and schedule overruns is more acute. The gap between estimated and actual costs is exacerbated by the fact that CSA's projections do not account for inflation.

CSA harbours a certain amount of distrust in the cost estimates produced by Canadian industry. This can be explained by the fact that there are very few companies in the Canadian space industry, so it can exert additional pressure on the actual prices it charges CSA. CSA managers do not have the tools they would need to validate the accuracy of cost estimates provided by the private sector.

Space Technologies and Space Science use a weak estimating process in Phase A. Their process is essentially based on estimates provided by the contracting parties, and CSA does not have valid estimating tools or a structured data or memory bank of historical project costs. Proceeding without realistic projections, the Agency loses a considerable amount of time reworking its projections, updating documents, discussing the costs with industry and recommencing the project approval process with the Executive Committee and Treasury Board.

In order to meet the cost schedule in Phase B and subsequent phases, more accuracy would be needed in the Phase A schedule, and by the end of Phase A, CSA would need to have more precise costs for the entire project. The cost estimates submitted for approval should attain an appropriate assurance level. Based on historical changes in project costs, we found that the Agency usually minimizes the full extent of the resources required to carry out a project (project life-cycle costs), commencing in Phase A. This adds to the work required to plan and develop the project and generates additional costs throughout the project.

Recommendations

Branches: Space Technologies, Space Science, Space Programs, and Operations and Assets

14) Validate requirements and needs and develop realistic cost estimates and project schedules for making decisions and allocating sufficient resources. Before authorizing phases after Phase A, validate the designs and the maturity of the technology.

15) Ensure that CSA has the right number of people with the appropriate knowledge, abilities and skills who will be available at the right time and place.

16) Define the role of the Cost Estimating Team, monitor the validity and accuracy of estimates provided by industry and invest in the development of project costing expertise.

17) Ensure that the PAMF and central agency policies are applied in the preparation of project cost estimates and that projects satisfy requirements in terms of a well-defined scope and realistic cost projections and schedule before they are launched.

Top of page

3.6.4 Maturity of the Technology Used

We expected that the Agency would have implemented and would be applying specific criteria on the level of maturity required before a project could move to the next stage. We expected that a system would be in place to measure acquisition of knowledge and ensure that adequate knowledge is acquired before projects are transferred to the next phase.

Studies have shown that in other space agencies, projects using a mature technology run 9% over their budgeted development cost, on average. In cases where an immature technology is being developed, this figure averages 41% of total costs. The maturity issue also has an impact on project schedules. Projects using a mature technology fall 7 months behind schedule, on average, while those relying on an immature technology average 13 months behind schedule.

CSA does not have detailed studies of this phenomenon, but the table of changes in project costs (page 28) shows an average cost increase of 39%. Schedules for CSA projects are extended several years. In the six projects in our sample, the average project fell 36 months behind schedule. The table on page 27 shows that overly optimistic working assumptions have an impact on these costs, including assumptions about the maturity of the technology.

CSA must ensure that it has a technology that has attained maximum maturity before commencing the phases that follow concept and feasibility development (Phase A). Otherwise, actual costs will be driven up and exceed estimates.

The Agency initiates the development phase with technologies that have not attained an advanced level of maturity. It also cuts back on tasks in prior phases, such as engineering model development. This increases project risks and raises the possibility of running higher costs and falling behind schedule in order to correct what could have been addressed earlier in the project.

By not establishing a minimum threshold of technological maturity before moving on to subsequent phases, the Agency significantly increases the risk that design changes will be required later during payload development (radar, satellites or scientific instruments). These costly changes are often an important source of project cost increases.

A technology that is not mature at the end of Phase A will only be completed in Phase C or D, and this drives up project costs and risks. The same principle applies to transferring a project to Phase B or Phase C without having acquired basic knowledge. Our review has shown that the performance of program employees is assessed on the basis of their ability to produce documents by a specific date or hold reviews on a specific date. Instead, the Agency should ensure that an adequate level of knowledge has been attained before a project is transferred to the next phase. CSA does not have a standardized system for performing major reviews of key project stages, and this creates inefficiencies. The goal of such reviews is to ensure that sufficient knowledge has been acquired. Without a serious review of each project phase, the Agency has no way of ensuring that sufficient knowledge has been acquired to move on to the next phase.

For all projects over $20 million, NASA uses a knowledge acquisition system to measure and assess project management performance in terms of its ability to develop the technology within budget objectives and project schedules as projects are implemented.

Recommendations

Branches: Space Technologies, Space Science, Space Programs, and Operations & Assets

18) Have an appropriate system for ensuring that a technology is mature before it is transferred to Phase B and that the required basic knowledge has been acquired before moving to the next level.

19) Use performance indicators that are not based on the production of documents or holding reviews by a specific date.

Top of page

3.6.5 Project Tracking

We expected that the Agency would have implemented and would be using an integrated cost accounting system in its projects and programs in compliance with central agency policies and according to sound management practice. We expected that CSA would account for all the costs incurred in each project.

We expected that CSA would use a "go / no go" decision process to stop or temporarily suspend a project when the initial funding proves insufficient or when international partners withdraw from a project. It could also use this mechanism to stop a project that requires many more resources than originally planned, such that the expected benefits may no longer be justified at this spending level.

Cost Tracking, Analysis and Allocation in Projects
Sound project management practices require that all the costs incurred to complete a project are allocated to it. Under TB project management policy, all direct and indirect costs must be allocated as project costs.

In our analysis of projects, we found a major flaw in how project costs are being compiled and allocated. When it prepares project budgets, CSA estimates the direct labour costs, operating expenses and capital costs. Direct labour costs are allocated to projects on the basis of these projections. Our analysis revealed that the estimates of direct labour costs are rarely adjusted to allocate the actual incurred cost. In addition, the overtime of some project managers is not recorded as a project cost. For most projects, all the direct labour costs are not allocated, and the current accounting systems do not allow us to identify the actual direct labour costs incurred before 2005.

The current systems cannot compile the actual time employees spend on each project as a means of accounting for the actual funds spent. It is therefore difficult for project managers to know their actual direct labour costs and avoid exceeding financial authorities. The direct labour and indirect costs are not all allocated to projects, and the same is true of general and administrative expenses. This is inconsistent with sound management practice, CSA and TB project management policy, the PAMF and accounting principles with respect to asset capitalization.

Several recommendations have been made in the past to identify and allocate actual project direct labour costs, but the Agency has decided not to respond. Some maintain that these costs are too limited to justify implementing a system that compiles actual costs. Since the actual direct labour costs associated with projects have never been compiled and allocated, it is difficult to determine their scope. For example, among the projects in our sample, direct labour costs averaged 7.5% of total estimated project costs.

By proceeding in this manner, CSA runs the risk of being poorly informed about the amounts it spends and not complying with the terms of signing authorities received from Treasury Board. When we tried to have all the costs for our series of projects, it was very difficult if not impossible to obtain total costs.

The Agency has adopted a practice of determining the full-cycle cost of Phases A to E for the purposes of project funding decisions. CSA does not analyse variances between cost projections and the actual costs incurred over the full life cycle of a project. It does not compile costs on a life-cycle basis, i.e., the costs of Phases A to E, inclusively. It therefore does not know the actual costs of its projects, including operations. It is impossible for CSA to perform an analysis of whether completed activities have generated value in relation to what it cost to build and develop the project's scientific and satellite instruments.

Transferring Projects to the Next Phase
Several managers mentioned that Phase D tasks are transferred to operations in order to bring the project to an end while staying within its funding authorization. This transfer occurs without the associated budgets necessarily being transferred. There are also development stages not planned in Phase D that must be completed in Phase E before the system can be operational. This approach prevents us from determining actual project development costs, since operating costs are not compared to the budget projections used to obtain project approvals.

The assessment report for the MVIS project states that the project cost $10 million, leading us to assume that CSA was within its $10 million spending authority granted by Treasury Board. The direct labour costs were not all allocated to the MVIS project. It is also necessary to add expenses related to integrating the MVIS project in Phase E, as well as development activities that were included in the $10-million cost projection but that were in fact performed in Phase E. We asked for the total cost analysis for the MVIS project, but it was impossible to gain access to all the project's costs.

According to the preliminary reports, development activities generated $1 million in additional costs. The MVIS project was transferred to the Space Operations Branch before it became operational. The costs incurred in the first year following the transfer all resulted from development activities. This was also true for a portion of the costs incurred in the second year of operation.

Without performing a major review at the end of each phase to ensure that an appropriate level of knowledge has been acquired before transferring a project to the next phase, CSA exacerbates the risk of cost overruns and late projects, and this hampers performance.

Control Methods
Phase reviews must be performed at specific stages in order to manage projects well and be able to stop a project if significant problems arise. They should not be used to evaluate employee performance. CSA has never stopped or delayed a project. We did not find any project cost-benefit analyses performed to ensure that a project was still worth pursuing. Given that other space agencies have abandoned projects before completion, this phenomenon at CSA remains unexplained.

Senior managers and even the Director General of Space Programs conduct monthly project reviews. There is also a monthly review of project-related risks, weekly reports prepared by project directors and managers and regular reviews with the contractors working on the project. Despite all these reviews, project costs climb above the initial budget levels.

As mentioned above, in the space industry it is not uncommon to have variances in the order of 9% when the technology is mature. According to the project table (page 28), costs for CSA projects have risen 39% on average. To this must be added the many changes made to projects, causing them to fall behind schedule, and the discussions aimed at reducing or reworking cost estimates in order to meet the allocated budget. All of this drives up costs.

Recommendations

Chief Financial Officer

20) Implement a system to compile actual project costs, including all direct and indirect costs, in order to properly account for all the funds spent.

21) Account for the full project life cycle, since CSA approves projects on this basis in order to show that they add value.

Branches: Space Technologies, Space Science, Space Programs, Operations and Assets, Directorate, Safety and Program Assurance

22) Ensure that all projects (with the exception of MCPs) are reviewed by an independent group before signing the PDR, before commencing manufacture (CDR) and at the end of the manufacturing period (AR). Follow NASA's example and adopt a system for acquiring knowledge and managing earned value by establishing the criteria for a successful conclusion to the project review (SRR, PDR, CDR, AR) held at the end of each phase.

Top of page

3.6.6 Risk Management

Our review of the management of project-related risks found that the implemented practices and procedures are generally being followed:

  • Project risks are qualified, evaluated and quantified using a uniformly applied methodology and under the supervision of the risk manager;
  • Risks are documented with PADs, regularly reviewed and provided to the concerned stakeholders from the RIAS database;
  • The risks are quantified and weighted in the RIAS database and subject to a long-term commitment (by blocking funds to cover the identified risks). This is documented in the Five-Year Risk Assessment and Source Funds Plan, which is managed by the Financial Planning Division. However, we found discrepancies between the Five-Year Risk Assessment and Source Funds Plan and the information maintained on individual projects;
  • A process requiring PRAB authorization was implemented in order to approve commitments when a specific realized project risk exceeds 75% of the initial risk reserve.

We expected that CSA would have implemented the Risk Management Framework it has developed over the years, that it would authorize sufficient risk reserves to mitigate its risks, and that it would proactively prevent risk.

Risk assessments are performed using appropriate and detailed methods that identify and justify various risks and capture both internal and external risk factors in a comprehensive risk assessment. The risk reserve is determined as a percentage of all the costs estimated for Phases B, C and D. Since this estimate falls short of actual funding needs, the risk reserve is inadequate.

Risks must be established and sufficiently detailed in the project management and approval documents so that decision-makers will be aware of them and able to anticipate the impacts they could have on a successful project. Our review of a series of documents revealed that very few projects are given a high risk score, yet project costs often rise when technical or partnership risks occur.

Our study of the occurrence of risk in various projects leads us to conclude that technical performance risks are underevaluated. An appropriate criterion, in the form of a performance threshold for the technology, is not established beforehand. It appears that several problems occur in CSA's interface with its partners in the various projects. Based on our analysis of occurrences of partnership-based risk, the financial assessments of risk are inadequate, and it appears that the Agency does not learn from experience.

According to our analysis, CSA maintains a risk reserve of approximately 10% of project costs. This level is not realistic, given the Agency's area of activity and the record on CSA project management. For example, according to risk management criteria, the international project JWST should have a hazard rate of at least 50%. The current rate for JWST is 16.5%.

The risk committee meets once every four months with the Director General of Space Programs. Experience has shown that risks are being underestimated. Few projects are assessed with high hazard rates, when in reality several projects present a high level of risk because of a partnership that has an impact on the technology. The available reserve does not shelter the Agency from the project cost increases that would result if these risks occurred.

Employees explained that one of the ways used to avoid cost overruns during a project is to cut back on the tasks required of the contractor. This is done to maintain budgets within allocated amounts. CSA reduces project costs by making changes to the original specifications and extending work schedules in order to reduce the cost per phase. In the process, CSA increases project risk and defers the funding problem to later phases.

Recommendations

Chief Financial Officer
Branches: Space Technologies, Space Science, Space Programs, and Operations and Assets

23) Make more realistic assessments of the risk reserve from the outset; otherwise, the entire risk management process is flawed. The current method for establishing a risk reserve would only be appropriate if project cost projections were more realistic.

24) Increase the risk reserve when there are international co-operation projects. Base the increase on an analysis of why project costs increase under partnerships.

President

25) In order to ensure an appropriate level of independence and impartiality in project management, have the officer in charge of risk management report to a corporate segment rather than to the Director General of Space Programs.

Top of page

3.7 Project Performance Assessments

We expected that the Agency would have implemented tools for measuring project performance outcomes. It is important for CSA to be able to show the contribution that funded projects make to strategic outcomes and measure their impacts for Canadians. We expected that these outcomes would be measured by the people who are responsible for them.

The Directors General of Space Technologies and Space Science provide the funds required throughout projects, from Phase A to Phase E, inclusively. This is because CSA gives them a large share of the available budgets to initiate the research that generates projects and programs. The Space Programs Branch, on the other hand, is in charge of project management in Phases B, C and D and has no control over the budgets allocated to projects. The same is true of the Space Operations Branch, which receives a portion of the operating budgets for the payloads or satellites that CSA operates.

According to the Program Activity Architecture (PAA), the directors general are responsible for measuring outcomes of program sub-sub activities, and the VP-STP is responsible for attaining outcomes at the two other levels: program activities and program sub activities for Earth Observation (EO), Space Science and Exploration, Satellite Communications and Space Mission Operations.

The PAA also indicates that the Director General of Space Technologies is responsible for R&D programs in EO space technology, and the DG of Space Science and Exploration is responsible for scientific programs in atmospheric environment science and human space exploration. The Director General of Space Programs has joint responsibility for project management in EO and SE, shared with each of the Directors General of Technologies and Space Science and Exploration. Responsibility for operations in these different program activities is sometimes shared between the Directors General of Space Operations, Space Technologies and Space Science.

CSA's matrix structure does not allow the directors general to be held accountable. It does not encourage partners to be accountable to Parliament and Canadians for the amounts spent on projects that are connected to PAA program activities.

Whenever we mentioned outcome measures, we found that our interviewees did not feel accountable to Parliament and Canadians for outcomes. Transferring projects from one branch to another according to its phase does not encourage employees to be accountable for outcomes.

When we looked at projects and programs, we met no one who felt responsible for attaining outcomes and objectives besides building an instrument. Few managers are able to report outcomes beyond the manufacture of an instrument. The Space Programs Branch is not involved in projects from the outset to define objectives. Despite the fact that employees from their segments are involved in the management process, sponsors lose a sense of where projects are heading in terms of their non-technical outcomes.

The content of management agreements also plays a role in this lack of accountability for outcomes. The agreements are based on meeting a given deadline or submitting a document or performing a review by a specific date. Employee performance assessments are based on conducting a review on a specific date. In this way, the Agency encourages fast execution at the expense of efficiency and effectiveness. Prematurely transferring a project to the next phase rather than conducting a proper review of the technology's development or a "go/no go" phase review is not efficient. It fosters neither sound management nor proper controls of costs and risks.

The same can be said of CSA program and project performance assessments. When performance assessments are based on the amount of data collected or the number of images produced, the Agency is not measuring program outcomes, but rather what is produced in the course of day-to-day activities. CSA does not measure the impact of its projects and programs for Canadians.

A sound governance framework that clearly defines roles, responsibilities and obligations with respect to accountability would increase the Agency's chances of improving its project management.

Recommendations

Directorate, Planning and Performance

26) Improve performance measurement for projects that interface with the PAA so that performance is compared with pre-established objectives and Parliament receives better reports.

Branches: Space Technologies, Space Science, Space Programs, and Operations and Assets

27) Designate someone to be accountable for the outcomes of all project phases, including operations, so that the Agency can report on project outcomes for Canadians.

28) Perform a regular analysis of total project costs vs. initial budgets to try and draw some conclusions and, through a cost-benefit analysis, examine the issue of whether the outcomes justify spending the funds.

Top of page

APPENDIX A Objectives and Audit Criteria

Objectives

The goal of this audit was to determine whether the Agency's projects are well managed. More specifically, it focused on the following:

  • Analysis of needs and options;
  • Risk management;
  • Tests and assessments;
  • Project management, tracking and control; and
  • Implementation of the most appropriate project management practices.

Scope

Our audit covered all aspects of project management. We also compared the Agency's practices and efforts to improve with the best practices used elsewhere.

Six projects were identified out of a total of 18 active projects. The judgmental selection procedure was used to select the elements to be audited. The projects selected fell under the responsibility of four branches: Space Science, Space Technologies, Space Programs and Space Operations. Our selection criteria were designed to collect a representative sample of projects.

The following projects, representing 26% of all CSA projects, were selected.

Sample (in millions of dollars)
Sponsor Phase Project Total Cost
Phases A to E
Budget Allocated in IPS Budget Allocated in SAP
Science C CHINOOK
Science C JWST
Science E MVIS
Technologies B-C-D NEOSSAT
Technologies A-2 RADARSAT-C
Science B-C-D UVIT
Total Sample 945.93 537.77 167.6
Coverage of total population (%) 40% 57.8%

For each project, we contacted the Project Management Office to discuss relevant management issues. We obtained and consulted available documents on needs and options analyses, risk management, tests and assessments, project management and the production of related reports.

More specifically, the objectives were:

Objective #1 Ensure that operations are planned.
  Criterion 1.1 There is a structured planning process.

The long-term investment plan includes capital projects.

The 10-year business plan is aligned with the Agency's strategy. It is followed and used when selecting initiatives to fund.

  Criterion 1.2 The initiatives are approved.

Approvals of projects and initiatives are based on realistic cost estimates for the entire project life cycle (Phases 0 to F).

  Criterion 1.3 Financial and human resources are allocated to approved initiatives.

Risk management allows for proper identification of risks. They are addressed with risk mitigation plans and adequate financial resources.

  Criterion 1.4 The planning process is designed to facilitate accountability.

Objective #2 Ensure that the initiatives and resources used are controlled.
  Criterion 2.1 Work organization, division of responsibilities and delegated authority contribute to the delivery of programs in accordance with the principles of effectiveness, efficiency and cost-effectiveness.
  Criterion 2.2 Managers provide employees with direction and the means to attain organizational goals.
  Criterion 2.3 Spending is duly approved and follows applicable acts and regulations. Spending authority is controlled.
  Criterion 2.4 There are management controls that allow managers to periodically measure achievements against objectives (in terms of cost, timelines and performance).
  Criterion 2.5 Reliable financial information systems can be accessed in a timely manner so that senior management can quickly respond to and address cost overruns.

The tracking process allows for effective budgetary control. It is used to identify, within a reasonable period of time, cost overruns or a need for additional resources.

Objective #3 Ensure that managers report outcomes and the methods they employ.
  Criterion 3.1 An appropriate accountability framework lays the foundation for efficient accountability.
  Criterion 3.2 Each program element is accounted for on a regular basis.

Initiatives are reviewed under a structured process so that programs can be reviewed in terms of Agency priorities, the 10-year plan and the long-term investment plan.

Top of page

Appendix B Summary of Findings from Selected Projects

CSA's cost estimates for the MVIS project are considerably lower than those reported in an external study. Here is how they changed over time:

Changes in Costs for the MVIS Project (in Thousands of Dollars)

Date of Estimate BCD Development Operations External Study
August 1999 $1,100 $800 $15,000
June 2000 $3,000 $800 $10,700 (before CSA expenditures) $12,300 (including CSA expenditures)
August 2001 $4,700 $800
December 2002 $10,0001 (with reduced project scope) $2,500
Final costs reported by CSA $9,9312 Development continues in Phase E. The system is not operational.
Costs for this project were not tracked by phase.

This project was characterized by:

  • Underestimated costs;
  • Cost overruns: all labour costs were not allocated to the project, and development activities were continued in Phase E;
  • Changes made to the scope of the project without reducing the budget;
  • Late delivery; and
  • No accounting for some of the costs incurred by CSA for this project, including salaries and operating expenditures.

The project's evaluation report does not indicate that CSA's spending authority was exceeded, nor the project's budget.

Since the payload is still not in orbit on the ISS, it is impossible to say whether it will provide the expected outcomes for Canadians.

1 Not all costs were accounted for in this project.
2 These costs are incomplete and do not include Phase E development activities.

Changes in Costs for the JWST Project (in Thousands of Dollars)

Date of Estimate BCD Development Operations External Study
October 2003    
February 2004    
July 2004    
April 2005      
November 2006    

These figures do not include Phase A and scientific support directly related to the project.

This project was not managed as an MCP despite signs that the risk level was high and costs would probably exceed $  million. It was only recognized as an MCP in the summer of 2006.

The risk reserve does not cover all the risks identified for this project; it is undervalued.

As is the case with other CSA projects, not all the actual labour and indirect costs have been allocated to the project.

It was very difficult to track the changes in this project and to measure their impact on outcomes for Canadians.

No cost-benefit analysis was performed after project costs increased to ensure that Canadians would still benefit from the project.

The end of the project has been postponed for several months, perhaps years, so its final costs will be higher.

Changes in Costs for the NEOSSAT Project (in Thousands of Dollars)

Date of Estimate BCD Development
CSA's portion only1
Operations External Study
April 2005
August 2005
October 2005
February 2006
May 2006
1 This is a $  project carried out in partnership with the Department of National Defence (DND).

Solving a problem due to the contractor's limited liability required considerable effort and caused an increase in direct labour costs.

In May 2004, the PAD indicated $  for Phase A, an estimated budget of $million for Phases B, C and D plus $  million for the launch. The total cost ($  million) was to be shared by the Department of National Defence ($ million) and CSA ($  million).

In August 2004, the Agency submitted this project with a budget of $  million for Phases B, C and D and $  million for operations, to be shared by the Department of National Defence and CSA.

A proposal was made for a $  technical study for Phase A in January 2005.

April 2005: EC approved $  million in total costs for Phases B, C and D.

The contractor informed the Agency of a(n) $  increase in costs for the payload. Despite this increase, the Agency believes that the project can be completed for a budget of $  million.

Changes in Costs for the UVIT Project (in Thousands of Dollars)

Date of Estimate BCD Development Operations External Study
June 2004    
January 2005      
March 2005    
May 2005      

February 2005: An external study of the cost of Phases B, C and D gave a total cost of $ , excluding the cost of DFL services and CSA salary, travel and G&A expenses.

May 2005: The UVIT project was submitted to the EC. The annual operating costs of $  per year were deemed too high. Since the costs submitted by external partners had increased, total costs rose from $  million to $  million. No source of funding had been identified for the remaining $ . It was suggested that one solution would be to lower the risk reserve, but this idea was rejected.1

Actual costs incurred by CSA for scientific support are not included in project costs. Since these costs would not have been incurred if the project had not been funded by the Agency, they are directly related to the project and therefore should be allocated to total project costs. This is not the case at present. We therefore have concluded that all project-related direct and indirect costs are not allocated to projects, as required by TB management policies and by sound management practices. As of November 2006, these costs totalled $ .

1 This information could be considered of a delicate nature.

Changes in Costs for the CHINOOK Project (in Thousands of Dollars)

Date of Estimate BCD Development Operations External Study
May 2005    
June 2005    
September 2005    
June 2006      

In August 2003, CSA submitted an estimate of $  million for Phases B, C and D of the SWIFT project.

In March 2005, CSA started talking about the CHINOOK mission.

After the international partner withdrew from the project, CSA decided to transform the project based on the work already completed and fund it as the CHINOOK project. It would be an all-Canadian project that would build on SWIFT project development.

In 2006, it was estimated that the actual costs of the CHINOOK project were surpassing the identified sources of funding.

Changes in Costs for the RADARSAT-CONSTELLATION Project (in Thousands of Dollars)

Date of Estimate BCD Development Operations External Study
June 2005      
March 20061    
January 2007  
Including the risk reserve
   

This project is presently in Phase A. Sources of additional funds totalling $  million have not been identified.

Phase A was extended in order to bridge Phases A and B while CSA waits for the proper spending authorities.

1 PAD phase BCD
Note: Classified or confidential information has been withdrawn from the text.

Top of page

Appendix C Management Action Plan

Ref. Recommendation Designated Authority Action Plan Details Timetable
Organization Function
3.2 Governance – Decision-Making Process
1) Implement the governance review begun in 2005 and review the method used to approve projects. Office of the President President The streamlining of the CSA organizational and committee structures has brought improvements to internal governance and decision-making. The establishment of independent directorates reporting to the Senior Vice-President for Safety and Program Assurance and Planning and Performance will ensure the appropriate oversight and approval of projects. The LTCP has established a priority-ranking framework that will improve the project approval process. March 2008
2) Set clear objectives, develop a strategic and implementation plan and share these objectives and the strategic and implementation plan with CSA staff. The Executive Committee has undertaken a strategic review including a planning and priority-setting exercise resulting in new strategic priorities for the CSA, which will be captured in a five year Corporate Business plan.

Furthermore, the Executive Committee as part of its yearly planning cycle, including financial planning will undertake to establish clear objectives and plans. A Senior Management meeting will be held twice a year and staff will be updated on a quarterly basis regarding these objectives and plans.

March 2008
3.3 Obtaining TB Funding Authorizations
3) Ensure that all employees are acquainted and are complying with the processes used to obtain TB financial approval and the PAMF. Directorate, Safety and Program Assurance

Branches: Space Technologies, Space Science, Space Programs, and Operations and Assets

Director, Safety and Program Assurance

Directors General

Following the streamlining of the CSA organizational and committee structures, the PAMF will be reviewed to reflect these changes. The Safety and Program Assurance Directorate will review the PAMF.

The Directors General will ensure that personnel are familiar with the process to obtain Treasury Board authorizations and will ensure the respect of Treasury Board policy.

March 2008






June 2008

4) Ensure that CSA obtains TB approval earlier in its project planning and that no project is exempt. Branches: Space Technologies, Space Science, Space Programs, and Operations and Assets Directors General It is the objective of the Director Generals to seek Treasury Board authorizations as early as possible. When good estimates are available, Preliminary Project Approval will be sought at the end of Phase A. It is recognized and Canadian Space Industry has been made aware that this could generate a gap between the end of Phase A and the start of Phase B.

Treasury Board submissions will be prepared using a team approach with members from corporate and operational sectors.

Ongoing
3.4 Transparent, Clear and Complete Information
5) Standardize and submit important information to the EC. Finance Chief Financial Officer

Planning and Financial Analysis

Planning and Financial Analysis will work with PAMF managers to identify the information required by the EC to make informed decisions. A first draft of information deemed critical should be provided to members of the EC before the end of November 2007. This will allow feedback and validate their needs as a decision-making committee. March 2008
6) Provide all the information required for timely decision-making and ensure that key project elements can be clearly understood. Branches: Space Technologies, Space Science, Space Programs, and Operations and Assets Directors General The streamlining of the CSA organizational and committee structures has brought improvements and simplifies the decision-making.

In order to foster communications and to ensure coordination across sectors and branches, the CSA President has appointed Champions from within the Executive Committee's members who will bring greater focus and alignment of programs and projects to the four CSA Strategic Thrusts and the CSP.

These Champions will represent the CSA for each of their respective Strategic Thrusts and the CSP. Through Thrusts Steering Committees newly created, they will facilitate crosscutting consultation within the agency, bring program, and project issues to the Executive Committee for discussion and/or decision.

Periodic Project Reviews and the implementation of the proposed Project Assessment Review will establish and improve the knowledge base of senior executives prior to decision-making.

Completed





Completed

 




















Ongoing

3.5 Project Approval and Management Framework (PAMF)
7) Relocate the team responsible for applying the PAMF to ensure that it has the independence and impartiality needed to exercise proper oversight, and provide the team with the expertise required to fulfill its role certifying compliance with legislation, policies and the PAMF. Office of the President President As part of the CSA reorganization an independent Directorate reporting to the Senior Vice-President for Safety and Program Assurance has been established and will ensure the appropriate oversight and approval of projects. Completed
8) Clarify the roles of the team responsible for applying the PAMF. Office of the Senior Vice-President Senior Vice-President Following the establishment of the Safety and Program Assurance Directorate under the Senior Vice-President, the roles and responsibilities will be reviewed. March 2008
3.6 Elements Influencing Project Costs
3.6.1 Project planning phase
9) Spend more time preparing and planning projects and make better assessments of the financial and human resources required to carry out projects and attain outcomes effectively and efficiently. Branches: Space Technologies, Space Science, Space Programs, and Operations and Assets Directors General Overall project planning will improve following the streamlining of the CSA organizational and committee structures and the implementation of an Executive Committee annual strategic review to update the CSA Five Year Corporate Business plan

The Director General Space Programs will continue with the support of the other Directors General in the implementation of the e-RAM as the planning tool for human resources required for space projects.

Refer also to Action 14 for further action on the Cost Estimating Team

Ongoing
10) Establish priorities and reduce the number of active projects in order to ensure that projects are adequately funded, attain approved objectives and are managed efficiently and economically. Office of the President President The Executive Committee has undertaken a strategic review, including a planning and priority-setting exercise. Furthermore, the Executive Committee as part of its yearly planning cycle, including financial planning, will undertake to establish clear objectives and plans. The LTCP has established a priority-ranking framework that will improve the project approval process. Ongoing

March 2008

Elements Influencing Project Costs
3.6.2 Changes in project scope
11) Standardize basic project management principles; ensure that project management and PAMF policies are known to all project managers, directors and directors general and to senior management. Ensure that these principles and policies are uniformly and rigorously applied by all managers. Directorate, Safety and Program Assurance Director, Safety and Program Assurance As part of the CSA reorganization, an independent Directorate reporting to the Senior Vice-President for Safety and Program Assurance, and the streamlining of the CSA organizational and committee structures will support the standardization of project management and oversight of PAMF application within the Agency. The PAMF will be reviewed in light of the CSA reorganization. March 2008
12) Perform a cost-benefit analysis at the start of each project and when the scope of a project is changed or project costs increase. Branches: Space Technologies, Space Science, Space Programs, and Operations and Assets Directors General The relevant Project Sponsor will implement a cost/benefits at the start of the project and when changes occur during the project using the priority criteria of the LTCP. This cost/benefit will be part of the presentation to the Executive Committee when project authorization is being sought. With new project or ongoing projects
13) Review the new cost estimate in light of the benefits identified at the beginning of the project, and ensure that the project still represents CSA's best means to obtain the desired outcomes for Canadians. The cost/benefit established at the start of the project will be updated by the Project Sponsor and will brief the Executive Committee when revised authority is required. With new project or ongoing projects
3.6 Elements Influencing Project Costs
3.6.3 Changes in cost estimates
14) Validate requirements and needs and develop realistic cost estimates and project schedules for making decisions and allocating sufficient resources. Before authorizing phases after Phase A, validate the designs and the maturity of the technology. Branches: Space Technologies, Space Science, Space Programs, and Operations and Assets Directors General The Director General Space Technology will review the mandate of the Cost Estimating team and will present a proposal after consultation, to the Executive Committee, on the role and the need of such a team.

Following the phase A of a project, the CSA will re-establish its Cost Revision Committee to review each project and put in place an independent third party process to review and validate the cost for all major projects.

The implementation of the proposed Project Assessment Review will provide a means to establish technology maturity and readiness to transition from one phase of a project to its next phase.

Ongoing







March 2008

15) Ensure that CSA has the right number of people with the appropriate knowledge, abilities and skills who will be available at the right time and place. The Director Generals will continue the implementation of the eRAM as the primary Human Resources planning tool for space projects as well as to ensure appropriate skills are available. Ongoing
16) Define the role of the Cost Estimating Team, monitor the validity and accuracy of estimates provided by industry and invest in the development of project costing expertise. The Director General Space Technology will review the mandate of the Cost Estimating team and will present a proposal after consultation, to the Executive Committee, on the role and the need of such a team. March 2008
17) Ensure that the PAMF and central agency policies are applied in the preparation of project cost estimates and that projects satisfy requirements in terms of a well-defined scope and realistic cost projections and schedule before they are launched. Cooperation between sectors will be put in place and used to develop roles and mandate for PMAF. March 2008
3.6 Elements Influencing Project Costs
3.6.4 Maturity of the technology used
18) Have an appropriate system for ensuring that a technology is mature before it is transferred to Phase B and that the required basic knowledge has been acquired before moving to the next level. Branches: Space Technologies, Space Science, Space Programs, and Operations and Assets Directors General The implementation of the Project Assessment Review will provide a vehicle to assess technology maturity prior to the initiation of a project. The Project Assessment Review proposes criteria associated with technology maturity at different phases of a project.

The reviews will also support the development of a knowledge base prior to decision-making.

March 2008
19) Use performance indicators that are not based on the production of documents or holding reviews by a specific date. Criteria for transition from one phase to another as defined in the Project Assessment Review document will be used. March 2008
3.6 Elements Influencing Project Costs
3.6.5 Project tracking
20) Implement a system to compile actual project costs, including all direct and indirect costs, in order to properly account for all the funds spent. Finance Chief Financial Officer

Planning and Financial Analysis

In 2006-2007, the Finance Directorate hired PWGSC's consulting services to review and integrate some financial policies. As at March 31, 2007, their mandate, which included a project costing phase, was not complete. PWGSC's consulting service has agreed to complete the work this fiscal year. Our objective is to have this new policy in place by March 31, 2008. The impact on CSA's financial systems, if any, will be assessed according to policy requirements. March 2008
21) Account for the full project life cycle, since CSA approves projects on this basis in order to show that they add value. The Finance Directorate will establish a procedure for formalizing the proper use of financial coding. This will facilitate consultations of financial information resulting from approvals obtained at different phases of life-cycle costing. A first draft for consultation should be available in January 2008. In the meantime, special attention will be given to projects that could be the subject of such an approval request. March 2008
22) Ensure that all projects (with the exception of MCPs) are reviewed by an independent group before signing the PDR, before commencing manufacture (CDR) and at the end of the manufacturing period (AR). Follow NASA's example and adopt a system for acquiring knowledge and managing earned value by establishing the Criterion for a successful conclusion to the project review (SRR, PDR, CDR, AR) held at the end of each phase. Branches: Space Technologies, Space Science, Space Programs, and Operations and Assets

Directorate, Safety and Program Assurance

Directors General





Director, Safety and Program Assurance

Implementation of the proposed Project Assessment Review will meet the objectives of this recommendation.



A committee will be established under the authority of the Director, Safety and Program Assurance in support to the project managers.

March 2008
3.6 Elements Influencing Project Costs
3.6.6 Risk management
23) Make more realistic assessments of the risk reserve from the outset; otherwise, the entire risk management process is flawed. The current method for establishing a risk reserve would only be appropriate if project cost projections were more realistic. Finance
Branches: Space Technologies, Space Science, Space Programs, and Operations and Assets
Chief Financial Officer and Directors General Assessment of risk will be part of the Cost Estimating Team mandate. See Action 14 March 2008
24) Increase the risk reserve when there are international co-operation projects. Base the increase on an analysis of why project costs increase under partnerships. Same as above. March 2008
25) In order to ensure an appropriate level of independence and impartiality in project management, have the officer in charge of risk management report to a corporate segment rather than to the Director General of Space Programs. Office of the President President The manager of risk management process was moved from the Directorate of Space Programs to the Senior Vice-president organization, therefore ensuring independence and impartiality. Completed
3.7 Project Performance Assessments
26) Improve performance measures for projects that interface with the PAA so that performance is compared with pre-established objectives and Parliament receives better reports. Directorate,Planning and Performance Director, Planning and Performance The Directorate is completing the Performance Measurement Framework of the PAA in accordance with the MRRS requirements stated by TBS March 2008
27) Designate someone to be accountable for the outcomes of all project phases, including operations, so that the Agency can report on project outcomes for Canadians. Branches: Space Technologies, Space Science, Space Programs, and Operations and Assets Directors General The Project Sponsor will assume this responsibility Ongoing
28) Perform a regular analysis of total project costs vs. initial budgets to try and draw some conclusions and, through a cost-benefit analysis, examine the issue of whether the outcomes justify spending the funds. Regular analysis of project costs vs. budgets carried out by Project Managers and presented at Project Reviews.

The Costs/Benefits Analysis will be presented at start of project, when significant changes of authority required during project and at the end of the program by the Sponsor.

Ongoing