Audit of the Design and Operation of the Sampling Plan

Download the PDF version (88 KB)

Audit Report

Project # 08/09 01-01

prepared by
the Audit and Evaluation Directorate

October 2008

Table of Contents

1.0 Summary

1.1 Audit Objective

The objective of the audit was to evaluate the design and operation of the monitoring system used to identify higher-risk transactions for review and validation.

1.2 Audit Opinion

In our opinion, the design and operation of the sampling plan require significant improvements.

1.3 Statement of Assurance

In my professional judgment as Chief Audit and Evaluation Executive, sufficient and appropriate audit procedures have been conducted and evidence gathered to support the accuracy of the opinion provided and contained in this report. The opinion is based on a comparison of the conditions, as they existed at the time of the audit, against pre-established audit criteria, and is only applicable to the particular entity examined. The evidence was gathered in compliance with Treasury Board policy, directives and standards for internal audit. The evidence has been gathered to be sufficient to provide senior management with the proof of the opinion derived from the internal audit.

1.4 Summary of Recommendations

The Accounting, Financial Reporting and Policies Division is responsible for the Agency's payment requisition monitoring system, subject to internal policies and directives. However, it also relies on the co-operation of managers and financial clerks and officers with responsibility for financial operations in the sectors concerned to improve the efficiency of the account verification process and to periodically review the sampling plan in an efficient and cost-effective manner.

Our audit showed that the sampling plan adequately identifies most higher-risk transactions for account verification purposes, since most payment requisitions have been coded high-risk.

Further to our review of the design and operation of the sampling plan, we recommend

  • that the sampling plan be reviewed and approved, annually and on a need basis, by the Chief Financial Officer (CFO);
  • that the various transaction categories' degree of risk be reviewed;
  • that periodic analytical reports be prepared; and
  • that a correction and follow-up mechanism for the errors detected be developed with the staff of the sectors concerned.

Signature of the Chief Audit and Evaluation Executive

Original signed by Jean-Guy Desrosiers

Audit team member

  • Jimmy Cheung

2.0 Audit Report

2.1 Background

The Canadian Space Agency (CSA) policy is to pay on time amounts actually owed to third parties. Account verification mechanisms must be designed and used in such a way as to ensure their effectiveness, with due regard for the risks related to the characteristics of each payment requisition and to monitoring requirements.

Sampling design

In April 2003, a manual sampling plan was adopted to identify, for review and validation, payment requisitions coding high, medium or low risk based on the nature of the transactions. Under this plan, 13 of the 20 transaction categories were coded high-risk.

The sampling plan is based on a feasibility study that took place between October 1, 2002, and February 28, 2003.

Sampling operation overview

Sampling operation overview.

Text description of image

2.2 Audit Objectives, Scope and Approach

The purpose of the audit was to evaluate the design and operation of the monitoring system used to identify higher-risk transactions for review and validation.

The audit dealt with the design and operation of the sampling plan instituted in April 2003. We also examined the breakdown of payment requisitions by transaction category for fiscal year 2007-2008.

Various audit processes were employed, including staff interviews and reviews and analyses of documents and records. A review of the existing sampling plan and the main departmental policies and directives was done in order to develop audit criteria.

2.3 Findings, Recommendations and Management Response

2.3.1 Effectiveness of Existing System

The Accounting, Financial Reporting and Policies Division is responsible for the Agency's payment requisition monitoring system, subject to internal policies and directives. Hence, we expected to find an effective system including the following main elements:

  • a sampling plan reviewed and approved annually and on a need basis by the CFO;
  • a sampling plan embodying the various transaction categories and the appropriate risk level and degree of significance;
  • a sampling plan whereby transactions in each transaction category could be selected; and
  • a sampling plan whereby all high-risk payment requisitions that were sensitive or for large amounts could be identified for account verification purposes.

In general, management did draw up a sampling plan whereby higher-risk transactions could be identified for account verification purposes. Indeed, our exam revealed that most payment requisitions had been coded high-risk.

However, we would like to point out certain findings that require management attention.

Sampling Plan

The sampling plan, including the sampling parameters (i.e. an acceptable level of confidence and a 10% maximum acceptable error rate (MAER)) should be updated and approved, annually and on a need basis, by the CFO.

Our discussions revealed that the annual sampling plan update by the CFO had not been done since April 2003.

Recommendation
  1. The Finance Directorate should review the sampling plan annually and on a need basis, including the sampling parameters, and have it approved by the CFO.
Management Response

We agree with the recommendations and have already taken steps to rectify the situation.

Sample

For account verification purposes, the sampling plan should take into account such elements as transactions involving large sums or those involving high risk.

Our audit brought to light the lack of any sampling, for review and validation purposes, of journal vouchers and requisitions for interdepartmental settlement, as no risk level was assigned to these two transaction categories, which do involve large amounts. Table A gives some representative data.

Table A - Other Transactions

  Fiscal 2007-2008
Transaction Category Number of Transactions Processed Value
Journal vouchers Note Transactions of up to $1 million; totalling a few million.
Interdepartmental settlements 1,200 Transactions of up to $1.3 million; totalling $21 million.

Note: Since the information is not available, we estimated that the number of transactions processed might involve several hundred journal vouchers (for example, miscoding, breakdown of monthly costs, PAYEs). Journal vouchers are cross-charges.

Recommendation
  1. The Finance Directorate should review these transaction categories' degree of risk to enable proper transaction sampling for account verification purposes. However, high-risk transactions and those involving large sums must be verified individually.
Management Response

Although our sampling plan does not deal with interdepartmental settlements (ISs) or journal vouchers (JVs), we do have certain controls in place (verification of delegations for JVs and ISs over $100,000). We have not made it a priority to include these verifications in the plan, because we consider the risk associated with the transactions to be lower than for transactions involving payments to outsider suppliers. For instance, JVs are usually corrections or transfers between responsibility centers, where a manager has already signed a Section-34 authorization for payment. However, starting November 1, 2008, all JVs deemed to involve risk (change of parliamentary credit, change of ledger account affecting a ledger involving risk [eg, hospitality, ex gratia payments, etc]) and all ISs over $100,000 will be fully verified. These kinds of transactions will be added and documented in our next sampling plan.

2.3.2 Cost-Effectiveness of Existing System

We expected to find a cost-effective system including the following main elements:

  • statistical data kept;
  • periodic analytical reports, in particular on errors detected, non-compliance, and the various corrective measures to be taken regarding the errors detected in order to periodically review the sampling plan, including the sampling parameters; and
  • proper communication between Accounting Services and the managers and financial officers and clerks in the sectors, to improve the account verification process.

There is a system in place, the Central Accounting Verification System (CAVS), that allows statistical data to be kept, such as the number of errors detected.

However, we would like to point out certain findings that require management attention.

Periodic Analytical Reports

The sampling plan should include a periodic analysis of the findings, followed by recommendations on the corrective measures to be taken. Moreover, an evaluation should be done at regular intervals of the statistical sampling method chosen for each type of transaction to make sure that it is still appropriate and adequate. That evaluation should contain information on the causes of errors and provide a mechanism for correction and follow-up with stakeholders.

Our audit brought to light the lack of periodic analytical reports on the CAVS, for example a report on errors detected that could be used as a periodic review of the sampling plan, to adjust the error rate and re-evaluate the risk associated with each transaction category; this would have had an impact on the sampling volume for verification purposes. Better communication about error rates between Accounting Services and the sectors concerned ought to improve the quality of payment requisitions.

We also noted that no corrective action had been taken on the errors detected. Neither had any follow-up been done in that regard with the sectors concerned. As regards to errors detected in the area of travel expenses, an email was sent to the travellers and to the sector clerks concerned. It is up to the travellers to advise their managers if necessary.

Moreover, our audit revealed that the error rate was in excess of the MAER, particularly for travel expenses, training costs and membership fees, in spite of the corrective measures already identified in the 2007-2008 action plan on the audit of travel, conference and hospitality expenses.

Table B gives details on the payment requisitions audited and processed by transaction category, together with the error rates.

Table B - Payment Requisitions

  2007-2008
Transaction Category Risk Coded Requisitions Audited Requisitions Processed % Audited Erroneous Requisitions Error Rate
Travel within Canada (more than $2,000) High 1,261 4,006 7% 613 49%
Travel within Canada (between $1,000 and $2,000) Medium
Travel within Canada (less than $1,000) Low
Travel to the United States (more than $1,000) High
Travel to the United States (less than $1,000) Faible
International travel High
Non-public servants travel High
Hospitality High 22 13,494 28% 4 18%
Grants & contributions High 94 9 10%
Contracts High 2,831 391 14%
Conferences High 5 0 0%
Relocations High 23 3 13%
Training High 40 15 38%
Acquisition card & petty cash purchases High 69 18 26%
Membership fees & ex gratia payments High 2 1 50%
Invoices (more than $10,000) High 1,826 225 12%
Invoices (between $2,001 and $10,000) Medium
Invoices (less than $2,000) Low
Total 6,173 17,500 35% 1,279 21%

It should be noted that payment requisitions do not include salaries, journal vouchers or interdepartmental settlements.

Recommendations

The Finance Directorate should

  1. ensure that periodic analytical reports from the CAVS are done to justify revising or not revising the sampling plan, including the sampling parameters, and are also used to communicate with the staff of the sectors concerned; and
  2. ensure that there is a mechanism for the correction of errors detected and follow-up with the staff of the sectors concerned, to improve the effectiveness of the account verification process.
Management Response

We agree with the recommendations and have already taken steps to rectify the situation.

Appendix - Management Action Plan

Ref. Recommendations Responsibility Identified Details of Action Plan Timetable
Organization Function
2.3.1 Effectiveness of existing system
i. The Finance Directorate should review the sampling plan annually and on a need basis, including the sampling parameters, and have it approved by the CFO. Finance Directorate Chief Financial Officer In 2006-2007, Information Technology (IT) developed a new system CAVS to enable the accounting team to calculate erroneous transactions. The system was supposed to allow us to:
  1. calculate error rates by type of error and transaction category;
  2. produce reports to implement corrective measures and follow-up with the parties involved in the verification process; and
  3. annually adjust the parameters of our sampling plan and have it approved by the CFO.
  4. The reports did not allow us to do those things. The error definitions were not explicit enough and the errors were not calculated correctly.

As of September 30, 2008, we had reviewed the description and classification of errors (critical, non-critical). We met with the head of the CAVS at IT to correct the problems identified above, which will allow us to perform activities 2. and 3.

Beginning in fiscal year 08-09 and at the end of each subsequent fiscal year, we will evaluate the sampling plan and adjust the parameters as needed. This plan will be approved by the CFO as of June 30 of each year to enable us to consider all of the transactions from the previous fiscal year.

June 30, 2009 (Approval by the CFO and Implement-Ation of the New Sampling Plan)
ii. The Finance Directorate should review these transaction categories' degree of risk to enable proper transaction sampling for account verification purposes. However, high-risk transactions and those involving large sums must be verified individually. Finance Directorate Chief Financial Officer Our sampling plan already takes large-sum and high-risk transactions into consideration.

As for JVs and ISs, here is what we intend to do:

All JVs deemed to involve risk (change of parliamentary credit, change of ledger accounts affecting a ledger involving risk [eg, hospitality, ex gratia payments, etc]) and all ISs over $100,000 will be fully verified. For fiscal year 2008-2009, those transactions will be analyzed for a five-month period, ie, from November 1 to March 31.

November 1, 2008
We will then review the sampling plan for the transactions described above and for the other kinds of transactions already included in the plan at the intervals indicated in recommendation 2.3.1 i. of the action plan. June 30, 2009
2.3.2 Cost-effectiveness of existing system
i. The Finance Directorate should ensure that periodic analytical reports from the CAVS are done to justify revising or not revising the sampling plan, including the sampling parameters, and are also used to communicate with the staff of the sectors concerned. Finance Directorate Chief Financial Officer In 2006-2007, IT developed a new system CAVS to enable the accounting team to calculate erroneous transactions. The system was supposed to allow us to:
  1. calculate error rates by type of error and transaction category;
  2. produce reports to implement corrective measures and follow-up with the parties involved in the verification process; and
  3. annually adjust the parameters of our sampling plan and have it approved by the CFO.

The reports did not allow us to do those things. The error definitions were not explicit enough and the errors were not calculated correctly.

As of September 30, 2008, we had reviewed the description and classification of errors (critical, non-critical). We met with the head of the CAVS at IT to correct the problems identified above, which will allow us to perform activities 2. and 3.

a. IT will modify our CAVS by December 31, 2008;

Dec 31, 2008
b. Accounting will accumulate data in the CAVS database from January 1, 2009, until the end of fiscal year 2008-2009; May 31, 2009
c. We will communicate the results of the analysis (rate of error by transaction type) to the Sector Financial Operations Division and take corrective measures to reduce the error rate (notice of information or training); and June 30, 2009
d. The data accumulated in b. will serve as a starting point for reviewing our sampling plan in 2.3.1 i. June 30, 2009
ii. The Finance Directorate should ensure that there is a mechanism for the correction of the errors detected and follow-up with the staff of the sectors concerned, to improve the effectiveness of the account verification process. Finance Directorate Chief Financial Officer Beginning in 2008-2009 and in subsequent years, we will produce reports on a quarterly basis, ie, as of June 30, September 30, December 31 and May 31 (at the end of P12-2).

We will communicate the results of the analysis (error rate by transaction type) in the month following the publication of the report to the Sector Financial Operations Division and take corrective measures to reduce the error rate (notice of information or training).

1st report for fiscal year 2009-2010 as of July 31, 2009