Audit of Equalization Certificates

Audit Report

April 22, 2013
Project Number: 80590-75

Executive summary

The equalization program has been in place since 1957 and is embedded in Section 36(2) of The Constitution Act (1982). The statutory authority for equalization is found in the Federal-Provincial Fiscal Arrangements Act (FPFAA) and regulations. Equalization has changed significantly since the original implementation and continues to undergo change during periodic reviews. The next review of the Act and regulations is scheduled to take place in 2013.

The equalization program examines, on a per capita basis, a province's ability to generate its own revenue compared to the fiscal capacity for all provinces. In addition to equalization for provinces, there is also Territorial Formula Financing, which is an annual unconditional transfer to the three territorial governments. The territorial financing enables the territories to provide residents with a range of public services comparable to those offered by provincial governments, at comparable levels of taxation and these data are contained on the equalization certificates.

Statistics Canada's role in equalization is outlined in the FPFAA and regulations. It is Statistics Canada's responsibility to prepare the revenue base and revenue source certificates. These certificates are used by the Department of Finance to calculate provincial equalization payments and territorial funding payments. In 2012, approximately 18.5 billion dollars were distributed to receiving provinces and territories under the provincial and territorial transfer programs.

The objectives of the audit were to provide the Chief Statistician (CS) and the Departmental Audit Committee (DAC) with assurance that Statistics Canada has the appropriate tools, people, resources, and quality assurance mechanisms necessary to develop accurate and complete equalization certificates which comply with the Federal-Provincial Fiscal Arrangements Act.

Key findings

The equalization certificate program is a small but complex undertaking for Statistics Canada. Equalization certificates require over 100 data series from eleven divisions within the department. In 2012, National Accounts Integration and Development Division (NAIDD) was delegated the responsibility for coordinating and preparing the equalization certificates. Within NAIDD and the data supplying divisions, roles and responsibilities for those preparing the information are clearly communicated and understood.

While the processes and tools in place with NAIDD are effective and help to ensure risks to the quality and timeliness of the certificates are minimized, the audit revealed that there are opportunities to strengthen the certificate framework environment by ensuring that systems access permissions within NAIDD are periodically monitored to ensure access is provided on a "need-to-know" basis. Additionally, NAIDD should actively monitor the operating environment within Statistics Canada to assess and respond to risks external to the division that may impact the equalization certificates. 

Within NAIDD, effective data quality control points have been integrated into the certificate preparation business process. The audit noted that the quality management framework developed by NAIDD could be improved by ensuring validation of the formulas in the final certificates takes place and configuring the templates to protect the data to ensure that data and formulas cannot be inadvertently overwritten.

Five of the six data supplying divisions (DSDs) examined in the audit have effective procedures documents that have been tailored to meet the needs of the specific programs as they respond to the requirements of the equalization program and evidence of quality review was found. However, in one DSD examined, the audit found no process documentation in place for the preparation and quality review of the data. The audit noted that this division has been subject to data anomalies in past submissions. The implementation of processes and procedures documentation for the preparation, quality review and sign-off of data submissions would reduce the risk of future data anomalies.

The majority of data supplying divisions (DSDs) examined had effective quality management practices in place to ensure the appropriate validation information on the divisional templates. Assessment of internal quality review, program director challenge and sign-off indicated that the framework could be strengthened by ensuring these processes are formalized and evidence of quality review and challenge is documented. Additionally, DSDs should ensure that segregation of duties is in place between data preparation and data validation, and that contingency plans are in place to ensure DSD submissions can be accurately completed.

The certificate preparation cycle takes place over several months and divisional data requests are timed to correspond with specific survey data releases. Although the certificate process is designed to ensure data provided are final, the audit noted that inconsistencies in the reporting of data revisions to NAIDD during the annual certificate process underscore that there is some uncertainty within DSDs related to the definitions and reporting requirements outlined in the Directive on Corrections to Daily Releases and Statistical Products. This elevates the risk that data on the certificates may not reflect the most accurate and up-to-date information. The audit team found that opportunities exist to strengthen the quality management practices by ensuring that DSDs develop procedures to advise NAIDD when data revisions or corrections occur to the data supplied on the divisional templates during the equalization certificate cycle.

Overall conclusion

The audit determined that the equalization certificates prepared by Statistics Canada in 2011 and 2012 were accurate and the tools, people, resources and quality assurance mechanisms in place support the completion of certificates in accordance with the FPFAA. Although the audit found minor data anomalies on the source and base certificates prepared in 2011 and 2012, these discrepancies did not impact equalization information for the provinces or territories for the years reviewed.

The certificate framework and quality management controls in place within NAIDD are generally effective; however, opportunities exist to strengthen these control frameworks within the certificate preparing division.

The certificate and quality management frameworks within DSDs could be enhanced by ensuring that within all DSDs process documents are developed and strengthening and documenting the quality assurance mechanisms in place.

Conformance with professional standards

The conduct of this engagement conforms with the Internal Auditing Standards for the Government of Canada, as supported by the results of the Quality Assurance and Improvement Program. Sufficient testing was carried to support the findings and related recommendations.

Patrice Prud'homme
Chief Audit Executive

Introduction

Background

The equalization program has been in place since 1957 and is part of the federal-provincial fiscal arrangements legislation. Equalization has changed significantly since the original implementation and continues to undergo periodic review. The next review of the Federal-Provincial Fiscal Arrangements Act (FPFAA) and regulations is scheduled to take place in 2013.

In general, equalization is a federal transfer payment program that has been designed to assess the fiscal capacity of the provinces and smooth out disparities. It is embedded in Section 36(2) of The Constitution Act (1982) which states:

Parliament and the Government of Canada are committed to the principle of making equalization payments to ensure that provincial governments have sufficient revenues to provide reasonably comparable levels of public services at reasonably comparable levels of taxation.

The equalization program examines, on a per capita basis, a province's ability to generate own source revenues to the fiscal capacity for all provinces. In addition to equalization for the provinces, the FPFAA includes Territorial Formula Financing, an annual unconditional transfer to the three territorial governments. Territorial financing enables the territories to provide public services comparable to those provided by provinces while taking into consideration the higher costs of providing programs and services in the territories.

The statutory authority for equalization and territorial financing is found in the Federal-Provincial Fiscal Arrangements Act and regulations. This Act outlines the role of the Chief Statistician as it pertains to fiscal equalization payments to provinces and territorial financing. Statistics Canada is responsible for preparing the revenue base and revenue source certificates. The regulations associated with the Act provide specific interpretation and outline data requirements and calculations.

The equalization certificate program is a small but complex undertaking for Statistics Canada; over 100 data series are collected from eleven divisions to prepare the equalization certificates. Reviews of data are completed by the eleven data supplying divisions, the certificate preparing division, Provincial and Territorial Focal Points, and the Department of Finance. The equalization program has a large impact on the country: in 2012, approximately 18.5 billion dollars were transferred to provinces under the FPFAA.

Until 2012, Public Sector Statistics Division (PSSD) within the National Accounts and Analytical Studies Field compiled the legislated input required for the administration of the FPFAA. Beginning in 2012, responsibility for certificate preparation was transferred to the National Accounts Integration and Development Division (NAIDD) within the National Accounts assumed responsibility for the preparation of the certificates for Statistics Canada. Effective April 1, 2013, NAIDD was subsumed into Industry Accounts and Expenditure Division and moved into the Economics Statistics Field.

Audit objectives

The objectives of the audit were to provide the Chief Statistician (CS) and the Departmental Audit Committee (DAC) with assurance that:

  • Statistics Canada has the tools, people and other resources necessary to develop timely, accurate and complete certificates in accordance with the requirements of the Federal-Provincial Fiscal Arrangements Act.
  • Statistics Canada has appropriate quality assurance mechanisms in place over the equalization certificate program to ensure compliance with the requirements of the Federal-Provincial Fiscal Arrangements Act.

Scope

The scope of this audit included an examination of the adequacy and effectiveness of the controls in place to prepare the equalization certificates. Specific areas examined included the preparation and quality assurance controls for the data and actual certificates, and examination of the adequacy and effectiveness of controls in place to ensure compliance with the data required by the FPFAA regulations. The audit covered statistical information for certificates prepared in 2011 and 2012.

Approach and methodology

The audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada, which includes the Institute of Internal Auditors (IIA) International Standards for the Professional Practice of Internal Auditing.

The audit work consisted of an examination of documents, interviews with key Senior Management and personnel, and a review for compliance with relevant policies and guidelines.

The methodology consisted of a review of applicable policies, procedures, and information related to quality, accuracy and sign-off of the certificates and source data. Testing of a sample of quality assurance mechanisms in place as they relate to the certificates and data submissions also took place. Audit testing included: validation to ensure that data supplied on the certificates matched current equalization regulations on the source and base certificates; confirming that all formulas on finalized source and base certificates were equivalent for each province and territory; ensuring that final certificate data matched data supplied by DSDs; validating that formulas within spreadsheets that were linked to equalization certificates were coherent and accurate; and validating that English and French versions of the certificates were equivalent, and units of measure were accurately denoted. The audit also examined evidence confirming that quality assurance activities took place within DSDs and NAIDD.

Authority

The audit was conducted under the authority of Statistics Canada Integrated Risk-Based Audit & Evaluation Plan 2012/13 to 2016/17, approved April 2012, by the Departmental Audit Committee.

Findings, recommendations and management response

Objective No. 1: Statistics Canada has the tools, people and other resources necessary to develop timely, accurate and complete certificates in accordance with the requirements of the Federal-Provincial Fiscal Arrangements Act.

Certificate framework

Certificate Preparing Division:
Within NAIDD, roles and responsibilities for staff coordinating the certificate preparation are clear and communicated. Adequate and effective procedures documents and tools are in place to support preparation of the certificates. However, internal systems access permissions should be periodically monitored to ensure access is provided on a "need-to-know" basis.

Extensive changes related to the transfer of the certificate program and loss of experience and knowledge across the organization elevates the risk to the quality and timeliness of the equalization certificates. NAIDD should actively monitor the operating environment to assess and respond to risks external to the division that may impact the equalization certificates.

Data Supplying Divisions:
Roles and responsibilities for staff in the divisions supplying data for the certificates are clear and communicated.

The majority of DSDs examined had effective procedures documents in place to fulfill the requirements of the equalization certificate program. However, opportunities exist to strengthen tools in place by ensuring processes and procedures for the preparation, quality review and sign-off of data submissions are clearly documented.

An effective certification framework should ensure that appropriate processes and procedures are in place so that individuals preparing and validating the information have the tools necessary to develop quality equalization certificates. This includes detailed process documents outlining how data should be prepared and what data validation is required to ensure the quality of the information. Roles and responsibilities should be clearly defined and communicated and the business and operating environments should be monitored for change. Controls in place should ensure files are restricted to authorized users only.

Certificate framework: process overview

Equalization certificates are developed based on over 100 data series received from eleven divisions within Statistics Canada. National Accounts Integration and Development Division (NAIDD) has been delegated responsibility for the coordination and preparation of the equalization certificates, which are comprised of revenue base, revenue source and population certificates.

NAIDD requests data from data supplying divisions (DSDs) using structured templates specifying the data to be provided. Requests are timed to follow the official release of data. Staff in DSDs prepare the required information, carry out quality assurance over the information prepared, and obtain program director sign-off on the completed template.

Information transmitted back to NAIDD is compiled into the certificates and reviewed. NAIDD performs basic quality checks over the information provided by DSDs and coordinates Work-in-Progress (WIP) reviews with Provincial/Territorial Focal Points and the Department of Finance. Once reviewed and finalized, the certificates are sent to the Director General of NAIDD and the Assistant Chief Statistician for review, and subsequently to the Chief Statistician for sign-off. Final signed certificates are sent to the Department of Finance, where provincial equalization and territorial funding payments are calculated using these data.

Figure 1: Certificate Preparation Process

Figure 1: Certificate Preparation Process

Description for Figure 1: Certificate Preparation Process

Certificate framework: Certificate Preparing Division

Roles and responsibilities

The audit found that within NAIDD roles and responsibilities have been clearly defined and communicated. The chief in the equalization section of NAIDD is the main point-of-contact for the Department of Finance, Provincial/Territorial Focal Points, and DSDs. The economist in the section is responsible for the coordination and preparation of the certificates, requesting, compiling and validating information received from DSDs, and liaising with DSDs when questions arise.

Clearly defined roles and responsibilities are in place within the equalization certificate program of NAIDD to support the effective management of quality for the certificates. Tasks and responsibilities for the preparation, validation and review of the certificates are complied with by NAIDD staff.

Procedures and tools

A detailed and comprehensive document entitled, "Equalization and Territorial Formula Financing" is in place within NAIDD. This document was developed as a knowledge transfer tool, when NAIDD learned that they were to take over preparation of the equalization certificates from Public Sector Statistics Division. The document outlines the history behind equalization and details the data required for the certificates. The document includes detailed timelines for requesting, compiling and validating data including the WIP reviews undertaken with Provincial and Territorial Focal Points and the Department of Finance. This document is an effective resource for employees working on the equalization certificate program.

The certificates are created and prepared using Microsoft Excel. Each row of the certificate data provides a reference to the associated FPFAA regulations, the name of the data and the units of measure for each data point (e.g. kilowatts, tonnes, dollars etc.). In both the revenue source and revenue base certificates workbooks are several linked worksheets which contain the data from finalized divisional templates. Using a linked workbook for the certificates ensures that certificate preparers do not have to transcribe data and can easily refer to data supplied from DSDs to validate and update information.

Monitoring the risk environment

Within NAIDD, risks are identified at the divisional level and risks specific to the timeliness and quality of the equalization certificates are not specifically or formally defined in the risk register. Interviews noted that risks specific to the equalization program have been informally identified and are managed through the business processes, including the completion of a quality review and following a detailed schedule to ensure the certificates are finalized by the date required by the Department of Finance.

NAIDD is dependent on DSDs to supply and provide comprehensive quality assurance on all of the data contained in the equalization certificates. Issues related to the quality and timeliness of supplying data will impact the information contained in the certificates and impact the equalization program. The divisional risk register notes this dependence as a risk to the relevance, quality, accuracy and timeliness of the divisional programs however no formal assessment, mitigation or monitoring of this risk are in place within NAIDD. As a result, NAIDD should actively monitor the operating environment across the organization to assess and respond to the risks external to the division that may impact the equalization certificates. For example, the audit noted that as a result of significant staffing changes across the organization (including change in responsibility for certificate preparation) there has been a substantial loss of knowledge and experience related to the equalization program. In 2012, staff preparing the certificates were new to the process as were staff preparing the divisional templates in 3 of the six data supplying divisions examined during the audit. This loss of experience and knowledge elevates the risks to the quality and/or timeliness of the certificate information, as employees may not understand the importance of the information they provide, or may not be aware that data requirements can change.

Systems access controls

NAIDD has effective electronic access controls in place for the external transmission of certificate information to authorized recipients. Access to draft certificates for Provincial/Territorial Focal Point and Department of Finance WIP reviews is restricted to designated officials only. An electronic File Transfer System (e-FTS) has been implemented with access authentication procedures and mechanisms in place to ensure safe transmittal of draft certificate information between NAIDD and authorized external parties.

Within NAIDD, information (data and production information) for the equalization certificates are stored on a Solaris Server within a designated folder. Access to the equalization folder is granted to those working in the Harmonized Sales Tax (HST) secretariat, but access permissions are not monitored to ensure access to authorized individuals only. The audit found that of the eleven individuals in the group with access permission, one no longer worked in the division, two worked in NAIDD but not on the equalization certificates, and one no longer worked at Statistics Canada. Opportunities exist to strengthen system access permissions through regular monitoring and ensuring that access is limited to employees on a "need-to-know" basis as per the Statistics Canada Security Practices Manual.

Certificate framework: Data Supplying Divisions

The audit examined six of the eleven divisions which provide data for the equalization certificates. Divisions were chosen to ensure coverage of all fields that provide data to the certificate program. Additionally, divisions in which data anomalies had been detected in past submissions were included.

Roles and responsibilities

The audit determined that roles and responsibilities as they relate to the preparation and quality assurance of the data prepared on the divisional templates within the 6 DSDs examined are well defined and communicated. Data for the divisional templates are prepared by economists, analysts and/or client services staff. In all 6 DSDs examined, individuals involved in the equalization program understand they are responsible for the preparation and accuracy of the information. Quality checks are completed prior to program director review.

Within DSDs, directors are the delegated authority for sign-off on the completed divisional templates. They understand that they are responsible for the quality of the information originating in their divisions. As such, their review is designed to provide a final challenge of the information prior to being sent to NAIDD to be incorporated into the certificates.

The audit noted that DSDs have clearly defined roles and responsibilities in place to support the effective management of quality of the data supplied to the equalization certificates.

Procedures and tools

The audit team examined all procedures documents and tools related to the equalization certificates within the six DSDs included in the audit. The audit noted that documentation is in place in five of the six divisions examined and that processes and procedures varied between these divisions. For divisions which require special tabulations for the templates, process documents outline the files to use, the statistical programming files and software required for the correct code for the tabulations, and the appropriate quality assurance activities to check the data to ensure the data are correct. In divisions in which the data can be retrieved directly from CANSIM tables, the documentation notes the applicable CANSIM table the data should be retrieved from, what to check to ensure the data are correct and how to prepare the templates. While the audit noted that the procedures and tools differ between the five divisions, each was tailored to meet the needs of the specific program as they respond to the requirements of the equalization program and were effective tools to ensure the quality of the information.

The audit revealed that in one of the DSDs examined, there was no process documentation in place for the preparation, quality assurance or program director sign-off of data prepared for the template. The person coordinating the submission for this division was new to the process in 2012 and having documentation that outlines the steps for data retrieval and validation would help ensure the all the required steps were completed prior to director sign-off. Additionally, because this division has been subject to data anomalies in past, the implementation of data preparation and quality review requirements documentation would reduce the risk of future data anomalies within these data submissions.

The majority of DSDs examined had effective processes and procedures documents in place to fulfill the requirements of the equalization certificate program. However, opportunities exist to strengthen the quality of equalization data coming from DSDs through the implementation of procedures documents that outline the data to be produced, the methodology to be followed, as well as required quality assurance practices required within divisions that do not have these tools in place.

Recommendations:

It is recommended that the Assistant Chief Statistician of Economic Statistics should ensure that:

  • Monitoring of systems access permissions is carried out on a periodic basis and updated as required;
  • Certificate preparing staff actively monitors business conditions external to the division on a scheduled basis to ensure that risks to the quality of equalization data are identified, assessed and mitigated.

Management response:

Management agrees with the recommendations.

  • The Assistant Director of the HST-EQ Secretariat within NEAD will ensure that regular monitoring of systems access is implemented.

    Deliverables and Timeline: Quarterly report on access rights to designated folder is prepared. This will be completed immediately.
  • The Assistant Director of the HST-EQ Secretariat within NEAD will ensure that a process is in place to monitor risks related to the quality of equalization data.

    Deliverables and Timeline: Develop and deliver a presentation on the process for preparing Equalization certificate and conduct bi-annual consultations with Data Supplying Divisions (DSDs) and Finance Canada, with a subsequent bi-annual report on business conditions affecting the Equalization certificate data. The presentation will be developed and delivered to DSDs prior to the end of the 2013-2014 fiscal year-end. Bi-annual consultations and reporting will begin during the 2013-2014 fiscal year.

Recommendation:

It is recommended that the Assistant Chief Statistician of the Economic Statistics Field and the Assistant Chief Statistician of the Social, Health and Labour Statistics Field should ensure that:

  • Processes and procedures for preparation, quality assurance and sign-off of data submissions are formally documented.

Management response:

Management agrees with the recommendations.

  • The Directors within DSDs will establish standardized processes for preparation, quality assurance and sign-off that are well documented.

    Deliverables and Timeline: Best practices and processes will be identified and standardized across DSDs. Processes and procedures, including quality assurance and sign-off will be documented and implemented. Identification and standardization will be completed by September 2013. Processes and procedures, quality assurance and sign-off within DSDs will be documented and implemented by March 2014.

Objective No. 2: Statistics Canada has appropriate quality assurance mechanisms in place related to the equalization certificate program to ensure compliance with the requirements of the Federal-Provincial Fiscal Arrangements Act.

Quality management

Certificate Preparation Division:
Within NAIDD, effective data quality control points have been integrated into the certificate preparation business process and data provided in the certificates are generally complete, coherent and accurate. However, opportunities exist to strengthen quality management practices by ensuring that validation of formulas within the certificates takes place and ensuring certificates are configured to protect the data and ensure formulas and data cannot be inadvertently overwritten.

Data produced on the revenue base and revenue source certificates correspond to the data specified in the FPFAA regulations.

Data Supplying Divisions:
Data contained in divisional submissions are approved by those with appropriate authority.

The majority of the DSDs examined had effective quality management practices in place to ensure the appropriate validation information on the divisional templates. Assessment of internal quality review, program director challenge and sign-off indicated that the framework could be strengthened by ensuring these processes are formalized and evidence of quality review and challenge is documented in all DSDs.

Quality management practices can be enhanced within DSDs by ensuring the segregation of duties between data preparation and data validation, and contingency plans are in place to ensure DSD submissions can be accurately completed.

Quality management practices within DSDs could be strengthened with the development of procedures to advise NAIDD when data revisions or corrections occur to the data supplied on the divisional templates during the equalization certificate cycle.

The quality management framework for the equalization program should include documented guidance for quality review and approval activities, clear segregation of duties between data preparation and quality review, and formal review and challenge prior to approval and sign-off on data quality. Quality management processes should be integrated into the business practices in place for the preparation of equalization certificates. The Statistics Canada Quality Guidelines note that certification or validation of statistical information should include a challenge of the data rather than a rationalization of the information. This challenge should ensure that data movements, unusual events or uneven growth are understood and can be explained.

Quality management framework: Certificate Preparing Division

The audit found that the data quality control points within NAIDD are effective and have been integrated into the certificate preparation business process. The certificate preparing division typically requests data that have been released in the Daily. Requesting finalized data ensures that the information have been subject to internal quality assurance practices within each DSD. When requesting the data, the NAIDD provides structured templates to each data supplying division. Templates specify the data to be provided, and include a section for DSDs to note methodological, classification or other changes that may impact the data. The audit confirmed that these templates are used by data supplying divisions.

Upon receipt of completed templates, NAIDD performs a number of basic validity checks, including checking against previous submissions, previous year data (growth analysis), and where applicable, validation against CANSIM, other publications and a check of totals. A quality review checklist is in place to support this activity. Unusual movements, variations and other issues are queried with the data supplying division and documented.

The certificate preparing division coordinates three Work-in-Progress reviews: one review with Provincial/Territorial Focal Points, and two reviews with the Department of Finance. Focal Points review provincial and territorial data to be included in the revenue source certificate and the Department of Finance reviews point-in-time versions of the revenue base and revenue source certificates in September and November. These reviews provide a check of coherence and reasonableness. The audit found evidence of items raised through WIP reviews including data challenges, requests for explanations and notices of updates for missing information being queried with DSDs, and responses documented.

Finalized templates are sent by the Assistant Chief Statistician, System of National Accounts Branch to the Office of the Chief Statistician (OCS) for signature. The audit confirmed that the OCS relies upon the data reviews completed within NAIDD and DSDs. This places further importance on the practices put in place by NAIDD and DSDs to mitigate potential risks to the quality of information provided by Statistics Canada to meet the organization's obligation with respect to the FPFAA and regulations.

The audit conducted testing to confirm that the data being supplied by Statistics Canada for the revenue base and revenue source certificates correspond to the data series specified in the FPFAA regulations. The audit tested each reference made to the regulations in the 2011/2012 revenue base certificate template, and in the provincial tabs supporting the 2009/2010 revenue source certificate templates. Each reference made in the templates was validated against the corresponding section of the regulations.

The audit examined the data provided in the base and source certificates for coherence, completeness and accuracy. The audit tested and found that the data on the final certificates matched the DSD templates. The audit also confirmed that the data found in the French and English versions of the certificates are equivalent, units of measure (e.g. kilowatts, tonnes, dollars etc.), and table stub blocks, header blocks and notes blocks have been correctly denoted.

The finalized certificates are linked to specific worksheets. These worksheets contain the data and specific calculations based upon the regulations. The audit tested the calculations on the base and revenue source certificates prepared in 2011 and 2012 and found that the computations on the linked data sheets were coherent and accurate, and that provincial and territorial formulas were equivalent.

The audit found that NAIDD does not validate that the formulas in each cell of the equalization certificates are referenced to the appropriate linked source data sheet or that they have not been overwritten which increases the risks to the quality of the information. Audit testing of the formulas on the revenue base and revenue source certificates prepared in 2011 and 2012 revealed instances of incorrect references and overwritten formulas on the finalized equalization certificates. While data formula and data anomalies were found on the final certificates, the audit noted these inconsistencies did not impact final figures for the years examined.

Overall, NAIDD has implemented effective quality control points within the certificate preparation process to ensure the quality of the information. Opportunities exist to strengthen the quality management practices by ensuring that validation of the formulas within the certificates takes place and ensuring the certificates are configured to protect the data and ensure formulas and data cannot be inadvertently overwritten.

Quality management framework: Data Supplying Divisions

The Statistics Canada Quality Guidelines notes that certification or validation of statistical information should challenge data rather than rationalize the information to help ensure information quality. Additionally, the Guidelines recommend that validation/certification should be carried out by analysts not involved in the production of the data. The audit revealed that the quality management practices followed within DSDs for preparation, validation, and sign-off varied. In five of the six DSDs examined, segregation between data preparation and data validation is in place. In one division in which data anomalies occurred in the 2011 submission, processes rely on a single person for both data retrieval and validation and only one individual in the division has the requisite knowledge to provide the information. There is no contingency plan in the event of an absence. This person-dependent process exposes the data to potential quality or timeliness risks.

Data supplying divisions are responsible for the quality of the information they provide. In five of the six divisions, the audit found evidence that data validation takes place and is documented. Evidence of validation consisted of spreadsheet comparisons to previous years, documented explanations for data large movements, validation against other data sets or provinces, and checks of totals. In the division without process documents there was no documented evidence of data validation occurring.

Program directors are required to sign the divisional template as acknowledgement of their approval of the information being submitted by their division. In 5 of the 6 DSDs examined, the management review of divisional templates is informal and undocumented, but management noted that prior to sign-off they must have assurance that the data are of sufficient quality. This informal review process varied: in some cases the review was general, whereby directors asked only if quality assurance took place (no specific challenge) prior to sign-off, while others provided a more detailed challenge function which included confirming what types of checks were done and how the data differed from previous years prior to providing sign-off.

In one of the six divisions examined, the management review is formalized. A formal presentation is made to the program director, outlining the data, major changes, largest contributors and reasons for large variations. The program director provides a challenge function over the information presented. Queries made by the program director during the challenge are documented, and the analyst must subsequently respond in writing to these queries. When the director is satisfied with the explanations, the templates are signed. The audit found that signatures were in place on all divisional templates in 2011 and 2012 for the six DSDs examined in the audit.

The audit found that the majority of the DSDs examined had effective quality management practices in place to ensure the appropriate validation information on the divisional templates. Assessment of internal quality review, program director challenge and sign-off indicated that the framework could be strengthened by ensuring these processes are formalized and evidence of quality review and challenge is documented. Additionally, DSDs should ensure that segregation of duties is in place between data preparation and data validation, and that contingency plans are in place to ensure DSD submissions can be accurately completed.

Reporting of corrections or revisions

The Directive on Corrections to Daily Releases and Statistical Products has been developed to:

"ensure that a standard and consistent approach is applied for the identification, reporting, documentation, approval, and posting of post-release corrections and unplanned revisions."

The Directive defines corrections, revisions and unplanned revisions as the following:

  • A correction is "the action of eliminating an error to data, text or metadata which was introduced by an error in the design, implementation or execution of a statistical process."
  • A revision is defined as, "a planned statistical activity whereby preliminary data are recompiled when more and better source data become available, or when previously released data are recompiled following the introduction of new classifications, frameworks and methodologies." 
  • Finally, an unplanned revision is considered as, "the action of recompiling previously released data in response to an event external to the statistical process and planned statistical activities, and results in significant improvement in the quality of published data or analysis." 

The Directive notes that unplanned revisions should be treated with the same due diligence and formal treatment as corrections. The policy notes that unplanned revisions are to be communicated to the senior management responsible for the subject-matter, and that stakeholders should be proactively informed of data updates.

The audit noted that among the DSDs examined during the 2012 certificate process, there were two cases in which data were revised and divisional templates had to be re-signed by the program directors and resubmitted to NAIDD. Interviews and documentation review by the audit team found that in one instance, the revision was defined as "unplanned" and followed the process described in the Directive with a note from the director to the OCS outlining the impact of the revision to the data as well as the impact of the revision to the data provided for the equalization certificate. In the second case, although the revision was not planned, the division did not feel it met the definition of an unplanned revision and therefore did not follow the procedures laid out in the Directive. As a result, it was not reported and was identified only through the quality review completed within NAIDD. Inconsistencies in the reporting of data revisions to NAIDD during the annual certificate process underscore that there is some uncertainty within DSDs with respect to the definitions and reporting requirements outlined in the Directive. This elevates the risk that data on the certificates may not reflect the most accurate and up-to-date information.

The audit team found that quality management practices could be enhanced by ensuring that DSDs develop procedures to advise NAIDD when data revisions or corrections occur to the data supplied on the divisional templates during the equalization certificate cycle.

Recommendation:

It is recommended that the Assistant Chief Statistician of Economic Statistics sh ould ensure that:

  • Equalization templates are validated and configured to prevent inadvertent changes to the data and formulas.

Management response:

Management agrees with the recommendations.

  • The Assistant Director of the HST-EQ Secretariat within NEAD will implement cell protection in order to lock the formulas and data in Equalization worksheets.

    Deliverables and Timeline: Cells protected following validation of data in the Equalization worksheets. This has been completed.

Recommendations:

It is recommended that the Assistant Chief Statistician of the Economic Statistics Field and the Assistant Chief Statistician of the Social, Health and Labour Statistics Field should ensure that:

  • Process documents and practices are updated to ensure that the certificate preparing staff are notified when revisions or data changes take place during the equalization certificate cycle.
  • There is a segregation of duties between data preparation and quality assurance of data for the certificates and that contingency plans are in place to ensure DSD submissions can be accurately completed.
  • Internal quality review and program director challenge prior to sign-off on divisional templates is formalized, and evidence is in place to attest to these checks being completed.

Management response:

Management agrees with the recommendations.

  • The Directors of DSDs will ensure that notification procedures are in place when Equalization data is revised.

    Deliverables and Timeline: Notification procedures will be established by November 2013.
  • The Directors of DSDs will ensure proper separation of data preparation and quality assurance verification of certificate data and that contingency plans are in place.

    Deliverables and Timeline: Implement processes in data supplying divisions that ensure that data preparation, proper Quality Assurance and sign-off are conducted by separate parties. Ensure processes are documented and back-ups are identified and trained. These activities will be completed by March 2014.
  • The Directors of DSDs will ensure that internal quality review of data on divisional templates is documented and a formalized and documented director challenge of template data is carried out prior to Director sign-off.

    Deliverables and Timeline: DSDs will document the results of quality review checks completed prior to submitting the templates to the Director for final challenge and review. Based upon existing best practice, implementation of a formalized Director challenge function of the data for divisional templates prior to sign-off and submission. These activities will be completed by March 2014.

Appendices

Appendix A: Audit criteria

Appendix A: Table 1 Audit criteria
Table summary
The table in Appendix A identifies the Audit Criteria, audit sub-criteria as well as the policy instrument used as the source of these criteria.
Objective / Core Controls / Criteria Sub-Criteria Policy Instrument
1) Statistics Canada has the tools, people and other resources necessary to develop timely, accurate and complete certificates in accordance with the requirements of the Federal-Provincial Fiscal Arrangements Act.
THEME: Certification Framework
CRITERIA: People

1.1 Statistics Canada provides employees with the necessary training, tools, resources and information to support the discharge of their responsibilities.
1.1.1 Employees have access to sufficient tools, such as software, equipment, work methodologies and standard operating procedures. Management Accountability Framework
1.1.2 Processes exist to identify change requirements impacting the equalization certificates, and update tools (guidelines, procedures, templates) on a timely basis.
1.1.3 Statistics Canada identifies resource requirements for the production of equalization certificates and analyses them against existing human resource competencies and capacities.
1.1.4 HR planning exists to address any gaps from the above analysis, and to ensure succession planning.
Accountability

1.2 Authority, responsibility and accountability for the equalization certificate program are clear and communicated.
1.2.1 Roles and responsibilities for data provision, data quality, preparation of certificates, and approval of data are defined and communicated. Management Accountability Framework
Stewardship

1.3 System application controls exist.
1.3.1 Application controls are in place to limit unauthorized access and potential for inappropriate changes. Statistics Canada Security Practices Manual
1.3.2 Procedures exist to ensure that application controls are effective.
Risk Management

2.3 Risk management is applied to the equalization certificate program.
2.3.1 Risks to the timeliness and quality of the certificates are identified, assessed, responded to, and monitored. Management Accountability Framework
2) Statistics Canada has appropriate quality assurance mechanisms in place related to the equalization certificate program to ensure compliance with the requirements of the Federal-Provincial Fiscal Arrangements Act.
THEME: Quality Management
Stewardship

2.1 Quality Assurance reviews take place for certificate data.
2.1.1 Activities needed to ensure data quality are documented and have been integrated into the process for the production of certificate data.

Statistics Canada Quality Guidelines

Federal-Provincial Fiscal Arrangements Act

Directive on Corrections to Daily Releases and Statistical Products

2.1.2 Quality of the certificate data is monitored.
2.1.3 Quality control breakdowns are reported to management.
Accountability

2.2 Equalization certificates are reviewed and approved.
2.2.1 Reporting is reviewed for quality (completeness, accuracy, relevance, timeliness, appropriateness, and reasonableness).

Statistics Canada Quality Guidelines

Federal-Provincial Fiscal Arrangements Act
2.2.2 Approval is evidenced (signoff, email, minutes, etc).

Appendix B: Acronyms

Appendix B: Acronyms
Acronym Description
ACS Assistant Chief Statistician
CS Chief Statistician
DAC Departmental Audit Committee
DSD Data Supplying Division
e-FTS Electronic File Transfer System
FPFAA Federal-Provincial Fiscal Arrangements Act
HST Harmonized Sales Tax
IIA Institute of Internal Auditors
NAIDD National Accounts Integration and Development Division
OCS Office of the Chief Statistician
PDF Portable Document Format
PSSD Public Sector Statistics Division
WIP Work-in-Progress
Date modified: