Audit of Acquisition of Data from Alternative Sources

Executive summary

Statistics Canada works with all levels of government and private sector organizations in the collection and compilation of statistical information. Although the agency has been acquiring data from alternative sources for decades, Statistics Canada data collection has traditionally been rooted in the administration of surveys. The agency has acknowledged that the traditional method of collecting data presents a series of unique challenges for meeting new and ongoing data needs and reducing response burden.

As part of the agency's modernization initiative that was launched in 2017, the agency is moving towards an "administrative data first agenda." This agenda seeks to use the acquisition of data from alternative sources as the primary method to collect statistical information in order to improve the balance between data quality, response burden and costs.

For the purpose of this audit, "data from alternative sources" are defined as all data other than survey data, which includes, but is not limited to administrative data (data obtained under the Statistics Act^{Footnote 1}) as well as data available to the public.

Why is this important?

As the national statistical office, Statistics Canada must ensure it delivers relevant statistical information in an efficient and transparent manner. In today's environment, Statistics Canada like many other public sector organizations is faced with continuous change and has had to innovate and discover new ways to acquire data in order to continue to serve Canadians and fulfill its mandate.

Against this backdrop, the agency has implemented an administrative data first agenda that is aligned with its modernization initiative. Within this administrative data first paradigm, the agency seeks to respond to statistical demands and ensure that data from alternative sources are acquired in a strategic, timely and transparent manner. The audit determined the extent to which the agency has implemented effective processes and controls to support these objectives, while ensuring the stewardship of agency interests.

Key audit findings

The Data Acquisition Committee provides support on a wide-range of topics, including: strategic advice on potential partnerships, data strategies and new initiatives proposed by statistical program divisions. Interviews with committee members indicated that there is some concern regarding the committee's expanding roles and responsibilities.

Although the agency has broadly identified some of the risks to acquiring data from alternative sources through various corporate mechanisms, no full risk assessment has been completed for the acquisition of data from alternative sources. The restructuring of the agency's Tier 2 governance committees offers the agency a real opportunity to strengthen oversight on risk management activities.

The Statistics Canada Data Strategy was developed to provide a roadmap for how the agency, as part of its modernization agenda, will continue to govern data assets and key strategic data capacities. The Statistics Canada Data Strategy identifies a series of short, medium and long term business objectives for acquiring access to alternative sources of data but these objectives have not yet been aligned to specific performance indicators.

Roles and responsibilities for acquiring data from alternative sources are clearly outlined and defined within Statistics Canada's policy instruments. However, the policy instruments need to be updated to reflect some recent changes to legislative requirements under section 8(2) and 8(3) of the Statistics Act and the updated role of the Data Stewardship Division (DSD) and statistical program areas in the acquisition process.

Two training courses have been offered to assist agency employees in carrying out their responsibilities for acquiring data from alternative sources. Both courses could benefit from providing guidance on the implications of the new legislative requirements under the Statistics Act and further explaining how program managers can leverage DSD and the Office of Privacy Management and Information Coordination (OPMIC) during the acquisition process.

Data acquisition agreements are used to acquire administrative data. Publicly available data do not need to be obtained under the Statistics Act and, generally speaking, are not subject to the same requirements as administrative data.

Legal oversight is provided by the OPMIC during the data acquisition process. When a data acquisition agreement is drafted, OPMIC is contacted to perform a review of the terms and conditions within the proposed agreement prior to obtaining signatures.

When the agency incurs a financial fee for acquiring data from alternative sources, a cost proposal is submitted to the agency by the provider that identifies the total cost along with supporting financial information. However, there is limited evidence to demonstrate how management reviews these costs to determine whether they are reasonable and represent value for money.

Statistical program managers were generally not using the standard evaluation questionnaire or any other suitable tool to assess data quality and fit for use. In collaboration with the Modern Statistical Methods and Data Science Branch, DSD is coordinating a more user friendly version of the quality evaluation questionnaire.

Overall conclusion

Management has implemented a governance framework to provide oversight and advance the agency's administrative data first agenda, with corporate tools in place that support this agenda and identify some of the risks to acquiring data from alternative sources. The proposed restructuring of the agency's governance committees offers an opportunity to strengthen risk management activities for the acquisition of data from alternative sources. The aim is to provide Statistics Canada with a more effective governance structure to support the acquisition of data from alternative sources. The agency has also identified business objectives for acquiring data from alternative sources although the absence of key performance indicators could hinder the agency's ability to measure progress against its business objectives.

Internal controls are in place and functioning with some minor deficiencies, which are attributed to recent legislative changes and an internal transformation of responsibilities related to the acquisition process. Through training programs and internal communication, management has taken positive steps to inform employees but should ensure this is consistently communicated across the agency.

Conformance with professional standards

The audit was conducted in accordance with the Mandatory Procedures for Internal Auditing in the Government of Canada, which include the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing.

Sufficient and appropriate audit procedures have been conducted, and evidence has been gathered to support the accuracy of the findings and conclusions in this report, and to provide an audit level of assurance. The findings and conclusions are based on a comparison of the conditions, as they existed at the time, against pre-established audit criteria. The findings and conclusions are applicable to the entity examined, and for the scope and period covered by the audit.

Steven McRoberts
Chief Audit and Evaluation Executive

Introduction

Background

The world is constantly changing, with new technologies regularly emerging, only to become obsolete and outdated almost overnight. With this ever-changing environment comes increased pressure for governments to have 'real-time' data to inform public policy as data are the lifeblood of decision-making in the public sector.

In the public sector, data help direct and inform decision-making by providing valuable information such as how many live in poverty; whether greenhouse gas emissions are increasing; or how public money is being spent. The success of the Canadian economy, and the prosperity of its communities depends in part on advancing programs that focus on strengthening public sector decision-making in key areas such as the financial, environmental and the social fields.

Statistics Canada plays an integral role in in this endeavour, as a foundational part of the agency's mandate is to collect and compile statistical information. For decades, the agency's business has been rooted in the administration of surveys, with data from alternative sources being used to complement its data collection. Recently, the agency acknowledged that traditional surveying of Canadians and businesses has struggled, because of increasing costs and response burden.

For the purpose of this audit, "data from alternative sources" has been defined as all data other than survey data, this includes, but is not limited to, administrative data (data obtained under the Statistics Act) as well as data available to the public. Data available to the public do not need to be obtained under the Statistics Act and are not subject to the requirements of the Statistics Act.

Administrative data first agenda

As a part of the agency's modernization initiative that focuses on user-centric service delivery and leading edge data integration, Statistics Canada has committed to an administrative data first agenda. This agenda is aimed at positioning the agency to better respond to statistical demands in the ever-changing and ever-evolving modern data world. In an administrative-first paradigm, administrative data are considered first before a survey is conducted, in order to complement or replace data acquired from a survey or to evaluate its quality.

Statistics Canada has taken steps towards realizing its administrative data first agenda by formalizing broad business objectives in a number of corporate initiatives. These objectives include:

• identifying, and gaining timely access to data from alternative sources for statistical purposes;
• communicating to the public in a proactive and transparent manner why the agency seeks to acquire data from alternative sources; and
• strategically managing the acquisition of data through the effective implementation of governance and stewardship.

A period of transition

When acquiring data from alternative sources, there are a number of processes, procedures and key stakeholders that play an integrated role. The Data Stewardship Division (DSD) is responsible for the creation and implementation of sound data stewardship protocols to ensure all data assets are well managed, secure, and fit for use. The Office of Privacy Management and Information Coordination (OPMIC) provides legal oversight and is responsible for reviewing data acquisition agreements for legal implications.

The process by which the agency acquires data from alternative sources is in a period of transition as a result of shifting internal roles and responsibilities, new legal interpretations to the Statistics Act and updates to internal governance structures. In the future, these changes will impact how the agency manages its risks for acquiring data from alternative sources.

Audit objective

The objective of the audit was to provide reasonable assurance to the Chief Statistician (CS) and the Departmental Audit Committee that management has adequate processes and controls in place to support the strategic, transparent and timely acquisition of data from alternative sources while ensuring the sound stewardship of public assets and agency interests.

Scope

The audit scope included an examination of acquisitions of data from alternative sources from private sector organizations where negotiations to acquire data from alternative sources began or were completed during fiscal years 2017/2018 to 2018/2019, including acquisitions that were underway or not yet completed as at May 31, 2019. The audit focused on three key controls areas, namely: governance, tools and training and internal controls.

Statistics Canada also acquires data from alternative sources from provinces, territories and federal departments and agencies. It was determined during the planning phase of the audit that acquiring data from the public sector poses less risk to the agency and for this reason, public sector data acquisitions were excluded from the scope of the audit.

Approach and methodology

This audit was conducted in accordance with the Mandatory Procedures for Internal Auditing in the Government of Canada, which include the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing. Field work consisted of a review of applicable processes, activities and tools to ensure compliance with Statistics Canada legislative and policy requirements.

Authority

The audit was conducted under the authority of the approved Statistics Canada Integrated Risk-based Audit and Evaluation Plan 2019/2020 to 2023/2024.

Audit findings, recommendations and management response

The data acquisition process

As the national statistics office, Statistics Canada is constantly exploring ways to acquire alternative sources of data to better support its statistical programs and provide Canadians with valuable information. Statistics Canada can acquire data from alternative sources from both the public and the private sectors and has acknowledged their importance in supplementing its statistical programs through the agency's modernization initiative. The DSD is responsible for developing sound data stewardship protocols to ensure all data assets are well managed, secure, and fit for use. DSD has developed an acquisition process that identifies the steps to acquire new sources of data from alternative sources. This process ensures that all proposed data acquisitions are consistently performed using a common approach.

Data acquisitions are contingent upon a number of key factors, including: the complexity of the acquisition, the sensitivity of the data and the pre-existing relationship with the data provider. Normally, the process begins with statistical program managers identifying data from alternative sources collected and held by other organizations that can support the legal mandate of Statistics Canada. Once a need has been identified, a request is sent to DSD to determine whether the data will be for broad use or localized use.

Once the request is complete and the type of data is determined, DSD and OPMIC provide guidance to the program areas on how to proceed with the acquisition. With support from DSD, program managers establish a strategy to reach out to the data provider and initiate early discussions with key stakeholders. The next step is to begin negotiations with the data provider. This is a key point in the process when Statistics Canada meets with representatives from the data provider to discuss the data being requested, legal and privacy expectations and any associated costs. It should be noted that Statistics Canada does not pay for data but rather for the time and effort required to compile the intended data sources.

When acquiring data from alternative sources with an associated cost, it can be obtained under the Statistics Act (data sharing agreement). Data can also be acquired without the use of the Statistics Act through a contract or written communication. When the data are obtained under the Statistics Act, the statistical program area is responsible for initiating expenditure initiation under section 32 of the Financial Administration Act and performing a financial review process with support provided by the OPMIC and DSD. For data acquisitions that do not fall under the authority of the Statistics Act, Corporate Support Services Division (CSSD) is responsible for reviewing the proposed costs and determining whether the proposal should be accepted for payment. Through interviews, the audit found that CSSD's departmental delegation of financial authorities for services is $100,000 and any contract above that threshold would be sent to Public Services and Procurement Canada.

An agreement is formalized through a data acquisition agreement, contract or written communication once all aspects of the negotiations have been finalized, including any potential cost. Typically, the agency begins receiving synthetic or test data from the provider and quality assessments must be completed to determine whether the data are usable within the agency's existing data environment. If the data are deemed fit for use, statistical program managers finalize and sign the data acquisition agreement with the data provider. The OPMIC plays a key role in this step as they are responsible for ensuring that the agreement contains terms and conditions that protect the legal interests of the agency.

Through consultation with DSD, the audit found that DSD's role in the acquisition process is expected to change. The intent is for program divisions to carry out their own data acquisitions and DSD to act as an intermediary on all matters related to data acquisitions. DSD's intent is to communicate these changes and make this process visible but at the time of audit, no timeline had been established.

The data acquisition process described above is supported through a series of controls including governance committees, training courses and internal policy instruments. All play an important role in identifying, and gaining timely access to data and communicating to the public why the agency seeks to acquire for statistical purposes. The audit determined the extent to which these controls were operating and comes at a time when the agency is undergoing significant changes as it looks to advance its administrative data first agenda.

Governance mechanisms are undergoing changes that are expected to strengthen risk management.

The DAC was created in February 2018 as a result of consolidating the Administrative Data Management Committee, Collection Planning Committee and Business Response Management Committee. The DAC's mandate is to provide leadership and direction for data acquisitions with the objective of implementing and maintaining an "administrative data first agenda," The DAC is made up of 18 members that include senior managers, directors and director generals from multiple fields across the agency. The committee meets approximately once per month and provides support on a wide-range of topics, including: strategic advice on potential partnerships, data strategies and new initiatives proposed by statistical program divisions.

Interviews with committee members indicated that there is some concern regarding the committee's expanding roles and responsibilities. DAC was originally intended to be a strategic-level oversight body to provide high-level guidance on activities directly linked to acquiring data from alternative sources. However, the audit found that some of the DAC's activities are more ad-hoc in nature and not always within its intended mandate. For example, the committee was tasked with determining whether certain surveys should be made mandatory or voluntary. It is important that the DAC operate within its intended mandate to ensure that there is adequate time and resources to address key activities related specifically to the acquisition of data from alternative sources.

Moving forward, the agency has an opportunity to strengthen its governance with the proposed changes to the existing Tier-2^Footnote2 governance structure. These modifications are designed to consolidate the 10 existing Tier-2 governance committees into 5 new committees. The proposed 'Data in: Data Acquisition and Management' is one of the five new committees and will aim to provide Statistics Canada with a more effective governance structure to support the acquisition of data from alternative sources. The committee will be responsible for identifying risks and issues related to acquiring data from alternative sources and implementing mitigation strategies to ensure the agency is able to respond proactively to change and uncertainty. The new responsibilities of the proposed committee are needed as risk management oversight for acquiring data from alternative sources needs to be strengthened to enable more effective decision-making throughout the agency.

The audit confirmed that no risk assessment has been completed for the acquisition of data from alternative sources, although the agency has broadly identified some of the risks to acquiring data from alternative sources through various corporate mechanisms.

Through interviews, agency employees indicated that there might be some merit to undertaking a risk assessment. The restructuring of the agency's Tier 2 governance committees offers the agency a real opportunity to strengthen risk management activities for acquiring data from alternative sources as it will improve decision-making in governance, strategy and objective setting. However, at the time of the audit this restructuring had not yet been fully implemented.

Work is required to ensure that key performance measures are in place.

The CS has stated the importance of continuing to modernize each step of the statistical process, by modernizing and embracing an administrative data first agenda. To achieve this, Statistics Canada has embedded performance measures in corporate initiatives like the SCDS. The SCDS was developed to provide a roadmap for how the agency, as part of its modernization agenda, will continue to govern data assets and key strategic data capacities. The SCDS is organized into two main pillars: data governance and data stewardship. Under data stewardship, there are a series of 7 'strategic data capacities' that are intended to have performance measurement criteria that would track progress against specific targets.

One of the 7 strategic data capacities is: 'data discovery'. This area links directly to the administrative data first agenda and identifies a series of short, medium and long term business objectives for acquiring access to alternative sources of data. These include: outlining why the agency seeks to acquire data from alternative sources, establishing mechanisms to include community engagement and implementing processes to support the acquisition of data from multiple access points across the agency.

However, the SCDS does not contain any objectives or performance measures related to the 'timeliness' and 'speed' with which the agency acquires data from alternative sources even though through interviews, agency employees stressed the importance of gaining access to data from alternative sources in a timely manner.

Overall, although the SCDS provides short, medium and long term business objectives for acquiring data from alternative sources, these objectives have not been aligned to performance indicators. Without these indicators, the agency cannot fully track its progress against key objectives that include timeliness and transparency in order to determine whether objectives for acquiring data from alternative sources are being achieved. In discussion with DSD, measurement criteria for the acquisition of data from alternative sources, as outlined in the SCDS, will be developed with program areas and in consultation with Corporate Strategy and Management Field. It was indicated by DSD that the indicators related to the short term business objectives will be developed by early 2020.

Updates are required to policy instruments to ensure they are aligned with new legislative requirements and DSD's new role.

The Policy on the Use of Administrative Data Obtained under the Statistics Act and the Directive on Obtaining Administrative Data under the Statistics Act are the supporting internal policy instruments that provide direction on the acquisition and use of data from alternative sources. Overall, they effectively outline the roles and responsibilities of key stakeholders across the agency and define the operational steps involved in acquiring data from alternative sources.

DSD has played a significant role in the acquisition of data from new alternative sources. Its responsibilities included: acquiring administrative data sources that had a broad scope and supporting statistical programs in their acquisition of administrative data. As previously stated, DSD's role in the acquisition process is expected to change. Program divisions will be expected to carry out their own data acquisitions and DSD will act as an intermediary on all matters related to data acquisitions. DSD indicated that they intend to make this new process visible and update the policy instruments by the end of fiscal year 2019-2020.

The policy instruments have not yet been updated to reflect the new legal interpretation. Under the new legal interpretation (provided to the agency in February 2018) there are two modifications. Specifically, under section 8(2), the CS must now publish any mandatory request for information before a request is submitted to a data provider. Under section 8(3), the CS must notify the Minister of any new mandatory request for information at least 30 days before it is published.

Without these updates in the policy instruments, the agency increases the risk that employees may not be aware of, or fully understand these requirements. This could hinder the agency's ability to acquire data in an efficient and effective manner. Although work is underway to update the policy instruments, it has not yet been finalized.

Training courses cover key components of the data acquisition process, but there are opportunities for improvement.

Providing effective training to agency employees presents a unique opportunity to expand their knowledge base and allows them to strengthen skills unique to their job responsibilities. Agency employees who understand how to leverage interpersonal skills within the negotiation process will assist the agency in acquiring data from alternative sources in a more expedient manner. However, for this to occur employees must be aware of and have adequate access to training opportunities so that they can develop the requisite knowledge and skills for acquiring data from alternative sources.

Since 2017-18, two training courses have been offered to employees at the agency to assist them in carrying out their responsibilities for acquiring data from alternative sources. However, both courses contain areas that could be improved. 'Obtaining Administrative Data under the Statistics Act' was a course developed under the direction of DSD and provides guidance on governing instruments, concepts and tools on how to overcome challenges during the data acquisition process. The course was last delivered to agency employees in January 2018, and does not include any guidance on the new legal interpretation of the Statistics Act that applies to data from alternative sources. It should be noted that DSD is currently updating the training course to address the implications of the new legislative requirements, and DSD's new role in the data acquisition process.

'Skillful Negotiations' was developed under the direction and guidance of the Employee Development and Wellness Division in order to assist agency employees in carrying out effective negotiations. The course effectively provides guidance on carrying out skillful negotiations including: identifying team roles, outlining different types of negotiations and how to leverage interpersonal skills. It also provides participants with the opportunity to provide feedback on the overall course content, including each of the four main training modules. Although the overall level of satisfaction from course participants is high (86.5%) the course could be improved by including specific guidance on the new legislative and legal requirements under the Statistics Act and how to effectively utilize DSD and OPMIC when seeking to acquire data from alternative sources.

Recommendation 1

It is recommended to the Assistant Chief Statistician, Analytical Studies, Methodology and Statistical Infrastructure to ensure that the realigned governance structure embeds effective risk management oversight for acquiring data from alternative sources.

Recommendation 2

It is recommended to the Assistant Chief Statistician, Analytical Studies, Methodology and Statistical Infrastructure to ensure that key performance indicators and measurable outcomes are developed and aligned to business objectives for acquiring data from alternative sources in the Statistics Canada Data Strategy.

Recommendation 3

It is recommended to the Assistant Chief Statistician, Analytical Studies, Methodology and Statistical Infrastructure to ensure that training courses and policy instruments are updated to ensure agency employees are provided with appropriate guidance and direction on new legislative requirements under the Statistics Act and DSD's new role in the acquisition of data from alternative sources.

Review of data acquisition files

The Directive on Obtaining Administrative Data under the Statistics Act requires agency employees to implement a series of internal controls within the acquisition process to protect and safeguard the agency. To provide reasonable assurance that controls were in place and functioning as intended, the audit examined 12 private sector data acquisitions (7 administrative data files and 5 publicly available data files) that had been completed within the audit scope. The key controls assessed included:

• use of a data acquisition agreement
• legal oversight
• financial oversight
• quality assessment

Data acquisition agreements are used to acquire non-publicly available data but less so for publicly available data.

Under section 5.1.2 of the directive, the preferred method to obtain administrative data from private organizations is a data acquisition agreement, pursuant to the Statistics Act. However, the directive prescribes that an agreement may also take the form of an exchange of written communications or a contract, assuming it contains all the essential elements of a data acquisition agreement (i.e. legal authority to obtain data, its intended use or any legal requirements to protect it).

The audit reviewed seven administrative data acquisition files (obtained under the Statistics Act) to determine the extent to which the agency documents the terms and conditions of data agreements. In all seven files, the agency formalized data agreements that were aligned with the requirements under the Directive.

Publicly available data do not need to be obtained under the Statistics Act and, therefore are not subject to the same requirements as administrative data. The Guideline on data available to the public does not require or suggest that formal data acquisition agreements should be used when acquiring data available to the public.

The audit reviewed five acquisitions of publicly available data and found that three of the five files did not contain any form of agreement. The remaining two files contained either a formal license agreement or an exchange of written communications with the data provider. Under normal circumstances, there is no requirement to enter into an agreement for publicly available data.

Adequate legal oversight is provided by the Office of Privacy Management and Information Coordination.

OPMIC is responsible for reviewing all new agreements and working with legal counsel to ensure that the appropriate terms and conditions are in place to protect the agency's legal interests. Specifically, section 6.2.4.5 of the directive states that the Director of OPMIC must: 'support statistical program managers by reviewing all final data acquisition agreements before they are sent to the other organization.' As the directive does not specify what the review must include, we found that the review function varied depending on the complexity of the data acquisition.

Overall, when a data acquisition agreement was drafted, OPMIC was contacted to perform a review of the terms and conditions within the proposed agreement prior to obtaining signatures. The audit found that OPMIC provided legal oversight for all seven administrative data files (obtained under the Statistics Act). The exceptions where OPMIC did not perform a review included the file that contained an exchange of written communications and the three publicly available files without an agreement. As the Directive only identifies a requirement when there is a data acquisition agreement in place, OPMIC is not expected to be involved in the acquisition process in the absence of a formal agreement.

Financial reviews are taking place, but are not always formally documented.

When Statistics Canada acquires data from private sector organizations, the agency can acquire the data with no associated costs or pay the organization to compile the requested data. It is important to note that Statistics Canada does not pay for the data itself, rather, for the time and effort required to compile the intended data sources. DSD indicated that when there is a cost, the data provider develops a cost proposal that outlines the variables included in the final price. The cost proposal is submitted to Statistics Canada for review and acceptance. Part of the decision making process is to review the quality evaluation assessment to determine whether the value of the data is commensurate with its potential costs.

The directive and policy outline the requirements for financial oversight for data acquisitions with a cost. Specifically section 6.6 of the directive states that private sector acquisitions with a cost are to be provided to the CSSD for financial review. However, as of March 2018, CSSD was no longer involved in carrying out financial reviews for acquisitions that fall under the Statistics Act. Instead, a new process was implemented in which the responsibility to conduct a financial review of costs now falls under the responsibility of the statistical program manager and senior management with support from DSD and OPMIC. The agency's policy instruments have not been updated to reflect this change in process. They also do not prescribe any requirements for how program managers can demonstrate that they have considered whether costs are reasonable and represent value for money. Through interviews with program managers, the audit found that the financial review process that is in place includes exercising due diligence for expenditure initiation under section 32 of the Financial Administration Act.

The audit reviewed 4 acquisition files with an associated cost that were obtained under the Statistics Act to determine whether 1) a cost proposal was provided and 2) whether the cost proposal was reviewed to validate the costs for reasonableness. In all four files examined, the data provider submitted a cost proposal. In three of the four files, there was email documentation demonstrating that management was aware of and accepted the costs being proposed by the data provider. However, each of the four files did not contain any analysis to demonstrate how management determined that costs were reasonable in order to ensure value for money and that the agency's financial interests were protected.

Quality reviews of data from alternative sources are generally not being carried out.

Statistics Canada policy instruments are clear on the importance of and responsibilities surrounding the quality assessment of new data from alternative sources. As stated in section 6.4 of the directive, senior managers of statistical programs are responsible for assessing the quality of potential administrative data and their statistical usability, by using the data quality evaluation framework maintained by DSD or any other suitable tool. Contained within the data quality evaluation framework, is an evaluation questionnaire tool that is available for managers to assess new sources of data. The quality review process is an important step because it helps the agency to determine the value of the data, which in turn, helps to rationalize potential costs.

The audit reviewed the 12 acquisition files to determine whether a quality assessment was completed prior to obtaining the data. Overall, the audit found that program managers were generally not using DSD's data quality evaluation questionnaire or any other suitable framework. Four of the 12 files had documentation to demonstrate that a quality review took place. Of the four files that had completed a quality review, three were administrative data and one was publicly available data. The remaining eight files did not contain documentation to demonstrate that a quality review was completed prior to signing the data acquisition agreement. Through consultation with DSD, we were informed that statistical program managers were not using the standard evaluation questionnaire because of a perceived time burden. The audit also found cases in which the individuals responsible for the quality review were no longer in their roles resulting in documentation not being available. Without consistently evaluating the quality of data from alternative sources, there is a risk that statistical programs may acquire data that are incomplete or incompatible with their programming needs.

DSD has since taken steps to address this by updating the quality evaluation questionnaire to ensure it is more streamlined. In interviews with DSD they have stated that the updated quality evaluation framework, as developed by the Modern Statistical Methods and Data Science Branch, will be completed by the end of fiscal year 2019-2020 and the intent is to have it available for use in early 2020.

Recommendation 4

It is recommended to the Assistant Chief Statistician, Analytical Studies, Methodology and Statistical Infrastructure to ensure that the data quality evaluation questionnaire is updated and that agency employees are aware of the requirement to assess data for quality and statistical usability prior to acquiring data from alternative sources.

Appendices

Appendix A: Audit criteria

Appendix B: Initialisms

CS

Chief Statistician

CSSD

Corporate Support Services Division

DAC

Data Acquisition Committee

DSD

Data Stewardship Division

SCDS

Statistics Canada Data Strategy

OPMIC

Office of Privacy Management and Information Coordination