Statistics Canada
www.statcan.gc.ca
Common menu bar links
Best practices
Archived Content
Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please "contact us" to request a format other than those available.
As mentioned above, many “best practices” were discovered over the course of the program reviews. This section outlines the major best practices in order of the process steps analysed. These seven standard steps were outlined above in section 2.0 of the report. Many less important best practices are described in the individual program reports.
Preparation for certification
Data collection/acquisition
Edit and data transformation
Imputation and estimation
Certification
Release step
Post release
Other best practices
Preparation for certification
This step in the production cycle can be characterized as a research and fact-finding stage whereby analysts follow current events related to their respective data programs.
Two important best practices emerged from the review.
First, a daily summary of economic reports in the media is prepared by Industry Accounts Division which classifies articles and events by NAICS code. A Daily Economic Brief (DEB) is sent around to many analysts to use in preparation of the production cycle. The briefs cover strikes, plant openings/closures, announcement of major contracts or projects, etc. A similar database is prepared by Balance of Payments Division on major international transactions. Analysts use these media databases in the production of estimates to verify that the survey instruments capture important events or to explain extra-ordinary movements in the data. This practice should be encouraged for all major sub-annual data releases and the DEB and other such databases should be shared across programs.
Recommendation 9: Media databases such as the Daily Economic Brief should be disseminated more widely to ensure their receipt by analysts in all key sub-annual economic programs.
Most analysts in the programs reviewed attend the monthly Daily Theme Analysis Panel. This is a meeting that takes place at the beginning of every month where a summary of trends to date in major economic releases is done along with a review of major events and economic issues for the month ahead. This promotes the exchange of analytical information across programs and helps analysts make the appropriate links to other data sources. This is a relatively new analytic function but is seen as very fruitful and should be continued.
Data collection/acquisition
In the case of the survey programs, all of them make use of the corporate collection systems. Some are totally reliant while others do some partial collection within the program.
Several best practices were identified at this stage:
Prior to each collection period, the LFS conducts end-to-end testing of systems using a pre-specified (fictive) data set where the expected output is known in advance. This ensures that any updates or changes in date ranges or specifications in the programs of the data system are functioning according to expectations.
In programs where the collection period is compressed over a short period of time as is the case for monthly surveys, daily reports on operations including completion rates or response rates by interviewer to monitor the collection process are extremely useful. They can trigger the need for follow up or extra effort by collection staff.
While the collection systems are extremely efficient in terms of resource allocation, there is a risk associated with collection staff not having sufficient subject matter expertise for the various surveys for which they work on collection. A communications network with collection staff has been developed in some cases including visits by subject matter staff to follow up on issues identified in the field and to inform collection staff of changes or issues to be watchful of at the collection stage. This practice is very successful in engaging collection staff in the process of data quality management.
Recommendation 10: Collection operations should include, as a matter of course, good practices such as end-to-end testing, monitoring, and direct communications between subject matter and collection staff.
In the case of derived statistics programs a best practice emerged whereby the program established written data delivery contracts with supplying divisions one year in advance. These contracts specified the variables, level of detail, delivery dates and revision process so that the expectations of both sides of the contract are known. They are particularly helpful when there is staff turnover in either the supplying or receiving division so that the arrangements are codified and no time is lost during the transition period. Since many interdependencies were identified between the various programs covered in this review, this becomes a very important practice in maintaining ongoing quality assurance, both for the recipient and the supplier of the data. This point is addressed by recommendation 7.
Edit and data transformation
A variety of editing practices ranging from manual edits to completely automated systems were identified. Programs are encouraged to use corporate tools to automate the identification and correction for non-response, erroneous response, outliers etc. A particular best practice is used by the Labour Force Survey in the form of consistency edits whereby coding of responses is verified for consistency with other variables collected.
Consistency verification is also a best practice used in most derived statistics programs whereby variables can be analysed from different sources for consistency.
Imputation and estimation
A variety of imputation methods were identified including the use of donor response, historical imputation or the use of auxiliary variables. Corporate tools (generalized systems such as Banff and GES) are available to support the various methods and programs are encouraged to use them.
The survey systems by and large depend on centralized methodology expertise in the estimation process including seasonal adjustment and benchmarking. This best practice promotes consistent high quality statistical methodology across programs. Programs which do not benefit from these resources should be encouraged to do so.
Recommendation 11: Programs should make use of corporate generalized systems and centralized methodology services as a way of reducing the risk during the editing, transformation, imputation and estimation steps.
Certification
The certification step of any process involves a variety of analytic techniques to test the validity of the data. Two particularly important best practices were identified in this regard.
Those programs which had access to easily usable time series analysis tools showed regular attention to time series consistency and were capable of in-depth coherence analysis. The availability of such tools on the desk tops of the analysts facilitated repetitive analysis functions (e.g., ratios, percentage changes) but also gave the capability to add ad hoc analysis in instances where the data presented some unexpected change which needed to be explained or verified before release. Access across all programs to such tools would benefit many of the programs.
Some programs use a canned package of analytic tables and charts to review not only their own data, but also related indicators published elsewhere in the statistical system. This is particularly useful for coherence analysis.
Recommendation 12: Analytic tools such as those described should be made widely available to analysts to assist in the certification process.
Another best practice used in certain programs is the practice of calling upon independent analysts who had not been involved in the production process to do high level analysis. These analysts are able to spot unusual movements or inconsistencies that were not observed earlier in the production cycle. Such analysts are often responsible for providing the analytic content of the text which will be used in the subsequent release of the data in The Daily. These same analysts are also available to follow up on specific issues identified during one round of production to follow the developments through subsequent months or quarters. Their insights can lead to adjustments in collection, editing or estimation phases in future rounds. The function may also include more in-depth analysis to be published outside of the regular production cycle and communications with key external users to get feedback on quality issues.
Another fruitful practice is the review of the data an early stage by staff of related programs. One example is the verification of import and export data by three programs each month (International Trade Division, Income and Expenditure Accounts Division and Balance of Payments Division). Another is the review of manufacturing shipments data by the Monthly GDP program for conversion to real estimates (price change effects removed). All programs involved view the data from different perspectives, which can be enlightening in the verification process.
The best practices in the two preceding paragraphs are examples of the analytic and research capacity referred to in recommendation 4.
A final stage of this process is the briefing of senior management of the highlights of the upcoming release. A dry run of this briefing before the presentation to Policy Committee helps ensure that all bases have been covered in the verification process.
Release step
At this stage, once the data have been certified, databases are prepared to load onto CANSIM and Daily releases are prepared as well as electronic and printed publications.
Again, a number of best practices were noted.
First, Divisions must organise a system of verifications against internal data bases and products – which could include a team that does last minute spot checks of the various products, including The Daily text, charts and graphs before release.
Second, use of corporate systems such as “smart publishing” to automatically generate graphs and tables for publications from CANSIM help mitigate the possibility of errors across different product lines.
Recommendation 13: The use of CANSIM and Smart Publishing should be expanded, as a way of reducing the risk of errors.
Third, setting up good co-ordination and communications with the Dissemination and Communications Divisions of Statistics Canada is a must. Periodic meetings to discuss roles and responsibilities or any proposed changes in product line or standards for publication are useful.
Post release
The post release phase largely consists of follow up with partners in the production process such as collection or methodology services or data suppliers, as well as following the media coverage and requests from other users of the data.
This phase is particularly important for derived statistics programs which rely entirely on others to supply their source data. The best practice here is regular and frequent meetings to discuss emerging trends and/or inconsistencies which may signal changes in data quality. This practice is followed regularly for some parts of the program and on a more ad hoc basis for others. Given the interdependencies already noted above among the nine programs studied, this practice should be formalized and regular.
Another interesting practice is the “Post Mortem Group” established by the Quarterly Income and Expenditure Accounts Program including Balance of Payments and Monthly GDP. A large group of users of the derived statistics programs are invited to a meeting the day following the release. The programs give a briefing on the emerging economic trends and the users pose various questions about the data and are encouraged to express any concerns related to data quality. This function helps ensure that the data quality concerns of the users are top of mind for the data producers as well and that the analytic needs of the users are well understood. It is also a useful forum for Statistics Canada to inform users of upcoming changes to the program including data quality improvements or risks. It helps users make better informed decisions when using the data.
A similar best practice was found in the Labour Force Survey, which works closely with the major media to ensure that the data are reported and interpreted correctly, even going so far as to check the numbers that are published by (for example) the Canadian Press.
Recommendation 14: Mission critical programs should have explicit programs of outreach and support to major users and the media as part of, and immediately following, each release.
Other best practices
A key aspect to quality assurance is management of change. All on-going surveys and programs undergo periodic review or re-design. In all of the cases studied, where the introduction of change was well managed, it involved significant testing of the new systems and overlap estimation periods to ensure smooth transition from the old to the new estimates. It was clear that the best practice in management of change was building these two key elements into the redesign plan.
Recommendation 15: Plans and budgets for redesigned programs should include provisions for the thorough testing of new systems and a sufficiently long parallel run, so that problems can be detected and corrected before switching to the new series.
