Disposition of recommendations

View the most recent version.

Policy Committee had for consideration the Report of the Task Force on the Review of Quality Assurance Practices. Policy Committee decided the following disposition for the recommendations.

Recommendation 1: Each program that was reviewed should develop a set of costed proposals for Corporate Planning Committee consideration that are designed to reduce the risks of producing erroneous data.

Disposition: Proposals should be developed for the following programs and be ready for consideration by CPC as part of a mini-LTP process by the end of May:

• Quarterly Balance of Payments
• Monthly International Trade
• Monthly Retail Trade Survey
• Monthly GDP by Industry
• Quarterly Labour Productivity
• Monthly Survey of  Manufacturing
• Dissemination Division and Communications and Library Services Division (DCD)

Each proposal should address the following points:

• Description of the risks
• What are the financial and human resources required?
• How would the requested funds reduce the risks?

Policy Committee noted that this exercise is to address ‘traffic accident’ risks and not general quality improvements.

Recommendation 2: Three programs should receive the highest priority for investment to further reduce the risk of errors: the Consumer Price Index, the Balance of Payments, and the International Trade program.

Disposition: same as for Recommendation 1. It was noted that the necessary resources have been allocated to the CPI as part of the 2007-08 LTP.

Recommendation 3: Investments should first be targeted at addressing the human resources issues affecting the programs, by ensuring sufficient, appropriately-trained staff to support the mission critical programs.

Disposition: same as for Recommendation 1.

Recommendation 4: All mission critical programs should have a strong and explicit research and analysis capacity, separate from the production operation, whose role is to challenge the data and to conduct research into the particular subject matter of the program. The most suitable form of this capacity may vary across programs.

Disposition: same as for Recommendation 1.

Recommendation 5: The best practices found in this review should be disseminated as widely as possible and should form the basis for the training course on quality assurance that is currently under development.

Disposition: the Learning and Development Committee will assume the prime responsibility to address this recommendation.

Recommendation 6: Statistics Canada should develop a formal Quality Incident Response Plan (QIRP) containing standard protocols for recognizing and dealing with data quality incidents, whether before or after data release. A useful model for this is a plan developed at the Australian Bureau of Statistics, which could be supplemented with our own best practices. Training on the QIRP would be an important part of this initiative.

Disposition: before deciding on the recommendation Policy Committee asked the Task Force to use one of earlier data quality incidents to simulate what would have been different if such a plan had been in place and then report to Policy Committee for further consideration.

Recommendation 7: Divisions should identify all data flows into and out of their division and ensure that formal arrangements are in place to ensure sufficient understanding of how the data were produced and how they will be used by the recipient division.

Disposition: approved. It should be part of the best practices and of the awareness training.

Recommendation 8: Statistics Canada should establish an ongoing program of quality assurance reviews, based on teams of Assistant Directors.

Disposition: approved in principle. The Methods and Standards Committee is asked to develop, for consideration by Policy Committee, an operation and implementation methodology, including criteria for the selection of programs.

Recommendation 9: Media databases such as the Daily Economic Brief should be disseminated more widely to ensure their receipt by analysts in all key sub-annual economic programs.

Disposition: approved. It was agreed that Karen Wilson will follow up and report, at the mini-LTP at the end of May, on the cost (if any) for implementing the recommendation.

Recommendation 10: Collection operations should include, as a matter of course, good practices such as end-to-end testing, monitoring, and direct communications between subject matter and collection staff.

Disposition: for the nine mission critical statistical programs that were reviewed and where the problem surfaced, it was agreed that this recommendation be implemented. For the other statistical programs not reviewed by the Task Force, the proposed Quality Assurance Review program mentioned in Recommendation 8 could be the trigger.

Recommendation 11: Programs should make use of corporate generalized systems and centralized methodology services as a way of reducing the risk during the editing, transformation, imputation and estimation steps.

Disposition: this recommendation is assigned to the Architecture Review Board for follow-up.

Recommendation 12: Analytic tools such as those described should be made widely available to analysts to assist in the certification process.

Disposition: it was agreed that Karen Wilson should come forward with specific suggestions for implementing that recommendation.

Recommendation 13: The use of CANSIM and Smart Publishing should be expanded, as a way of reducing the risk of errors.

Disposition: Policy Committee noted that this is already underway. Where there is a significant risk that can be reduced by moving forward with the use of Smart Publishing or CANSIM, programs areas should present their proposal as part of the mini-LTP at the end of May.

Recommendation 14: Mission critical programs should have explicit programs of outreach and support to major users and the media as part of, and immediately following, each release.

Disposition: approved. It was agreed to leave the follow-up to the Task Force to trigger the implementation of the recommendation where the Task Force noticed no such program existed.

Recommendation 15: Plans and budgets for redesigned programs should include provisions for the thorough testing of new systems and a sufficiently long parallel run, so that problems can be detected and corrected before switching to the new series.

Disposition: the LTP guidelines should ask directors to articulate in their requests whether or not and to what extent a parallel run is needed.

Recommendation 16: At the time of the drafting of this report, efforts were being made at the Agency level to increase the intake of new employees. Nevertheless, it will take some time to alleviate the pressure of staff shortage on programs. In the meantime, mission critical programs should receive priority in staffing and prioritization of work should be done in order to identify what activity can be expended in the short term.

Disposition: the mini-LTP process at the end of May will address this recommendation.

Recommendation 17: More recognition and resources should be given to the monthly production programs. Recruits should be encouraged to spend some time in a monthly production program to change the perception. The benefits of working on a regular production survey should be better disseminated.

Disposition: it was agreed that this recommendation should be addressed on a number of fronts, for examples:

• The Working Group looking at the Assistant Director workload will be expanded to include the Chief level.
• The new staffing system should put more weight on operational requirements versus individual interests.
• We should ensure that weight be given to providing the necessary candidates to mission critical statistical programs.
• We should let know people that their track record is valued in selection processes.
• Should job shadowing be required for the mission critical statistical programs, it should be part of the mini-LTP at the end of May.

Recommendation 18: We recommend the expanded use of communities of practice, asset qualifications and other mechanisms in our staffing processes in order to ensure an adequate supply of specialists for programs such as the CPI, BOP, LP and IT. 

Disposition: agreed. The Corporate Resourcing Committee is to follow up.

Recommendation 19: In order to reduce risks associated with the departure of experienced staff, each program should develop a specific succession plan for key functions in the program. The plan should include training, rotation of staff within the community of practices and some job shadowing when retirements can be identified.

Disposition: it was agreed to implement that recommendation where managers feel that there is vulnerability. It is understood that this applies only to the nine mission critical statistical programs and the DCD program that were reviewed by the Task Force.

Recommendation 20: Under current conditions, proposals to further increase the timeliness of the programs should be regarded with extreme caution, especially if there is a risk that the time would be taken from the certification or data release steps.

Disposition: Policy Committee took note of the recommendation.

Recommendation 21: All programs should have contingency plans, developed in concert with Dissemination and Communications, to deal with unplanned last minute changes.

Disposition: for the nine mission critical statistical programs that were reviewed, the relevant Director General should ensure follow-up. For the rest of the statistical programs, it should be part of the training.

Recommendation 22: Each program should review its interdependencies among systems and prepare a plan with its associated costs to reduce it. It should be noted that the present mainframe migration is also an occasion to reduce the interdependencies of systems.

Disposition: same as for Recommendation 1. This is restricted to the nine mission critical statistical programs and the DCD program.

Recommendation 23: Programs should ensure that they conduct thorough testing and prepare complete systems documentation.

Disposition: This recommendation is restricted to the nine mission critical statistical programs and the DCD program and should be part of the mini-LTP at the end of May if there are risks identified.

Recommendation 24: Service areas should ensure that better planning and communication protocols are in place to avoid leaving programs short-staffed. Also it is suggested to rotate leave periods throughout the summer.

Disposition: HR Branch, in consultation with Methodology Branch and Informatics Branch, should develop guidelines regarding the ‘leave with income averaging program’ that should be distributed widely within the Bureau. With regard to the nine mission critical statistical programs, the service areas should be required to provide ample notice (six months) of expected staff changes.

Recommendation 25: Formal protocols should exist at a senior enough level to ensure that data that are critical to programs are provided on a timely basis and that changes at the source of the data are communicated.

Disposition: this is ongoing.

Recommendation 26: Future reviews should include direct interviews with all significant service areas, time permitting.

Disposition: this recommendation should be rolled in Recommendation 8 (establishment of a Quality Assurance Review program).

Recommendation 27: The Summary Report and the individual program reports should be made available on the ICN.

Disposition: The Summary Report will be put on the ICN, with a note that the detailed reports are available on demand.

Recommendation 28: The nine programs should report on activities or modifications related to the QA Review at the time of their next Biennial Program Review or Quadrennial Program Review. This will ensure that any changes since the QA Review are documented and publicly available.

Disposition: agreed.

Recommendation 29: Statistics Canada should present the results of the review to the National Statistics Council and other external advisory committees for their review.

Disposition: The Chief Statistician will, at the next meeting of the National Statistics Council in May, report on the results. The Summary Report, along with the text on the disposition of the recommendations, should be made public by putting it on the STC website. Don Royce will act as the spokesperson with the media. There should be also a presentation to the Labour Management Consultation Committee.