Quality…the sequel

Last month, the StatCan Blog focused on how the Quality Secretariat defines quality.

Now, it’s time to tell you how it’s done!

Specifying the need

Each time a new statistical program is created or an existing one is revised, Statistics Canada typically conducts broad consultations with partners. Understanding their information needs is essential to designing questions that will produce an output relevant to these needs.

The National Statistics Council and subject-matter advisory committees provide ongoing advice on existing and emerging statistical needs. The federal-provincial-territorial committees provide input on the interests of provinces and territories. Managers also consult with representatives from major user groups, other federal departments and agencies, municipalities, think tanks, and the private sector.

Creating a design

Most people think about surveys or the census when they think about data. But a great many of our statistical programs depend in whole or in part on administrative data as well. These are data collected by other agencies. They are shared with Statistics Canada for specific purposes under a strict set of rules. Like other statistical information, administrative data receives the highest level of protection once it is under Statistics Canada’s purview.

When we do undertake surveys, their design is subject to intense scrutiny. Several factors come into play, including the inherent variability of the phenomena being measured, the levels of aggregation at which estimates are needed, the intended use of the data, the required precision of estimates and the cost of collecting and processing the data. Sometimes, we use administrative data in lieu of survey questions, thus reducing the burden on respondents. Quality indicators are embedded in every phase of the survey design to ensure data are cost-efficiently produced at the level of accuracy needed.

Respondents are often curious about how they were chosen when they are contacted to provide data. But typically, for social surveys, it is not the actual individuals who are selected. Instead, it is usually the address that is chosen. Residential addresses are randomly selected for participation in a survey using leading-edge statistical methods.

Collecting data

When Statistics Canada undertakes a survey, data can be collected via one or more modes. For example, for the 2011 Census, responses could be provided via Internet or on paper questionnaires either sent by mail or completed with the help of an interviewer.

Specialists ensure that survey questions are clear and that respondents can answer them easily. Online questionnaires are put through their paces on computer screens and smartphones to make sure they function properly. And online questionnaires include simple steps and a help function to assist respondents in providing high quality data.

“A well-designed questionnaire is rather like a clean window,” Laurie Reedman, chief of the Quality Secretariat, explains. “You look through it and you do not notice the window. Ideally, the respondent should not be aware of the questionnaire and not have to fight with it to provide an answer. They should read or listen to the question and be able to answer naturally. It should be that simple. This is what we strive for.”

In collecting data in person, proper interviewer training ensures consistent and accurate reporting. Training also emphasizes the importance of maintaining the goodwill of respondents and reinforces the concept of a partnership between the statistical agency, the citizens of Canada, businesses, governments and other institutions.

Prior to producing results, the information collected is subjected to various processing steps designed to capture the data into electronic form and to ensure its quality is good. In processing data, the objective is to produce data that are not necessarily perfect but rather of high quality and fit for their intended use.

Analyzing and certifying data

Subject-matter experts then analyze the data. These experts compare data from identical or similar surveys conducted in the past and related administrative, census or international data, to ensure coherence. For example, the monthly estimates of the Survey of Employment, Payroll and Hours are compared with similar data from the Labour Force Survey.

Using standard concepts, classifications and target populations also helps to promote coherence. And, to assist users in interpreting Statistics Canada’s published data, survey definitions, data sources and methods are available on our website. Whenever feasible, common methodology and processing systems are used across surveys.

Sharing and storing data

As Statistics Canada’s official release vehicle, The Daily provides the public with information about every new Statistics Canada product or dataset; thus making its data readily accessible. It also provides supplementary information and metadata so people can interpret and use this data.

Data must also be safely archived. “A stored piece of information is only as good as your ability to retrieve it when you need it,” Ms. Reedman notes. “Changing technology has again provided both opportunities and challenges to archiving our data.”

Looking back

An evaluation at the end of each step in the quality assurance process closes the loop. What went well? What was learned? How can it be improved? 

“Quality assurance practices often overlap with good project management practices,” Ms. Reedman notes. “It is simple things like having a schedule, good information management to organize your documentation, using standardized processes so staff can rotate, and you are not left with a knowledge gap if somebody leaves the team.”

Sometimes, the Quality Secretariat is asked to investigate a process because someone has noticed a glitch. The Quality Secretariat steps in to monitor and provide advice or training. Lessons learned are shared with others.

The secretariat also works with the agency’s International Cooperation Division where its expertise on quality assurance frameworks is shared with other countries experiencing similar challenges, and with developing nations to increase their statistical capacity. Collaborating with other statistical agencies around the world helps Statistics Canada remain a world leader in quality statistical information, standards and practices. The quest for quality data knows no boundaries.

Next month: Globalization and economic statistics