Defining quality

Laurie Reedman has one of the coolest job titles at Statistics Canada—Chief of the Quality Secretariat. It has a nice ring to it, like she is the official StatCan taste-tester.

You may wonder why StatCan needs a Quality Secretariat. The agency has been around for almost 100 years; hasn't it solved all the data quality questions by now?

Turns out quality management is a moving target.

New realities

A number of new realities have added complexity to the way StatCan manages its business, according to Ms. Reedman. One reality is that data users are more knowledgeable and are requesting more sophisticated data and analysis. Another is the advent of Big Data and deciding how it may fit into StatCan's overall program.

Meanwhile, communicating with respondents has taken a twist because the younger generation has exchanged land lines for cellphones and is much harder to reach. Response rates are declining across all national statistical agencies, although more slowly in Canada, so StatCan must find new ways to collect data and make full use of it. "It is important to push the science, the mathematics behind sampling theory, to still deliver the most accurate values, given the reality of non-response," Ms. Reedman says.

Knowing the quality of each statistical program is crucial to strategic planning and to accountability for results. Statistics Canada is constantly assessing its ways of doing business, to make sure efficiency is maximized, while striving to maintain data quality.

Statistics Canada's mission is "Serving Canada with high-quality statistical information that matters." The Quality Secretariat's mandate is to promote and support quality assurance practices. That means Ms. Reedman and her team look at quality as a factor that must be incorporated into every stage of the statistical process—from a survey's conception to the publication of analysis and results in the local newspaper. All along that process, quality at Statistics Canada is every employee's business.

"You cannot pull into the Quality Car Wash at the end of the process and expect to clean everything up," Ms. Reedman says. "What distinguishes Statistics Canada is our reputation for producing high quality data. Statistics Canada is well known across Canada. Canadians know us and they trust us. I think that distinguishes us from other data collection activities."

Keeping that trust is the raison d'être of the Quality Secretariat.

Dimensions of quality

Quality is not a black and white concept—it's relative, not absolute. For example, when statisticians create a survey, a balance must be struck between resource constraints, the willingness of busy people to sit down and fill out questionnaires, and the competing demands for greater quantities of information—and more detail.

The first step is to define quality. Statistical agencies have commonly accepted definitions of data quality for official statistics and have adopted a concept called ‘fitness for use'. It means that users must be provided with the information necessary to judge its fitness for their intended use.

Within the 'fitness for use' concept, six dimensions of quality help guide and inform Statistics Canada employees:

• Relevance asks whether the data product is meaningful to Canadians, whether the questions are useful and whether policymakers get the information they need.
• Accessibility relates to how easy it is for users to obtain data.
• Accuracy is the degree to which the data describes the phenomena it was designed to measure. Think the 19 out of 20 times note at the end of a newscast about statistics.
• Timeliness refers to the time span between the end of the reference period to which the data refers and the date the information becomes available for use.
• Interpretability is the availability of the supplementary information and metadata necessary so users can interpret and use data.
• Coherence reflects the degree to which data aligns with other statistical information. The use of standard concepts, classifications and target populations promotes coherence, as does the use of common methodology across surveys.

However, there may be tradeoffs. For instance, timeliness typically involves a tradeoff against accuracy and relevance. Leave data on the shelf too long, and it will not be useful or relevant. Rush to publish without the proper safeguards, and accuracy may be compromised. StatCan's task is to strike the right balance so Canadians receive quality data that matters in a timely fashion.

Next month: Producing quality