This job aid is designed to help you critically assess the data presented to you. No data is perfect. By understanding the strengths and limitations of the data, you can avoid being misled—and make smarter, more informed decisions.
Source
Can you identify the specific data source?
If the estimate comes from a report, can you identify its original source?
Is the source reliable and trustworthy?
Are there known limitations or caveats associated with the data used to produce the estimate?
Methods
How was the data obtained—was the methodology transparent?
Did it come from a sample survey of a well-defined population of interest, a marketing poll, an administrative database, or did it reflect insights from a qualitative study like a focus group?
Timeliness
Is this data current?
When was this dataset acquired?
Do you know when the data was last updated?
Does timing affect its validity?
Sample
What is the sample size?
What is the sampled unit (e.g., individuals, households, businesses)?
What is the response rate (out of all sampled units, what portion responded)?
How were respondents chosen (were they randomly selected, or did they volunteer)?
Does the sample represent the population of interest (for example, all Canadians) or are some units systematically excluded from the sample, resulting in poor coverage of this population?
Definitions and measures
How is the concept of interest defined, and how is it being measured?
Is the definition based on a recognized standardFootnote 1?
If you are comparing estimates from different sources, were the underlying concept definitions and measurements the same, or at least comparable?
Accuracy
How was the estimate calculated?
Were qualityFootnote 2 indicators made available to assess the estimate's fitness for use?
Context
What is the broader context of the data?
How do these estimates compare…
- … to estimates from other data sources?
- … to similar estimates from previous years, if available?
- … across groups based on current knowledge about group differences?
Has sufficient background been provided to acknowledge the lived experience or historical realities of the population(s) to which the data refers?
Disaggregation
Are the data available at the right level of disaggregationFootnote 3 for your analysis?
Additional data sources
Are there other data sources that could provide a more nuanced understanding of the concept being measured?
Could linkingFootnote 4 multiple datasets reveal new information?
Ethical considerationsFootnote 5
What is the expected public benefit of using this data?
What are the potential negative consequences of using this data (e.g., privacy intrusiveness, lack of transparency, harm to individuals or groups, trust erosion, sustainability issues, security concerns, etc.)?
Are the potential benefits credible considering the potential consequences?