The Canadian Health Measures Survey 2007/2008 to 2012/2013

Evaluation Report

February  2013

  • Executive summary
    • Context
    • Evaluation design and methodology
    • Key findings
  • 1. Introduction
  • 2. Canadian Health Measures Survey
    • 2.1 Background
    • 2.2 Canadian Health Measures Survey - objectives, activities and expected outcomes
    • 2.3 Governance
    • 2.4 Program resources
    • 2.5 Evaluation context, scope and issues
  • 3. Methodology
    • 3.1 Approach and design
    • 3.2 Document and literature review
    • 3.3 Administrative data review
    • 3.4 Financial data
    • 3.5 Key informant interviews and group interviews
    • 3.6 Survey of primary data users
    • 3.7 Limits of the evaluation
  • 4. Key findings
    • 4.1 Relevance
      • 4.1.1 Program responsiveness
      • 4.1.2 Alignment with the government's and Statistics Canada's priorities and obligations
    • 4.2 Performance
      • 4.2.1 Effectiveness – Achievement of outputs and expected outcomes
      • 4.2.2 Performance - Efficiency and economy
  • 5. Conclusions
    • 5.1 Relevance
    • 5.2 Performance – Effectiveness
    • 5.3 Performance – Economy and efficiency
  • 6. Recommendations
  • 7. Management response and action plan
  • Appendices
    • Appendix 1: Canadian Health Measures Survey process map
    • Appendix 2: A composite logic model for all activities under the Action Plan
    • Appendix 3: Canadian Health Measures Survey (CHMS) evaluation matrix
    • Appendix 4: A list of publications reviewed as part of the literature review

Executive summary

This report presents the results of the Canadian Health Measures Survey (CHMS) evaluation, which was conducted from April to October 2012. It provides credible and neutral information on the ongoing relevance and performance of Statistics Canada's CHMS activities. The scope of the evaluation is the time period from 2007/2008 to 2012/2013. This evaluation examined the five Treasury Board Secretariat core issues, in accordance with the Policy on Evaluation.

Context

CHMS is Canada's first nationally representative direct health measures survey. The survey was launched in 2007 to address long-standing limitations and data gaps within Canada's health information system. The principal objective of CHMS is to collect new and important data on Canadians' health status by

  • providing a platform and infrastructure for obtaining data and information through physical and laboratory measures to meet the emerging needs of several branches within Health Canada (HC) and the Public Health Agency of Canada (PHAC), as well as other add-on studies
  • collecting and disseminating direct health measures data, including those on environmental contaminants
  • promoting research using direct health measures data by providing access to nationally representative data.

Evaluation design and methodology

The evaluation was conducted in accordance with the Treasury Board Secretariat (TBS) Standards on Evaluation for the Government of Canada. Evaluation design, effort and resources were calibrated while taking into consideration the roll-up horizontal evaluation that would use the findings of this study as an input. To assess CHMS, evaluation issues and questions from the current Horizontal Evaluation Matrix for the Action Plan were slightly updated to better reflect the evaluation questions and indicators applicable to Statistics Canada.

The approach used to evaluate CHMS can be defined as theory-driven and based on a non-experimental design using post-collected information. Findings are collected from multiple lines of evidence: document and literature reviews, administrative and financial data reviews, a series of key informant interviews, and a survey of primary data users.

The evaluation efforts ensure an appropriate balance between the level of effort and the context. Nevertheless, because of contextual elements and resource constraints, there are a number of limitations associated with this evaluation:

  • Timing of the evaluation – The findings are limited by how complete the CHMS cycles are (only Cycle 1 was fully complete at the time of the evaluation).
  • Measurement of outcomes – There was a limited availability of performance data systematically collected on an ongoing basis according to the performance measurement framework for the Action Plan developed in 2009.
  • Interviews – A limited number of key informant interviews were conducted; however, this limitation can be mitigated during the roll-up evaluation if the leading team at HC finds it necessary.
  • Document and data review – Some challenges were faced when assessing the efficiency and economy of the CHMS program. The practice of compiling financial data by fiscal year and not by cycle, the lack of baseline data on cost and resources used by level of output produced, and the lack of data on similar surveys were some of the factors that affected the completeness and the conclusiveness of the evaluation evidence.

Key findings

Relevance

CHMS represents a unique program in Canada, providing direct health measures data to support health research, policy, and decision making. Evidence shows that the program is relevant to Canadians and health organizations, with a clear present and future need for the program. Although most feel that CHMS responds to current and emerging content needs, there was no content plan ever developed for CHMS.

CHMS is well aligned with the priorities of the federal government and Statistics Canada. The federal government is in the best position to deliver CHMS, and has specific legislative obligations being met through the program, making the delivery of the program a legitimate, appropriate and necessary activity.

Performance - Effectiveness

CHMS outputs have been produced as expected and the expected immediate outcome is being achieved: reliable and usable data are available on the baseline health status of Canadians, and on the level of, and exposure to environmental contamination. Some issues exist concerning the accessibility of the data, which could impair the long-term outcomes if not addressed. More specifically, even though most researchers are aware of the data, many are unaware of how to access them. This view is corroborated by external interviewees, who believe that the lack of—or complexity of—accessibility is a barrier to the use of the data. Some key informants commented on the unavailability of microdata files for CHMS on the Internet, and on how researchers must rely on the research data centres (RDCs) to use CHMS data files. Internal and external researchers raised various concerns related to RDCs, such as a lack of awareness of how to access the data, and timelines to get proposals approved (sometimes up to six months).

Evaluation findings demonstrated evidence of awareness of CHMS data among the general public, researchers and data users as a result of publications based on Cycle 1 data. However, issues, such as the 'reach' of publications, the lack of promotion of CHMS data to scientific communities, and the timing of this evaluation were identified as limitations to awareness. Researchers felt that there needs to be better communication and better collaboration to avoid duplicating research and analysis should different research groups look at the same data.

The evaluation findings provided evidence that CHMS data are being used for research, policy, and decision making—the expected long-term outcome. Researchers have already used CHMS data to validate self-reported data from other surveys, as well as to do research for scientific discovery. In addition, CHMS data are used for policy and decision making in the areas of physical activity, environmental exposure, nutrition markers, and oral health. However, it is still too early in the program's life cycle to broadly demonstrate CHMS's full potential. Information on publications (research studies) are only tracked internally and not shared with external stakeholders, making the collaboration and reporting, in some instances, more difficult.

Performance – Economy and efficiency

The shift from predominantly cost recovery to core funding of CHMS, as a result of the 2008 Action Plan, helped stabilize the survey, which had a positive impact on CHMS. In particular, it allowed for better long-term planning and for seeking opportunities to increase operational efficiency. The evaluation results demonstrated sufficient human and financial resources to support the program. Some external parties have indicated that increased efficiency and economy could be achieved through more flexible hiring practices and through the use of local clinics and infrastructure as an alternative to the mobile clinics. While these options might be worth further consideration, their feasibility must be assessed against the extent to which they could impair the quality of data collection.

Furthermore, the use of existing CHMS infrastructure by other federal partners allows them to achieve their objectives in an efficient way while reducing time and costs. Sharing complementary knowledge between Statistics Canada, federal partners and academia increases the analytical capacity in the health domain.

CHMS is the only nationally representative direct measures survey in Canada and complements other Canadian and international studies. Therefore, there is no evidence of duplication of efforts that might influence efficiency.

1. Introduction

The purpose of this study is to evaluate the Canadian Health Measures Survey (CHMS) as part of the Action Plan to Protect Human Health from Environmental Contaminants. The Action Plan is a horizontal initiative between Health Canada (HC), the Public Health Agency of Canada (PHAC) and Statistics Canada that was launched in 2008/2009. HC is the lead department for the horizontal evaluation, which is due for completion by March 2013.

This evaluation was conducted as prescribed by the Departmental Risk-based Audit and Evaluation Plan for 2012/2013 to 2016/2017, approved by the Departmental Evaluation Committee on March 27, 2012. The project was managed and carried out by Evaluation and Professional Practices Division, Audit and Evaluation Branch.

2. Canadian Health Measures Survey

2.1 Background

CHMS is Canada's first nationally representative direct health measures survey. Planning and development for CHMS started in 2003 as part of an extension of the Health Information Roadmap Initiative and in response to the need for a national, comprehensive source of accurate direct health data, which was expressed by policy makers, provincial health departments, researchers and health professionals from many fields. The survey was launched in 2007 to address long-standing limitations and data gaps within Canada's health information system.

To assist Canadians in reducing their health risks resulting from environmental contaminants and to further develop environmental health monitoring and surveillance, the federal government introduced the Action Plan to Protect Human Health from Environmental Contaminants (the 'Action Plan') in 2008. The Action Plan's long-term objective is to reduce health risks to Canadians, particularly vulnerable populations, from harmful environmental contaminants. To accomplish this objective, the Action Plan has two basic components: the first is the "Environmental Health Guide" and the second is monitoring and surveillance. Statistics Canada's CHMS falls under the monitoring and surveillance component. This component consists of a series of surveys and surveillance activities, with the objective of providing Canadians with a better understanding of environmental exposure and potential related health risks.

2.2 Canadian Health Measures Survey - objectives, activities and expected outcomes

CHMS collects key health information on Canadians through direct health measurements such as blood pressure, height, weight, and blood and urine samples. In addition, CHMS uses household interviews to gather information for other variables, including nutrition, smoking habits, alcohol use, medical history, current health status, sexual behaviour, lifestyle and physical activity, environmental and housing characteristics, as well as demographic and socioeconomic variables.

Objectives

CHMS has an important role in supporting the broader health agenda of the federal government.

The principal objective of the CHMS is to collect new and important data on Canadians' health status by

  • providing a platform and infrastructure for obtaining data and information through physical and laboratory measures to meet the emerging needs of several branches within HC and PHAC, as well as other add-on studies
  • collecting and disseminating direct health measures data, including those on environmental contaminants
  • promoting research using direct health measures data by providing access to nationally representative data.

Activities

CHMS is designed and implemented in consecutive cycles, each of which consists of three activities: planning, collection and dissemination. Table 1 shows the CHMS life cycle. The CHMS process map is presented in Appendix 1.

Table 1 Canadian Health Measures Survey main activities and related tasks, cycles 1 to 3
  Cycle 1 Cycle 2 Cycle 3
Planning Survey development 1. Identify survey content From 2003/2004 to 2007/2008 Jan. 2008 to Dec. 2009 Fall 2009 to Dec. 2011
2. Develop survey content
3. Develop software systems and tools
4. Develop protocols
Training 5. Train interviewers, health measuring specialists and clinic laboratory technicians
Data collection 1. Conduct interviews at home March 2007 to Feb. 2009 Sept. 2009 to Dec. 2011 Jan. 2012 to Dec. 2013
2. Conduct visits to the mobile clinic
3. Complete lab tests and return results to Statistics Canada March 2007 to June 2009 Sept. 2009 to April 2012 Jan. 2012 to April 2014
4. Store biospecimens in the Biobank March 2007 to Feb. 2009 Sept. 2009 to Dec. 2011 Jan. 2012 to Dec. 2013
Dissemination 1. Perform data processing and post-collection March 2007 to June 2009 Sept. 2009 to April 2012 Jan. 2012 to April 2014
2. Analyze and disseminate results Jan. 2010 to April 2011 Sept. 2012 to April 2013 To come
Infrastructure (enabling function) 1. Methodology Ongoing Ongoing Ongoing
2. System development
3. Other

Expected outcomes

The activities and outputs of CHMS are expected to lead to the following expected outcomes as specified in the Roll-up Evaluation Framework for the Action Plan to Protect Human Health from Environmental Contaminants.

Immediate outcome (2 to 5 years):

Reliable and usable data for decision makers, researchers and Canadians on the baseline health status of Canadians, and the level of, and exposure to environmental contaminants.

Intermediate outcome (2 to 5 years):

Increased awareness among data/ information users of the collected data/ information.

Long-term outcome (5 to 10 years):

Decision makers increasingly use the information on the associations between contaminants and illness to guide decision-making in public health practice, research, policy regulation and programs & services.

Leading to the ultimate outcome (10 to 20 years):

To reduce the health risks to Canadians, particularly vulnerable populations, from harmful environmental contaminants.

2.3 Governance

Statistics Canada's Policy Committee, headed by the Chief Statistician and supported by the agency's assistant chief statisticians, is responsible for the administration of CHMS under the Statistics Act. In addition to the agency-level approvals and processes, the Canadian Population Health Statistics Program (CPHSP) Committee is responsible for ensuring that the Population Health Program at Statistics Canada responds to the health needs of all Canadians. CPHSP comprises assistant deputy ministers from HC, PHAC, and Statistics Canada.

HC and PHAC's joint Research Ethics Board (REB) Footnote 1 helps ensure that CHMS meets the highest ethical standards, and that the greatest protection is provided to participants who serve as research subjects. It is guided by the principles of the Tri-Council Policy Statement, Ethical Conduct for Research Involving Humans, Footnote 2 which sets the standard for research ethics boards in Canada. The REB reviews and approves the procedures and general conduct of the survey prior to the start of each CHMS cycle.

A privacy impact assessment is presented to the federal Privacy Commissioner for every cycle of the survey in compliance with the Privacy Act. Further, provincial privacy commissioners are advised of the survey collection activities within their jurisdiction and steps are taken to protect the privacy of survey participants according to applicable provincial law.

Since the inception of CHMS, four advisory committees have provided expertise and advice on various aspects of the survey: the Expert Advisory Committee, the Physicians Advisory Committee, Laboratory Advisory Committee, and Quality Assurance and Quality Control Advisory Committee.

In June 2012, the Biobank Committee was created. The Biobank Committee is chaired by a director general and consists of four members internal to the federal government and four external members (e.g., academics specializing in genetics or environmental contaminants). It is responsible for granting access to Biobank specimens, which are part of the CHMS data. CHMS management has recently taken the initiative to re-align the structure of the advisory committees with respect to the current, more mature stage of the survey.

2.4 Program resources

The initial funding for CHMS was allocated in the 2003 federal budget as part of an extension of the Health Information Roadmap Initiative. In 2007/2008, CHMS was funded $975,860 through an agreement between Statistics Canada and HC. This funding supported the completion of the CHMS planning activity for Cycle 1, prior to the launch of the Action Plan in 2008/2009.

Under the Action Plan, the total investment ($84.6 million over a five-year period) is divided between HC, PHAC and Statistics Canada, as presented in Chart 1.

Chart 1 Federal partners in the Action Plan

Bar graph displaying resources allocated over the years
Description: Chart 1 Federal partners in the Action Plan

The source of funds attributed to Statistics Canada equals $54.5 million over a five-year period, which represents 64% of the total allocation of funds under the Action Plan. Based on this funding, CHMS was expanded to include additional measures on environmental exposure as well as direct health measures for children under the age of six.

The CHMS program received total core funding of $47.7 million for the period covered from 2008/2009 to 2012/2013. A total of $48.7 million has been allocated to CHMS from 2007/2008 to 2012/2013, inclusively. Footnote 3

CHMS also received external funds from HC, PHAC and other entities to support special requests for additional laboratory tests and analysis that were not planned in the core funding. From 2007/2008 to 2012/2013, a total of $37.6 million was allocated as external cost-recoveries.

Therefore, the total CHMS resource allocation under the scope of this evaluation (i.e., core funding plus external cost-recoveries) represents $86.3 million over a five-year period and is detailed in Chart 2.

Chart 2 Total Canadian Health Measures Survey resource allocations (core and external cost recovery), from 2007/2008 to 2012/2013

Bar graph displaying resources allocated over the years
Description: Chart 2 Total Canadian Health Measures Survey resource allocations (core and external cost recovery), from 2007/2008 to 2012/2013

2.5 Evaluation context, scope and issues

This report presents the results of the CHMS evaluation, which was conducted from April to October 2012. It provides credible and neutral information on the ongoing relevance and performance of Statistics Canada's CHMS activities outlined in the Action Plan, as well as necessary performance measurement information to support the horizontal evaluation. It also informs Statistics Canada's senior management on the overall performance of CHMS. The scope of the evaluation is from 2007/2008 to 2012/2013. It was expanded one year beyond the timeframe of the horizontal evaluation (2008/2009 to 2012/2013) to capture CMHS's Cycle 1, launched in 2007, because this was the only cycle fully completed at the time of the evaluation. This evaluation assessed the relevance and performance of CHMS by examining the five Treasury Board Secretariat core issues, in accordance with the Policy on Evaluation.

Table 2 Evaluation issues and questions

Relevance

  1. Is there a continued need for the CHMS initiative and is it responsive to the needs of Canadians? (Treasury Board Secretariat (TBS) issue 1)
  2. Is the CHMS program consistent with government priorities and Statistics Canada's strategic outcome? (TBS issue 2)
  3. Is there a legitimate, appropriate and necessary role for the federal government and for Statistics Canada in implementing the CHMS program? (TBS issue 3)

Achievement of expected outputs

  1. Were CHMS' design and implementation processes appropriate to produce the expected activities and outputs?

Performance – Effectiveness (TBS issue 4)

  1. To what extent has CHMS achieved its expected outcomes?
  2. Were there any factors (internal or external) that contributed or detracted from the achievement of those expected outcomes?
  3. Have there been any unintended impacts (positive or negative) from CHMS?

Performance – Efficiency and economy (TBS issue 5)

  1. Are sufficient human and financial resources in place to support the implementation of CHMS?
  2. What are the efficiencies that are being gained as a result of the Action Plan?
  3. Are there any duplication, complementarities and possible alternatives?

3. Methodology

The evaluation was conducted in accordance with the TBS Standards on Evaluation for the Government of Canada. To assess CHMS, evaluation issues and questions from the current Horizontal Evaluation Matrix for the Action Plan were slightly updated to better reflect the evaluation questions and indicators applicable to Statistics Canada. The CHMS Evaluation Matrix is presented in Appendix 3.

3.1 Approach and design

The approach used to evaluate CHMS can be defined as theory-driven and based on a non-experimental design using post-information for most of the evaluation questions. Findings are also collected from multiple lines of evidence. That is, more than one method was used to measure each of the evaluation indicators, thereby strengthening the validity of the findings. To that end, five methods (including qualitative and quantitative approaches) were selected to gather evaluation data:

  • a review of relevant documents and literature
  • a review of administrative data
  • a review of financial data
  • a series of key informant interviews
  • a survey of primary data users.

Each line of evidence used for the evaluation is described in the coming sub-sections. Calibration Footnote 4 played an important role in this specific evaluation, which will serve as an input into the horizontal evaluation of the Action Plan being conducted by HC.

3.2 Document and literature review

A review of program documentation provided a thorough understanding of CHMS and contributed to the design of the methodology for this evaluation, including the refinement of the data-collection instruments. The information gathered from document and literature reviews provided useful context for interpreting, confirming and supplementing information gathered through the other methods.

Because of time and budget constraints, the number of external documents reviewed was limited. Documents reviewed were, for the most part, based on publications in Health Reports, Statistics Canada's website and official documents, as well as external publications from other federal government departments.

A review of some published literature was undertaken to gather data to respond to evaluation questions, such as program relevance and lessons learned by other agencies delivering similar programs. A list of publications reviewed is provided in Appendix 4.

3.3 Administrative data review

The administrative data review was based on internal files, documents and records provided by the program area, as well as the minutes from the evaluation team meetings held from April to June 2012.

Administrative data on performance measures (e.g., the number of website hits, research data centre [RDC] requests for CHMS data, and client service requests for custom tabulation) were used to evaluate the availability and awareness of CHMS data among decision makers, researchers and Canadians.

3.4 Financial data

Budget and full-time equivalent (FTE) data were taken directly from the Financial Management System (FMS) to assess resource use in relation to the production of outputs and progress toward expected outcomes. Actual expenditures for CHMS core and cost-recoveries were also taken from FMS.

Capital asset and depreciation information was taken into consideration for financial analysis. This information was collected from a list of capital assets and depreciation provided by Financial Planning Division and was updated by the program area as of July 2012.

3.5 Key informant interviews and group interviews

Key informant interviews provided data on the perceptions and opinions of individuals (e.g., program managers and decision makers, as well as primary data users and researchers, both internal and external to the Government of Canada). The individuals have played a significant role, or had extensive experience with CHMS, as specified by the Roll-Up Evaluation Framework for the Action Plan to Protect Human Health from Environmental Contaminants.

Interview guides were designed to address all of the evaluation questions, and adapted to each audience, as required. Names of potential key informants for each group were identified through internal consultation. The method used to select key informants varied by group type. It was either a census or a sample based on the informant's number of years of experience and exposure to the data, and the informant's number of articles published using CHMS data or representation of a subject-matter area (e.g., environmental contaminants, tobacco, and infectious diseases). A total of 24 key informants were interviewed during June and July 2012. Table 3 indicates the planned versus actual interviews by informant group type.

Table 3 Planned versus actual interviews and group interviews
Group type Specifics Selection method Number of planned interviews or group interviews Number of actual interviews
Program managers Strategic program managers Census One group interview (two participants) One group interview (two participants)
Analytical program managers Census One group interview (three participants) One group interview (three participants)
Operational program managers Census One group interview (two participants) Two individual interviews
Chiefs Census One group interview, if time permitted (four participants) No group interviews
Decision makers (partners / cost-recovery clients) Cost-recovery clients and researchers Sample Three interviews Two interviews
One group interview with government researchers (four participants)
Primary data users and researchers Internal Statistics Canada researchers Census One group interview (six participants) Two group interviews (six participants in total)
External researchers and Expert Advisory Committee members Sample Five interviews Five interviews

3.6 Survey of primary data users

A list of primary CHMS data users was compiled based on the list of researchers who had access to the shared CHMS data files at HC and PHAC, and the list of researchers who had requested access to the data through RDCs. Footnote 5 RDC requests include researchers in other federal government departments and researchers external to the federal government. The 134 primary data users who were not selected for an interview received a survey questionnaire via email to complete either in June or July 2012. Of them, 24% were researchers from HC, 41% from PHAC and 35% were RDC researchers.

A draft survey questionnaire was developed to reflect the methodology embedded in the evaluation matrix. It was tested with two internal researchers via one-on-one interviews. As a result of the testing, the suggested corrections were implemented and the updated questionnaire was tested again with one external researcher via telephone interview, before being finalized.

The survey was in the field for two weeks. One reminder email was sent to non-respondents after the first week. In all, 30 people responded to the survey, which represents a 22% response rate. Among responses received, 26.7% were returned from HC (this represents 25% of their population), 36.7% were returned from PHAC (20% of their population), and 36.7% were returned from RDC researchers (23.4% of their population).

3.7 Limits of the evaluation

To the extent possible within the available budget and timeframe, the evaluation methodology incorporated multiple methods of data collection from different primary and secondary sources to ensure that the findings were valid and that they captured key points of view on CHMS. Whenever possible, the opinions and observations expressed by stakeholders were corroborated with evidence from program documentation and data.

However, because of contextual elements and resource constraints, there are a number of limitations associated with this evaluation. As specified earlier in the report, calibration played a crucial role (given that this is a part of the horizontal evaluation) and scope was defined accordingly to ensure an appropriate balance between level of effort and context. The following limitations should be taken into account when reviewing the findings from this report:

Timing of the evaluation – The timing of the evaluation limits the capacity to report on the program's outcomes because only Cycle 1 of CHMS had been completed. CHMS releases for Cycle 2 will be available by the end of September 2012, and Cycle 3 is collecting data. Therefore, the findings are limited by the degree of completeness of these cycles.

Measurement of outcomes – The budget for the evaluation along with the limited data available on expected outcomes limited the ability to report on the success of CHMS. Although a very specific performance measurement framework was outlined in the Action Plan for Statistics Canada, there is limited evidence that performance data were systematically collected on an ongoing basis according to this framework.

Interviews – A limited number of key informant interviews were conducted. Moreover, interviews with key informants and stakeholders did not include any independent respondents with no stake in the program. Therefore, there is a possibility that interview respondents had a positive bias toward the program. HC will have the opportunity to interview additional key informants during the roll-up evaluation.

Document and data review – Some challenges were faced when assessing the efficiency and economy of the CHMS program. Some of the factors that impacted the completeness and the conclusiveness of the evaluation evidence were the practice of compiling financial data by fiscal year and not by cycle, the lack of baseline data on cost and resources used by level output produced, and the lack of data on similar surveys.

Key findings

4.1 Relevance

4.1.1 Program responsiveness

Summary of Findings:

Evidence from program documentation, the literature and interviews points to the ongoing relevance of CHMS as well as to the continued need for the program. Moreover, CHMS is seen as being flexible enough to address emerging needs.

There is evidence that national baselines for environmental exposure have been established as a result of CHMS data, a first step toward identifying trends and consequent links between environmental contaminants and health in the future. Since long-term tracking is required to establish trends and to address potential implications, interventions, controls, etc., clear evidence related to outcomes needs to be collected overtime.

4.1.1.1 Continued need

According to documents reviewed, CHMS stems from a need to fill gaps in health information. It was developed to fulfill specific important needs, such as the need for

  • national baseline data on the extent of major health concerns (e.g., obesity, hypertension, vitamin and nutrition deficiencies, chronic and infectious diseases, oral health status of Canadians and the level of access to dental care)
  • national data on exposure and prevalence levels of environmental chemicals in the population
  • children's health information.

The principal objective of CHMS is to collect data on Canadians' health status by direct measures. While Statistics Canada has been collecting health status and related data for many years by other means (i.e., self-reported surveys and administrative data), these data were seen as generally limited in two important ways:

  1. Many kinds of data, such as blood pressure and physical fitness, simply cannot be ascertained in an interview; they require direct physical measurement.
  2. Health information derived from self-reporting or administrative records may be seriously biased for some variables. For example, one review Footnote 6 shows a consistent reporting bias of even simple measures like height and weight. This bias has the potential to misinform data users. Footnote 7, Footnote 8

"An incredible service to researchers is the fact that CHMS data provide a nationally representative "normal" sample across a broad age range, in both sexes that can be compared to special interest groups in research studies."

External data user

According to the health scientists and researchers interviewed, CHMS data are needed to validate and adjust self-reported data to increase research accuracy, since these data are based on evidence and are nationally representative.

"CHMS is used primarily as a surveillance mechanism to know the health of the nation, and in this role it provides the best data on physical activity, fitness, obesity, blood pressure, cardiovascular risk factors, etc. As much as it is important for us to know the extent of these health issues in Canada and how they change over time to help direct policies and priorities, the same importance is attributed to CHMS (being the only source of data we have on these things in Canada)."

External data user

Indeed, according to the Statistics Canada paper "Canadian Health Measures Survey: Rationale, background and overview" in the Health Reports special issue: Background Papers on the Canadian Health Measures Survey, efforts to correct for self-reporting bias are complicated by the probability that bias for some measures may be unstable over time and susceptible to media attention and social marketing campaigns, among other influences. Direct health measurements provide more robust, objective measures and allow for the assessment of variables that simply cannot be determined accurately through self-reporting (e.g., metabolic syndrome, environmental toxin exposure, lung function). Such data are needed for public health education, health promotion programs, health care planning, health surveillance and research.

In Canada, a number of program and policy initiatives that occurred from 2001 to 2006 further led to the need for surveillance and monitoring of public health indicators, thus providing a direct or indirect impetus for the creation and ongoing support of a direct health measures survey. Table 4 exhibits examples of such initiatives.

Table 4 Examples of initiatives that led to the increased need for a direct physical measures survey
Initiative Description
Chronic Disease Prevention Alliance of Canada (2001) Advocacy for integrated research, surveillance, policies and programs, and the resources needed to positively influence the determinants of health and reduce incidence of the chronic diseases that account for the largest burden of morbidity, mortality and cost in Canada, namely, cardiovascular disease, diabetes and cancer. Footnote 9
Building on Values: The Future of Health Care in Canada (Romanow Report) (2002) Report of the Commission on the Future of Health Care in Canada, which reported on consultations with Canadians on the future of Canada's public health care system and recommended policies and measures that offer quality services to Canadians and strike an appropriate balance between investments in prevention and health maintenance and those directed at care and treatment. Footnote 10
Review of Human Biomonitoring Studies of Environmental Contaminants in Canada 1990-2005 (2006) Provided strong evidence of a need for more comprehensive and intensive biomonitoring of environmental contaminants in Canada. Footnote 11

The need for a direct health measures survey is not unique to Canada. Several countries have a history of conducting surveys that include direct health measures and that have yielded important findings, which validates the need for this type of survey. For example, the U.S. National Health and Nutrition Examination Survey (NHANES), Footnote 12 conducted since the early 1960s, Footnote 13 has provided data to construct standard growth charts for children, thereby allowing doctors and parents to better understand developmental health trajectories. Footnote 14 In the 1960s, NHANES confirmed findings linking high cholesterol and heart disease. It also provided the first evidence that Americans had high lead levels in their blood, which motivated governments to phase out the use of lead as an additive in gasoline and paint. Footnote 15 In Australia, a direct health measures survey conducted from 1999 to 2001 found that for every known case of diabetes, there was one undiagnosed case, and that nearly one million Australians over age 25 have diabetes. Footnote 16 Finland, too, has a legacy of important public health and scientific findings from national direct health measures surveys. Footnote 17, Footnote 18, Footnote 19

In addition, the findings from the survey of CHMS primary data users also demonstrated and confirmed the current need for CHMS data for many purposes, such as trend analysis, monitoring and tracking changes, providing objective physical measures and lab data, providing a nationally representative sample, research, health impact assessments, diagnoses, prevention, informing policy, and policy development support. All survey respondents confirmed that there was a need to continue to collect and provide CHMS data on an ongoing basis.

4.1.1.2 Inclusion of environmental exposures in the CHMS scope

A growing concern of Canadians about environmental contaminants has been observed in the past decade. For example, a 2006 EKOS Research Associates Inc. survey showed that 89% of Canadians believe the health of their children is being adversely affected by environmental pollution. Nearly one in four Canadians has sought medical treatment for a condition that they believed to be related to the environment.

Two major federal government initiatives, the 2006 Chemicals Management Plan and the 2007 Clean Air Agenda committed to address issues related to environmental contaminants. The need for CHMS data is specifically relevant as the steps already taken by the government concerning these initiatives rely on CHMS to collect environmental contaminant measures. Both industry and non-governmental stakeholders supported these initiatives. They continue to insist that decisions be made on the basis of scientific evidence, and that Canadians have credible information on the impact of chemicals in the environment and the steps that they should take as a result.

A review of CHMS content demonstrates that the survey does address emerging environmental issues. Furthermore, it is evident that children under the age of six are represented in the sample. Table 5 provides an overview of the scope of the environmental exposure variables in the first three cycles of the survey. These data, according to a cost-recovery client, provide a baseline and the huge potential for identifying links between environmental contaminants and health; however, long-term tracking is required to establish such trends (some take up to 20 years to establish) and to address potential implications, interventions, controls, etc. Consequently, at this time such links are not possible to identify.

Table 5 Canadian Health Measures Survey environmental exposures: Number of variables measured, per cycle, per category and source
Description of the measures, per category and source Number of measures collected
Cycle 1 Cycle 2 Cycle 3
Variables measured at households
(fluoride; volatile organic compounds – common fuel pollutants [BTEX], trihalomethanes)
In tap water Not in scope Not in scope 11
Variables measured as part of the mobile examination clinic collection
(Aldehydes; aliphatic alcohols; aliphatic hydrocarbons; aliphatic ketones; aromatic compounds; chlorinated hydrocarbons; chlorbenzene derivatives; ethers; siloxanes; terpenes; others)
In indoor air Not in scope 102 81
Lab measures from human specimen
(Volatile organic compounds; acrylamine; benzene metabolites; carbamate insectides; chlorophenols; metals and trace elements; organophosphate metabolites; parabens; perfluorinated compounds; phthalate metabolites; polyaromatic hydrocarbons; pesticides; tobacco; herbicides)
Number of measures from human specimens
Blood 25 12 21
Urine 51 89 46
Total 76 101 67
Of them, measures taken from children under the age of 6
Blood 0 12 8
Urine 1 80 38
Total 1 92 46
4.1.1.3 Addressing current and emerging needs

When asked about emerging needs or information gaps that CHMS is expected to address, key informants and several documents identified a number of emerging needs that can be related either to new content themes or to the enhancement of the analytical power through the data. A summary of the most common suggestions collected are presented below.

New or enhanced content

  • Data on non-insured components of health (e.g., oral health, hearing, eyesight)
  • More comprehensive information on dietary and nutrition intake
  • Medication and dietary supplement intake
  • Bone health
  • Lung function
  • Noise exposure, expanded range of environmental exposure (more than 23,000 chemicals that need to be assessed and regulated)
  • Genetic testing, genetic links, personalized medicine, genome
  • Sleep
  • Specific measures (e.g., oral glucose tolerance test, cortisol [stress hormone])

Analytical capacity requirements Footnote 20

  • Record linkages
  • Longitudinal data
  • Geographic information
  • Data on sub-national level (e.g., regional, provincial, specific social groups, specific age groups, vulnerable populations)

Most key informants interviewed and data users surveyed feel that CHMS responds to current needs. As a result of CHMS data, Canada has objective, nationally representative baselines on important health measures, which have the potential to be used by doctors, health professionals Footnote 21 and researchers Footnote 22 as reference for the health characteristics of Canadians.

"It is a period [of eight years] that is not too far in the future to keep us too static, but also is not too small, to ensure some continuity and stability. It (the survey) will have core content that runs across the entire period; there will be rotating content that is going in and out on shorter time periods within these eight years, based on priority needs; and there will be a small capacity free so that we can, on a quick time line, add new content to address urgent emerging needs."

Internal interview

Further, CHMS is seen by interviewees as being able to address emerging needs, in terms of new content, because:

  • It provides an infrastructure that is comprehensive in scope, and a system of collecting direct measures that is diverse and flexible to accommodate the information needs of important health issues that may emerge in the future.
  • The content priorities are determined through the federal governance structure, following broad consultations with federal partners and other stakeholders; consequently, CHMS adjusts the survey content to be responsive to the needs indentified by them.

Although most feel that CHMS is responsive to current and emerging content needs, according to interviewees and documents reviewed, there was no content plan ever developed for CHMS to ensure the survey remains relevant and can continue to address emerging content needs.

With respect to the Biobank, Footnote 23 several researchers mentioned its significance, importance and usefulness, highlighting that it is the first Canadian nationally representative repository of biosamples. Internal and external researchers testify on the relevance of the Biobank, and that it provides huge potential for use in future research or surveillance of health issues, when a need arises.

Some analytical limits of CHMS persist because of the sample size. There are many examples where key informants specify that they need larger sample sizes for conducting more complex research and testify that this limitation impacts the use of the data for their own purposes. This aspect is presented in greater detail in the section on performance (validity, reliability and access).

4.1.2 Alignment with the government's and Statistics Canada's priorities and obligations

Summary of Findings:

Evidence from program documentation, the literature, and interviews all suggest that the CHMS objectives, under the Action Plan, are consistent with federal government priorities, and are aligned with Statistics Canada's outcomes.

Findings demonstrated that CHMS supports the federal government in fulfilling its legislative obligations, international commitments and its responsibility to protect the health of Canadians. Since these priorities are ongoing, there is still a legitimate, appropriate and necessary role for the federal government to continue CHMS.

4.1.2.1 Alignment with the government's priorities and Statistics Canada's strategic outcome

Government priorities

Collecting health data has been underscored as a federal priority on several occasions. To recall, the initial funding for the CHMS survey was outlined in the 2003 Federal Budget, where the Government of Canada highlighted the importance of "the availability of accurate and timely information on trends in health status and health system performance as a crucial tool to inform responsive, patient-centered health policy decisions." Moreover, investing in health and safety to provide Canadians with better information on the links between pollution and illness was reaffirmed in the 2008 Federal Budget.

The alignment of CHMS with federal government priorities is ensured by its governance at the strategic level. CHMS is one of several Statistics Canada health information activities under the CPHSP and, as such, its strategic priorities and decisions are determined by the CPHSP tri-partite steering committee representing HC, PHAC and Statistics Canada. By its mandate, the CPHSP committee is responsible for ensuring that federal government health information needs are addressed.

Document reviews revealed evidence that CHMS supports the federal government in fulfilling its legislative obligations, its international commitments and its responsibility to protect the health of Canadians. For example:

  • CHMS data inform and support the federal government in its ongoing monitoring, surveillance, risk management and regulating activities, which are required by multiple legislative acts such as the Health Canada Act, the Canadian Environmental Protection Act, the Pest Control Products Act, the Food and Drugs Act, and the Hazardous Products Act.
  • CHMS information helps to address the federal government's international commitments and obligations, such as the North American Free Trade Agreement, the Canada–U.S. Great LakesWater Quality Agreement, United Nations negotiations on the Global Treaty for Mercury, the Stockholm Convention on Persistent Organic Pollutants.
  • CHMS data inform the creation and updating of many federal government standards and guidelines related to health and environment (such as Guidelines for Canadian Drinking Water and the Federal Tobacco Control Strategy). The survey also provides essential information in support of several broader federal government initiatives, such as the Chemicals Management Plan, the Health Information Roadmap Initiative, and the Clean Air Agenda.

Statistics Canada's strategic outcome

Supporting CHMS and other projects that provide statistical information and analysis about the state of Canadians' health is set out as one of Statistics Canada's activities in the 2011-2012 Report on Plans and Priorities. The CHMS also constitutes one of the programs designed to achieve Statistics Canada's strategic outcome as defined in the agency's 2011 Program Activity Architecture:

Canadians have access to timely, relevant and quality statistical information on Canada's changing economy and society for informed debate, research and decision making on social and economic issues.

In summary, the work of Statistics Canada to conduct CHMS is consistent with the agency's priorities and supports the achievement of its strategic outcome. It also supports the achievement of the Action Plan's objectives by increasing the knowledge base on contaminant levels and potential impacts on health, and consequently contributes to the Action Plan's ultimate outcome (i.e., reducing the health risks to Canadians from harmful environmental contaminants).

4.1.2.2 The federal government's and Statistics Canada's roles in implementing the Canadian Health Measures Survey program

"Given that Statistics Canada is the mandated statistical body for the country it makes sense to do that work and make data available."

External researcher

The Constitution Act, 1867 establishes "census and statistics" as an area of federal jurisdiction. Parliament has exercised its responsibility for the census and statistics primarily through the Statistics Act. The act creates Statistics Canada as Canada's national statistical office and establishes its mandate, powers and obligations. Under the act, Statistics Canada must collect, compile, analyze and publish statistical information on the economic, social and general conditions of the country and its people. Under Section 22-c of the Statistics Act, Statistics Canada has the mandate to collect, compile, analyze, abstract and publish statistics on health and welfare.

"Statistics Canada has the infrastructure and the reputation of trust, confidentiality and neutrality to achieve the high response rates and thus produce high quality information."

Researchers

Key informants indicated that while health is a provincial jurisdiction, there is an important role for the federal government to provide credible, nationally representative survey measures for comparability, consistency, and objectivity. According to them, "there is nobody in a better position than the federal government to create and maintain a statistical program of the magnitude, visibility, and complexity of the CHMS. Internationally, similar programs are also delivered at a federal level."

According to senior management, CHMS was designed in response to a data gap on the general health of the population, based on direct health measures. If CHMS no longer existed, the impact would represent a step backwards in how we understand health in Canada, especially for children aged 3 to 11, as there are currently no data available for this age group. This evaluation finding is reaffirmed by interviewed researchers. Without CHMS, researchers will lose the ability to adjust self-reported data based on objective direct measures. In health policy, a lack of CHMS data may impair decision making and program outcomes, as CHMS provides the clearest picture available of health issues and their determinants for Canadians.

4.2 Performance

4.2.1 Effectiveness – Achievement of outputs and expected outcomes

The expected outcomes measured in this evaluation are specified in Statistics Canada's Action Plan to Protect Human Health from Environmental Contaminants Roll-up Evaluation Framework. The Results-based Management Accountability Framework established in 2008 for the Action Plan presents a timeline to reach each outcome. The evaluation findings have focused on measuring expected outcomes at the immediate level because of CHMS's life cycle and the timing of this evaluation. However, there is some evidence demonstrating that CHMS is on track and progressing toward its intermediate and long-term outcomes.

Prior to assessing if outcomes have been achieved, it is important to understand the delivery of the outputs as stated in the Action Plan, as they are the catalysts through which expected outcomes are achieved.

4.2.1.1 Outputs

Evidence indicates that CHMS activities are carried out as planned, and the CHMS outputs have been produced as expected.

The status of CHMS activities, as of August 2012, has already been presented in Table 1. As evident, Cycle 1 is fully completed, Cycle 2 is in its final dissemination phase, and Cycle 3 collection is in progress.

Based on administrative data and interviews with program management, the evaluation findings demonstrate that CHMS outputs were produced as planned. Indeed, the Action Plan stated that CHMS should have 5,000 participants in cycles 1 and 2 and 2,500 participants in Cycle 3 (at the time of the evaluation), and that children under the age of 6 should be included in the sample. In terms of measures, the Action Plan required additional environmental measures, including air and water samples.

Table 6 depicts the completion status of the outputs and demonstrates that the targets outlined in the Action Plan were achieved.

In addition, as observed by reviewing the administrative data, biomonitoring data were collected from the 3-to-5 age group, predominantly in cycles 2 and 3. There were 135 biomonitoring measures Footnote 24 taken from children under the age of 6 in Cycle 2, and there are 88 such measures currently being collected from this age group in Cycle 3, compared with only 1 measure taken from this age group in Cycle 1.

Moreover, a range of environmental exposure measures have been collected, including biospecimens (all three cycles) and indoor air samples (cycles 2 and 3). Also, a new source of measures within the environmental exposure theme has been included in Cycle 3: tap water. Footnote 25

Table 6 Status on outputs, as of August 2012
Activity Outputs Cycle 1 Cycle 2 Cycle 3
Status (quantity) Status (quantity) Status (quantity)
Collection   Planned number of participants: 5,6001 Planned number of participants: 5,700 Planned number of participants: 5,700
Household questionnaires Completed (6,604) Completed (7,830) In progress
Physical measures data Completed (5,617) Completed (6,432)
Physical activity monitor Completed Completed
Biospecimens Blood Completed (5,385) Completed (6,163)
Urine Completed (5,552) Completed (6,363)
Biobank samples Collected Collected
Indoor air samples Not applicable Completed
Tap water samples Not applicable Not applicable
Dissemination Dissemination plan Available completed Available completed Too early
Daily releases 10 In progress
Microdata files 10
Custom tables Approximately 40
Peer-reviewed articles in Health Reports 23
Peer-reviewed articles in other journals Approximately 30
Reports 5
Fact sheets 14
Summary tables 65
4.2.1.2 Immediate outcome

Reliable and usable data for decision makers, researchers and Canadians on the baseline health status of Canadians, and on the level of, and exposure to environmental contaminants.

Evidence indicates that CHMS data are reliable and usable, and that they are accessible through shared files with partners, as well as through Statistics Canada's RDCs.

Through the evaluation study, users expressed some concerns with the comparability and analytical power of CHMS data, and shared their challenges with the availability of the data.

Reliability and usability

Most of the key informants interviewed stated that CHMS data were reliable and robust. Three-quarters of the survey respondents indicated that they were satisfied or very satisfied with the data's reliability and validity.

Furthermore, evidence from administrative data demonstrates a high response rate for Cycle 1 (88% for the household questionnaire and 85% for the mobile clinic) and for Cycle 2 (90% for the household questionnaire and 82% for the mobile clinic), which is a valid indicator of the quality of the information produced.

Although researchers indicated that they were satisfied with the reliability and usability of CHMS data, they raised the following two concerns: 1) comparability of the data and 2) inability to conduct analysis at the sub-national level.

Concern 1 – Comparability of data

According to interviewees, there are many valid ways to measure health indicators and environmental contaminants, but if they are not measured the same way from one cycle to the next or from one survey to the next, they are difficult to compare for analytical purposes. For example, CHMS measures cadmium (or caffeine) in urine, while NHANES measures it in blood—both are good measures, but are not comparable.

The following possible causes for measuring things differently were mentioned by researchers: 1) laboratory measures are victim to various laboratory methods, measurement approaches, measurement errors and changing measurement protocols and 2) competing interests and money.

Concern 2 – Inability to conduct analysis at the sub-national level

Most researchers mentioned that because of the size of the sample, they were unable to conduct analysis at the sub-national level or more complex analysis on specific subsets of the population (e.g., vulnerable populations, people with rare conditions). This, in turn, limited the reliability and the usability of the data. However, the impact of the limitations is not consistent across content themes (e.g., no issues with physical activity were mentioned).

According to program documentation, CHMS was established to be a national survey and therefore provide national estimates. Conducting analysis at a sub-national level was not in scope for the initiative.

Availability

"The RDCs are providing such a fabulous service to researchers. But one has to really know what to do; it is not a fishing expedition. If they know what they want to do they get support."

Internal manager

Evaluation findings revealed that data are not as accessible as they could be. Some of the comments collected are related to the unavailability of microdata files for CHMS on the Internet and the fact that researchers must go to RDCs to use CHMS data files. In terms of RDCs, internal and external researchers raised various concerns, such as a lack of awareness of how to access the data; timelines to get proposals approved (sometimes up to six months); issues with location and times (days and hours) of use; restrictions on material to be brought in and out; and, in some locations, the ability and knowledge of staff to work with the CHMS data files. Footnote 26

Findings from internal interviews revealed that no issues were identified by CHMS management regarding the availability and accessibility of CHMS data. According to management, the two federal partners have shared files and external researchers have full access to the RDCs. As for the primary data users surveyed, almost three-quarters confirmed this evidence and answered that they were satisfied to very satisfied with awareness on availability. Moreover, CHMS management also stated that a communication strategy will be launched in fall or winter 2013 to make sure that all information about accessing CHMS data and Biobank data is available and visible to the public. Because of the timing, however, it was not possible for the evaluation to validate this statement.

4.2.1.3 Intermediate outcome

Increased awareness among those who use the collected data and information

Evaluation findings demonstrated evidence of awareness among the general public, researchers and data users as a result of publications based on Cycle 1 data. However, it is still too early to present valid measures of increased awareness as the release of Cycle 2 data and its related products is currently in progress. Furthermore, some issues such as the 'reach' of publications and the lack of promotion of CHMS data to scientific communities were identified as limitations to awareness, in addition to the timing of this evaluation.

According to CHMS administrative data, the general public are interested in Cycle 1 data, as seen in the number of published newspaper articles, TV and radio broadcasts, Internet articles, and website hits, presented in Table 7.

Table 7 Media uptake, January 2010 to March 2012
Type of media Total from Cycle 1
number
Newspaper articles (printed) 171
TV and radio broadcasts 37
Internet articles 248
Twitter entries (tracked only for the first week after the release) 63
Other articles 463

As Table 8 summarizes, the general public has also shown an interest in and awareness of the CHMS products available on the Statistics Canada website.

Table 8 Website hits, April 2009 to December 2011
  Total from Cycle 1
number
Total (English and French) 184,412
For summary tables only 6,973
For metadata (dictionaries, etc.) 7,216
Data table downloads1 3,519
Data tables page views1 13,858

Finally, Table 9 indicates that external researchers have expressed an interest in Cycle 1 data based on the number of research proposals submitted (64), inquiries to client service areas and requests for custom tabulations.

Table 9 Requests for data, inquiries and custom tabulations
  2008/2009 2009/2010 2010/2011 2011/2012
number
Requests received for Biobank use for research studies ... ... ... 6
Research proposals using CHMS data (university and federal RDCs) ... 4 28 32
Inquiries and requests addressed by client services (from January 2010 and December 2010) ... ... 194 ...
Custom tables prepared based on requests (from January 2010 and December 2010) ... ... 115 ...

Although all three tables demonstrate that there is interest in, and awareness of CHMS data, it is difficult to quantify the level of awareness. These numbers, however, can form the baseline against which to measure increased awareness as future CHMS data cycles become available and products are released.

According to all key informants, awareness of data availability exists among researchers to varying extents. It was also noted that the data are still new (Cycle 1 data were released from January 2010 to April 2011) and that, as more publications come out, awareness should increase. However, even with these efforts, interviewees indicated that there may continue to be some issues in increasing awareness because of the 'reach' of the publications and the lack of active promotion of the data to scientific communities.

4.2.1.4 Long-term outcome

Decision makers increasingly use the information on the associations between contaminants and illness to guide decision making in public health practice, research, policy regulation, and programs and services.

The evaluation findings demonstrated evidence that CHMS has already played a role in validating data from other self-reported surveys as well as been used in research and for scientific discovery. More specifically, there was evidence of use for policy and decision making in the areas of physical activity, environmental exposure, nutrition markers and oral health; however, it is still too early in the program's life cycle to demonstrate CHMS's full potential.

Despite the many examples of use, the limited accessibility of the data has been raised by key informants as a potential factor that could detract from its increased use (CHMS expected outcome).

Evidence of use

Canadian Health Measures Survey data used within Statistics Canada

Internal researchers highlighted specific areas where CHMS data are used within Statistics Canada. For example, CHMS helps validate other self-reported surveys, such as the Canadian Community Health Survey. It also helps improve micro-simulation models (by increasing the confidence of projections), which are developed by Statistics Canada and used by policy makers for creating or updating policies.

Canadian Health Measures Survey data used in decision making

External researchers provided examples from their area of expertise of where CHMS data have been used. These examples mainly reflect CHMS's contribution to

  • determining priorities and pressing issues
  • managing risk within HC and PHAC
  • making strategic and financial decisions
  • planning for the future
  • updating guidelines and regulations
  • making international comparisons
  • informing global negotiations.

In the same vein, almost all primary data users from the evaluation survey had used CHMS data for research purposes. Of them, 50% had used CHMS data for policy analysis, and some of them had used the data for program and service development.

Table 10 depicts the most significant examples to date of CHMS data and research used to influence policies and regulations in four thematic areas: environmental exposure, physical activity, nutrition markers, and oral health.

Table 10 Evidence of use of Canadian Health Measures Survey data
Theme Evidence of use Organization and year of publication
Environmental exposure Report on Human Biomonitoring of Environmental Chemicals in Canada Health Canada (HC), 2010
Physical activity The Canadian Physical Activity Guidelines HC and Canadian Society for Exercise Psychology, 2011
Canadian sedentary behaviour guidelines for children and youth
Active Healthy Kids and Canada Report Card on Physical Activity for Children and Youth National organization Active Healthy Kids Canada, 2011, 2012
Nutrition markers Updated guidelines related to folic acid intake in women of child-bearing age, based on results of Vitamin B12 and red blood cell folate Society of Obstetricians and Gynecologists of Canada, 2011
Clinical Utility of Vitamin D Testing: An Evidence-based Analysis (report) Ontario Medical Advisory Secretariat, 2009
Vitamin D data informed the debate regarding the update of the recommended intake guidelines Institute of Medicine, USA, 2011
Oral health Summary Report on the Findings of the Oral Health Component of the Canadian Health Measures Survey 2007-2009 HC, 2010
Canada's Oral Health Report Card: A call to action Canadian Dental Hygienists Association, 2010
Oral Health – More than Just Cavities (report) Ontario's Chief Medical Officer of Health, 2012

The selected examples of use are drawn from a very short period (approximately 2.5 years since the data were released). Key informants underlined that "it is a very new survey on the grand scheme, having been around for such a short period of time." Nonetheless, they believed that CHMS's reputation is high and it will have a positive impact greater than any other national survey. They testify that "in various meetings they attended, under various hats, in the physical activity sector, or health sector, the survey comes out and is being mentioned as a useful and respected source of information," that it generates a lot of attention around decision-making tables at HC and PHAC, and that, even though "they might not be in a stage to have a published document per se, [..] they are certainly making decisions based on it."

4.2.1.5 Potential factors that contribute or detract from the achievement of expected outcomes as identified by key informants

Accessibility as a barrier to use

"There is wonderful data collected, the survey should be proud, but if it stays inside it is useless to advance science, inform policy, etc."

External researcher

The evaluation findings revealed some issues with access to the data that may lead to adverse effects on their use for research and policy development. Specifically, though most researchers are aware of the data, many are unaware of how to access them. This view is corroborated by external interviewees who believe that the lack of accessibility (or complexity of access) is a barrier to the use of the data.

Documents reviewed revealed that an analytical plan, a strategic document describing the CHMS analytical program and dissemination strategy, was developed in 2005 and updated several times—the last time in 2010. This plan was intended to be a living document that would facilitate the optimal use of CHMS data and define the scope of the analysis to maximize the data's impact and utility. The plan outlines the analytical objectives and the analytical activities related to the survey, including those that support them.

According to external researchers, most of the deliverables that were specified in the plan were completed, but the interviewees were presently unaware of the current analytical plan. They felt that there should be better communication and better collaboration to avoid duplicating research and analysis should different research groups look at the same data.

In the same vein, the evaluation study found that the program does not formally and systematically track all publications that use CHMS data, and interviews with external researchers confirmed that they do not always inform CHMS management about their peer-reviewed publications. This fact contributes to the previously identified weaknesses with regard to coordination and better communication of analytical efforts among researchers and other interested parties. The need for comprehensive, formal and up-to-date information on all studies and publications that are based on CHMS data was clearly identified by several external interviewees as a prerequisite for good collaboration. Researchers would know whether someone has already studied and published results on their potential focus of interest before starting their studies; thus, "they would be able to plan and scope their research better."

According to CHMS management, starting this fall, CHMS will be part of the Health Statistics Division analytical planning process, which is conducted every year. The analytical plan will be produced based on consultations with key stakeholders. As part of this planning exercise, decisions will be made about future articles and shared responsibilities between Statistics Canada, HC, PHAC and external experts.

Other factors

Budgetary restrictions and financial pressure were specified by most of the interviewees as a potential factor that could detract from the achievement of expected outcomes. External researchers (cost-recovery clients) specified cuts to cost-recovery funding, which would mean shorter content and fewer measures.

Program management mentioned that the sustainability of the clinic staff and the knowledge that resides in a few key people could potentially affect the delivery of CHMS. In fact, they are currently developing a new human resources strategy for the mobile clinic staff and are trying to ensure that knowledge transfer occurs. It was also specified during interviews that they are working on a succession planning strategy for mitigating the risks associated with the lack of knowledge transfer; however, no evidence became available within the timeframe of the evaluation study.

One external researcher mentioned a possible "perception danger" as a factor that might affect the survey. He feared that because a trend cannot be observed at the present time, it may be perceived that the measures are not worth the money. He explained that some trends can take a very long period of time (e.g., 20 years) to emerge.

4.2.1.6 Unintended impacts from the implementation of the Canadian Health Measures Survey

There were no significant unintended impacts revealed by this evaluation. Given the short period of time CHMS has been in place, it may be too early to observe unintended outcomes.

According to program management, however, since the survey was modeled after NHANES—a survey that has existed for the last 50 years—there was an opportunity to take into consideration their experience and lessons learned, which, in turn, decreased the likelihood of unintended impacts. According to the internal interviewees, the complexity of the survey and the required extensive and thorough planning process to ensure the controlled environment of operations and quality of data further limits the likelihood of variations and unintended effects occurring and being observed.

In terms of positive impacts, a couple of examples were mentioned by interviewees: 1) some individual medical problems were identified and respondents were notified as soon as possible, which allowed them to seek advice from their doctor in a timely manner; 2) some environmental contaminant data that were collected as supplementary became needed earlier than expected. For example, measures of selenium were included in the metal profile along with other planned measures, because it was easy to include them with the same collection and laboratory efforts. Recently, they became needed for a selenium risk assessment that was not in the plans during collection, as nationally representative data on selenium.

4.2.2 Performance - Efficiency and Economy

The evaluation findings on the efficiency and economy of CHMS are based on financial analysis of resource availability and use, and trend analysis of expenditures and variances. Footnote 27 Anecdotal information obtained through interviews provides key informants' views and perceptions regarding the efficiency of the program, the alternative approaches that might increase efficiency, and the extent to which CHMS duplicates or complements other surveys or existing programs.

The shift from predominantly cost recovery to core funding of CHMS as a result of the 2008 Action Plan helped stabilize the survey, which had a positive impact on CHMS. In particular, it allowed for better long-term planning and for seeking opportunities to increase operational efficiency, which could be demonstrated more clearly after the implementation of performance data collection on efficiency over time.

The use of existing CHMS infrastructure by other federal partners allows them to achieve their results while reducing time and costs. Sharing complementary knowledge between Statistics Canada and other federal partners and academia increases the analytical capacity in the health domain.

CHMS is the only nationally representative direct measures survey in Canada and complements other Canadian and international studies. Therefore, there is no duplication of effort that might influence efficiency.

External interviewees identified areas that could increase efficiency and economy, and suggested potential alternative approaches to some aspects of conducting the survey (e.g., flexible hiring practices and the use of local clinics and infrastructure instead of mobile examination clinics). However, these options could impair the quality of data collection.

4.2.2.1 Resources in place to support the implementation of the Canadian Health Measures Survey

"The dedicated funding (until this fiscal year) helped tremendously to establish CHMS. We now have a nice work flow that is consistent and predictable as opposed to being at the mercy of renewal of the funding every year. We were able to plan better long-term."

Internal interviewee

Evidence shows there was a clear shift from cost recovery to core funding after the 2008 Action Plan, as detailed in charts 3 and 4 below. Cost recovery changed from 90.6% in 2007/2008 to 24.2% in 2011/2012; core funding changed from 9.4% in 2007/2008 to 75.8% in 2011/2012. Many interviewees confirmed the Action Plan's positive impact by stabilizing the funding for CHMS.

 

Chart 3 Core funding, as of April 2012

Bar graph displaying resources allocated over the years
Description: Chart 3 Core funding, as of April 2012

Chart 4 External cost-recovery funding, as of April 2012

Bar graph displaying resources allocated over the years
Description: Chart 4 Total Canadian Health Measures Survey resource allocations (core and external cost recovery), from 2007/2008 to 2012/2013

All variances between CHMS's current budget and actual expenditures for both core and cost-recoveries are less than 4% over the five-year period. Financially, this indicates good resource management. Internal interviewees revealed that this could be attributed to the careful and thorough planning required by the complexity of the survey.

In terms of number of FTEs, the budgeted number almost always underestimated the actual number required, as detailed in Table 11. The effect was that non-salary funds had to be re- profiled to salary funds.

Table 11 Number of full-time equivalents
  Full-time equivalents
2007/2008 2008/2009 2009/2010 2010/2011 2011/2012 2012/2013
number
Core budgeted 0 13 54 84 78 ...
Core actual 2 17 93 96 119 ...
Cost-recovery budgeted 57 77 0 0 0 ...
Cost-recovery actual 69 74 0 0 0 ...

The trade off of cost versus quality

Analysis of financial data on operational expenditures (e.g., equipment and assets) revealed that maintaining the mobile examination clinics (MECs) (e.g., the medical and lab equipment, as well as the trained and knowledgeable staff who travel with the clinics) had accounted for the program's most significant costs. When looking at the assets, it is noticeable that the most significant categories are medical and laboratory equipment—the approximate cost of one MEC is $335,000, with 35,000 for the trailer and $300,000 for the equipment inside. It is also clear that the MECs currently in operation are almost at the end of their life cycle (their overall percent value is 10%). Consequently, finding opportunities to optimize the cost and resources used in MEC operations could be significant for CHMS's performance in terms of efficiency and economy.

"If we took only one of the objectives (say, biomonitoring only), you would not necessarily need mobile clinics across the country. However, if you are interested in having this comprehensive view of the overall health of Canadians and want to correlate it with lab measures, the way the survey is designed is the most efficient way to achieve this consistency and the comparability of the results over time."

Interviewee

Program management and some experts admit that the cost of collecting direct measures through MEC is high. However, they feel that controlling the MEC environment is the key to ensuring high-quality results. Even a simple inconsistency in the environment—for example, odours—could affect the measures. In the end, data quality was paramount, and directly linked with high costs.

Furthermore, interviews and document review indicated that the method of collecting health measures in mobile clinics, although expensive, is the one used by other countries and surveys (for example, NHANES). The decisions made indicated that the model selected aimed to have the same level of rigor and consistency.

However, some external researchers, while recognizing the importance of controlling the environment around data collection and keeping it consistent, still believe that other alternatives, such as the use of local existing facilities and structures, is worth further consideration for the purpose of increasing efficiency without compromising quality. They also indicated that the program's efficiency and responsiveness may be improved by using more flexible hiring procedures (e.g., hiring people external to the public service, whenever possible).

Evidence of the program's effort to generate efficiencies

Interview findings indicated evidence of the program's effort to apply good stewardship and to use resources in an economic way. For example, when selecting locations for trailers (e.g., using federal or public locations for a nominal fee over private, for-profit locations to minimize rental costs), and in contracting out some services, such as maintenance and IT support, to local companies rather than sending maintenance personnel from head office. As specified by the program managers during interviews, they are trying to minimize the overtime and travel costs of personnel.

In addition, as the program matured, its operating structure was re-organized through mergers. This generated cost-savings through lower salary expenditures, while at the same time providing a more efficient management environment. The current CHMS management continues to look for internal operational efficiencies through regular operational reviews.

4.2.2.2 Possible efficiencies that are being gained as a result of the Action Plan

Internal and external interviewees agree that the CHMS infrastructure provides to HC and PHAC the opportunity to reduce their program costs. They can invest their funding in additional health measures rather than in developing their own parallel infrastructure.

"We are the national statistical agency and we are working well within our national mandate. They are using existent, well established, highly functioning national infrastructure. That is being of itself one definition of efficiency."

Internal interviewee

CHMS worked in collaboration with HC's First Nations and Inuit Health Branch (FNIHB) when biomonitoring the First Nations population, which is part of FNIHB's responsibilities under the Action Plan. According to CHMS program management, using comparable methods, procedures, tools, and reference labs created efficiencies for FNIHB at the operational level, but also provided better comparability of the results because the same measures were used in First Nations population as in the general Canadian population. However, program management believes that there is room for even more collaboration between the two initiatives.

Internal and external researchers suggested that a collaborative approach in research provides the best opportunities to combine the strengths of all parties involved, and to increase the knowledge capacity and the analytical power of the studies. Statistics Canada's researchers can leverage their technical capacity, including easy access, manipulation and statistical interpretation of the data. External researchers tend to be the subject-matter experts and thus, they are well positioned to interpret, produce and share results with the scientific community. Through this collaboration, CHMS contributes to the increased analytical capacity in the health domain.

4.2.2.3 Possible duplications, complementarities or alternatives

Evaluation findings revealed that most key informants believed there is no duplication of CHMS with other provincial or international surveys. In fact, right now in Canada there are no other nationally representative direct health measures survey data. While several provincial surveys do exist (e.g., Ontario Health Survey), most of them do not include physical measures or they use self-selected sampling, which, according to program management, could cause bias issues (e.g., only healthy people may choose to participate). Internationally, many countries (approximately 20) are conducting similar programs, and, according to internal management interviews, there are efforts to compare the results of individual countries.

As previously mentioned, evidence has also pointed to the fact that CHMS plays a role in providing complementary data to other kinds of surveys (e.g., self-reported, administrative) collected in Canada, as it helps to validate and adjust them.

Although all key informants believed that CHMS should be funded by the federal government, some external researchers did raise the question of whether the federal government should be conducting all activities versus contracting out some of the work. Alternative models for program delivery were suggested, such as completing some of the CHMS work by non-federal government contracts; however, this alternative may impact respondents' level of trust in CHMS, as well as CHMS's response rate and data quality.

5. Conclusions

5.1 Relevance

CHMS represents a unique program in Canada, providing direct health measures data to support health research, policy, and decision making. Evidence shows that the program is relevant to Canadians and health organizations, with a clear present and future need for the program. Although most feel that CHMS is responsive to current and emerging content needs, there was no content plan ever developed for CHMS.

CHMS is well aligned with the priorities of the federal government and Statistics Canada. The federal government is in the best position to deliver CHMS, and has specific legislative obligations being met through the program, making the delivery of the program a legitimate, appropriate and necessary activity.

5.2 Performance – Effectiveness

CHMS outputs have been produced as expected and the expected immediate outcome is being achieved: reliable and usable data are available on the baseline health status of Canadians, and on the level of, and exposure to environmental contamination. Some issues exist around the accessibility of the data, which could impair the long-term outcomes if not addressed. More specifically, even though most researchers are aware of the data, many are unaware of how to access them. This view is corroborated by external interviewees, who believe that the lack of accessibility (or complexity of access) is a barrier to the use of the data. Some key informants commented on the unavailability of microdata files for CHMS on the Internet and on how researchers must rely on the RDCs to use CHMS data files. Internal and external researchers raised various concerns related to RDCs, such as a lack of awareness of how to access the data, and timelines to get proposals approved (sometimes up to six months).

Evaluation findings demonstrated evidence of awareness of CHMS data among the general public, researchers and data users as a result of publications based on Cycle 1 data. However, issues such as the 'reach' of publications and the lack of promotion of CHMS data to scientific communities were identified as limitations to awareness, in addition to the timing of this evaluation. Researchers felt that there should be better communication and better collaboration to avoid duplicating research and analysis should different research groups look at the same data.

The evaluation findings provided evidence that CHMS data are being used for research, policy, and decision making—the expected long-term outcome. Researchers have already used CHMS data to validate self-reported data from other surveys, as well as in research for scientific discovery. In addition, CHMS data are used for policy and decision making in the areas of physical activity, environmental exposure, nutrition markers and oral health. However, it is still too early in the program's life cycle to broadly demonstrate CHMS's full potential. Information on publications (research studies) are only tracked internally and not shared with external stakeholders, making the collaboration and reporting, in some instances, more difficult.

5.3 Performance – Economy and efficiency

The shift from predominantly cost recovery to core funding of CHMS as a result of the 2008 Action Plan helped stabilize the survey, which had a positive impact on CHMS. In particular, it allowed for better long-term planning and for seeking opportunities to increase operational efficiency. The evaluation results demonstrated sufficient human and financial resources to support the program. Some external parties have indicated that increased efficiency and economy could be achieved through more flexible hiring practices and the use of local clinics and infrastructure as an alternative to the mobile clinics. While these options might be worth further consideration, their feasibility must be assessed against the extent to which they could impair the quality of data collection.

Furthermore, the use of existing CHMS infrastructure by other federal partners allows them to achieve their objectives in an efficient way while reducing time and costs. Sharing complementary knowledge between Statistics Canada, other federal partners and academia increases the analytical capacity in the health domain.

CHMS is the only nationally representative direct measures survey in Canada and complements other Canadian and international studies. Therefore, there is no evidence of duplication of efforts that might influence efficiency.

6. Recommendations

Three recommendations emerge from the evaluation findings that advocate for enhanced planning and coordination with various stakeholders with respect to content and analytical studies, improved accessibility of CHMS data, and strengthened abilities for systematic performance measurement to demonstrate the achievement of results.

Recommendation 1 – Relevance

It is recommended that the management of CHMS enhance planning and external coordination with researchers and stakeholders—both internally within the federal government and externally with researchers from academia—with respect to content determination (content plan) and the planning of analytical research based on the content.

Recommendation 2 – Performance

It is recommended that the management of CHMS increase awareness of the data and improve their accessibility by promoting CHMS to a wider audience of potential clients and users, and by providing support on how to use the data.

Recommendation 3 – Performance measurement

It is recommended that the management of CHMS improve the performance measurement system, as a tool, to systematically collect performance data and monitor the progress toward achieving its outcomes.

3a) Improving data

To accurately assess the achievement of the long-term outcome and the impact of CHMS, it is recommended that management of the CHMS ensure that a formal tracking, such as the Client Relationship Management System (CRMS), is in place for publications and studies based on CHMS data, to demonstrate their use. The information on publications using CHMS data must be shared on a recurrent basis with researchers and stakeholders to enhance further external coordination.

3b) Financial information

It is recommended that accurate financial information is made available by the Finance Branch to support CHMS performance data collection to demonstrate the level of efficiency over time for decision making and accountability reporting.

7. Management response and action plan

Recommendation 1

Relevance:
It is recommended that the management of CHMS enhance planning and external coordination with researchers and stakeholders—both internally within the federal government and externally with researchers from academia—with respect to content determination (content plan) and the planning of analytical research based on the content.

Statement of agreement or disagreement

Management agrees with this recommendation.

Management Response

The evaluation reinforced management's strategic objective this fiscal year to develop a longer-term content plan for CHMS to ensure ongoing relevance. This work has already begun and will be completed within the next 12 months. In addition, the evaluation provided evidence that better planning for analytical work across all three federal departments—Statistics Canada, PHAC and Health Canada—would be a valuable new strategic objective. A Terms of Reference has been drafted for a new CHMS Analytical Working Group, which will be chaired by Statistics Canada and composed of representatives from PHAC and Health Canada.

Recommendation 1
Timeline Deliverables Responsible party
January 1, 2014 Eight-year content plan Director, Health Statistics Division (HSD)
April 1, 2013 CHMS Analytical Working Group with PHAC and Health Canada Director, HSD

Recommendation 2

Performance:
It is recommended that the management of CHMS increase awareness of the data and improve their accessibility by promoting CHMS to a wider audience of potential clients and users, and by providing support on how to use the data.

Statement of agreement or disagreement

Management agrees with this recommendation

Management response

Increasing the use and awareness of CHMS is an important objective that has been clearly identified within the evaluation. Management agrees that more work can be done in this area and has developed three clear initiatives to better promote and support the use of CHMS. First, we have begun training additional staff with HSD's Client Services area to respond to external client service requests for information on CHMS and for custom tabulations using CHMS data. Second, management is developing a communications plan to increase awareness of both the CHMS data within the RDCs and the availability of the Biobank samples for researchers. Third, management is considering holding more workshops across the country to increase users' capacity to analyze CHMS data. This last strategy, however, requires resources to pay for travel and additional costs to conduct such workshops. Achieving this objective will depend upon our ability to identify sufficient resources to increase the number of workshops held over the next two years.

Recommendation 2
Timeline Deliverables Responsible party
April 1, 2013 Two additional trained staff Director, HSD
April 30, 2013 Communications plan Director, HSD
Ongoing Workshops Director, HSD

Recommendation 3

Performance measurement
It is recommended that the management of CHMS improve the performance measurement system, as a tool, to systematically collect performance data and monitor the progress toward achieving its outcomes.

3a) Improving data
To accurately assess the achievement of the long-term outcome and the impact of CHMS, it is recommended that management of the CHMS ensure that a formal tracking, such as the Client Relationship Management System (CRMS), is in place for publications and studies based on CHMS data, to demonstrate their use. The information on publications using CHMS data must be shared on a recurrent basis with researchers and stakeholders to enhance further external coordination.

3b) Financial information
It is recommended that accurate financial information is made available by the Finance Branch to support CHMS performance data collection to demonstrate the level of efficiency over time for decision making and accountability reporting.

Statement of agreement or disagreement

3a) Management agrees with this recommendation.
3b) Management agrees with this recommendation

Management response

3a) Management already has an adequate tracking system to monitor publications and studies based on CHMS data; however, it has not been systematically shared with external researchers. As such, management is proposing to post this list on our external CHMS website and update it annually.

3b) Management is undertaking an operational review of the CHMS Operations Section to ensure that processes and practices are optimal. This review will be completed over the next six months and will lead to implementing performance indicators to measure the efficiency of the program.

Financial Branch Action Plan:
The Finance Branch will develop standard reports and financial indicators in support of all programs.  The organization has detailed financial information already available that can be easily formatted to better address evaluation needs of programs and be made available on a timely basis.  The determination of the format and elements to be measured will be developed in collaboration, and validated with program managers and the departmental evaluation group.  The final recommendation will be presented and approved by the Administrative Practices Committee.

Recommendation 3
Timeline Deliverables Responsible party
December 2013 Tracking list on website Director, HSD
September 2013 Operational review report Director, HSD
January 2014 Efficiency indicators Director, HSD
Financial Branch Action Plan
Timeline Deliverables Responsible party
March  2014 Key standard program financial indicators approved DG, Finance
As per Schedule to develop Program PM Strategy (see RBAEP) Program financial coding structure reviewed and adjusted to support Program Performance Measurement Framework. DG, Finance and Director, HSD
Communications Branch Action Plan
Timeline Deliverables Responsible party
January 2014 Development of a formal tracking system DG, Communications

Figures

Figure 1: Canadian Health Measures Survey Logic Model

Model of Canadian Health Measures Survey Logic
Description: Figure 1 Canadian Health Measures Survey Logic Model

Appendices

  • Appendix 1: Canadian Health Measures Survey process map
  • Appendix 2: A composite logic model for all activities under the Action Plan
  • Appendix 3: Canadian Health Measures Survey (CHMS) evaluation matrix
  • Appendix 4: A list of publications reviewed as part of the literature review

All appendices are available upon request. Please contact AEB-Professional-Practices@statcan.gc.ca

Notes:

Footnote 1

For more information on the joint REB for HC and PHAC, see www.hc-sc.gc.ca/sr-sr/advice-avis/reb-cer/index-eng.php.

Return to footnote 1 referrer

Footnote 2

REB website, www.hc-sc.gc.ca/sr-sr/advice-avis/reb-cer/index-eng.php.

Return to footnote 2 referrer

Footnote 3

CHMS core funding includes only the operating budget and excludes the employee benefit plan and PWGSC accommodation costs.

Return to footnote 3 referrer

Footnote 4

The 2010 Annual Report on the Health of the Evaluation Function prepared by TBS outlines the importance of calibrating evaluations, and in this context, to 'calibrate' is to determine the appropriate balance between the level of evaluation effort and the evaluation context (e.g., program's materiality, associated risks, etc.) (p. 36).

Return to footnote 4 referrer

Footnote 5

The program area was able to supply the names of their partners and cost-recovery clients, the names of the researchers who had access to the shared files at HC and PHAC, and the names of those who had requested access to RDCs to use CHMS data files. It is most likely that the names received are not an exhaustive representation of everyone that has been exposed to the data.

Return to footnote 5 referrer

Footnote 6

Tremblay, MS. The need for directly measured health data in Canada. Canadian Journal of Public Health 2004; 95: 165-8.

Return to footnote 6 referrer

Footnote 7

Connor Gorber S, Tremblay MS, Moher D, et al. A comparison of direct versus self-report measures for assessing height, weight and body mass index: a systematic review. Obesity Reviews 2007; 8: 307-26.

Return to footnote 7 referrer

Footnote 8

Statistics Canada. Canadian Health Measures Survey: Rational, background and overview, Health Reports, Background Papers on the Canadian Health Measures Survey, Special Issue, supplement to Volume 18, Dec 2007.

Return to footnote 8 referrer

Footnote 9

www.cdpac.ca

Return to footnote 9 referrer

Footnote 10

www.hcsc.gc.ca/english/care/romanow/hcc0086.html

Return to footnote 10 referrer

Footnote 11

Statistics Canada. "Canadian Health Measures Survey: Rational, background and overview" Health Reports: Background Papers on the Canadian Health Measures Survey, Special Issue, supplement to Volume 18, Dec 2007. Statistics Canada Catalogue no. 82-003-XIE2007101

Return to footnote 11 referrer

Footnote 12

Centers for Disease Control, National Center for Health Statistics. National Health and Nutrition Examination Survey. Available at: www.cdc.gov/nchs/nhanes.htm. Accessed January 1, 2007.

Return to footnote 12 referrer

Footnote 13

The NHANES program began in the early 1960s and has been conducted as a series of surveys focusing on different population groups or health topics. In 1999, the survey became a continuous program that has a changing focus on a variety of health and nutrition measurements to meet emerging needs. The survey examines a nationally representative sample of about 5,000 people each year. (Source: www.cdc.gov/nchs/nhanes.)

Return to footnote 13 referrer

Footnote 14

Kuczmarski RJ, Ogden CL, Guo SS, et al. 2000 CDC growth charts for the United States: methods and development. Vital Health Statistics 2002; 11(246): 1-190.

Return to footnote 14 referrer

Footnote 15

National Center for Health Statistics. National Health and Nutrition Examination Survey Data Accomplishments. Available at www.cdc.gov/nchs/about/major/nhanes/DataAccomp.htm Accessed June 1, 2007.

Return to footnote 15 referrer

Footnote 16

Dunstan DW, Zimmet PZ, Welborn TA, et al. The rising prevalence of diabetes and impaired glucose tolerance. Diabetes Care 2002; 25: 829-34.

Return to footnote 16 referrer

Footnote 17

Aromaa A, Koskinen S, Huttunen J. Health in Finland. KTL– National Public Health Institute. Ministry of Social Affairs and Health. Helsinki, Finland: Edita Ldt., 1999.

Return to footnote 17 referrer

Footnote 18

Aromaa A, Koskinen S. (eds.). Health and Functional Capacity in Finland. Baseline Results of the Health 2000 Health Examination Survey. Helsinki, Finland: KTL – National Public Health Institute, 2004.

Return to footnote 18 referrer

Footnote 19

Statistics Canada. Canadian Health Measures Survey: Rational, background and overview, Health Report, Background Papers on the Canadian Health Measures Survey, Special Issue, supplement to Volume 18, Dec 2007.

Return to footnote 19 referrer

Footnote 20

These requirements were compiled based on interviewees' preferences and suggestions. However, it is known that some of them are already in place (e.g., the capacity of record linkages), while others are out of scope for a national representative survey such as CHMS.

Return to footnote 20 referrer

Footnote 21

Medical doctors and health professionals use various guidelines for interpreting conditions observed in patients and for determining appropriate treatments. CHMS data are used as Canada-representative reference data for creating and updating some of these guidelines (e.g., the blood lead interpretation guide and the guidelines related to folic acid intake in women in child-bearing age). In the absence of Canadian, nationally representative health reference data, such guidelines have relied on data or practices from other countries, such as the United States.

Return to footnote 21 referrer

Footnote 22

When studying a potential health condition or issue, researchers must compare their research interest with the 'normal' state (for example, to confirm the significance of an issue or to assess the effect of an intervention). According to researchers, CHMS data provide a quintessential, Canada-representative control group for such studies, thus, providing "an incredible service to researchers in so many areas."

Return to footnote 22 referrer

Footnote 23

As part of CHMS data collection, samples of blood and urine are collected from respondents and stored in a Biobank. The Biobank is hosted at the National Microbiology Laboratory in Winnipeg. There are strict guidelines and protocols in place, compliant with the Statistics Act and the regulations of the Research Ethics Board of HC to ensure confidentiality, security and appropriate access to the samples. Access to the Biobank became available in June 2012 and is regulated by the Biobank Committee.

Return to footnote 23 referrer

Footnote 24

Biomonitoring measures include environmental exposure measures, but also biomarkers from seven other themes related to health.

Return to footnote 24 referrer

Footnote 25

Details already presented in Table 5.

Return to footnote 25 referrer

Footnote 26

These concerns were not limited to CHMS, but apply to all surveys requiring the use of an RDC.

Return to footnote 26 referrer

Footnote 27

Some challenges were faced when assessing the efficiency and economy of the CHMS program. The practice of compiling financial data by fiscal year rather than by cycle, the lack of baseline data on cost and resources used (as a result of the novelty and uniqueness of CHMS in Canada) and the lack of time and evaluation resources to perform any international comparisons with similar surveys were some of the factors that impacted the completeness and the conclusiveness of the evaluation evidence.

Return to footnote 27 referrer

Date modified: