Evaluation of the Demography, Aboriginal and Other Social Statistics Program
2010-2011 to 2012-2013

Executive Summary and Management Response and Action plan

November 2014

Executive Summary

This report presents the results of the evaluation of the Demography, Aboriginal and Social Statistics (DAOSS) Program. It covers three fiscal years, from 2010/2011 to 2012/2013. The evaluation was undertaken by the Evaluation and Performance Measurement Division.

This report was approved by the Departmental Evaluation Committee and the Chief Statistician on November 26, 2014.

In accordance with the accountability requirements in the Treasury Board 2009 Policy on Evaluation and its Directive, this report is available to the public with the Executive Summary and Management Response and Action Plan being posted on the departmental website in both official languages.

Statistics Canada also shared this report with its program delivery partners and key stakeholders, including the National Statistics Council.

Evaluation scope, purpose and methodology

The evaluation covers the DAOSS Program, with a particular focus on the Population Estimates and Projections (PE&P), the General Social Survey (GSS) and the Aboriginal Statistics components. It addresses a number of questions related to the continued need for the program, its alignment with government priorities, its consistency with federal roles and responsibilities, the achievement of its expected outcomes, and the extent to which it demonstrates efficiency and economy. It was included in the Departmental Risk-based Audit and Evaluation Plan for 2012/2013 to 2016/2017, which was approved by the Departmental Evaluation Committee in March 2013.

In accordance with the Government of Canada's Policy on Evaluation, the purpose of the DAOSS evaluation is to provide an evidence-based, neutral assessment of its value for money, and more specifically, its relevance and performance. In addition, the evaluation design has incorporated all components of the Standard on Evaluation for the Government of Canada to ensure quality, neutrality and utility. The list of questions addressed in this evaluation incorporates all core issues identified in the Government of Canada's Directive on the Evaluation Function.

Information from multiple sources was used to address the evaluation questions, including document and literature reviews; a review of financial and administrative data; a series of key informant interviews with program representatives, internal and external stakeholders; a survey of data users; and a bibliometric and webometric analysis. The study was calibrated to capture the horizontal aspects of the relevance and the efficiency/economy at the DAOSS Program level, and to explore in depth the aspects of performance that are specific to each of the three focus components, namely, the PE&P, the GSS and the Aboriginal Statistics components. The presentation structure of the findings and conclusions is aligned with the calibration approach.

Demography, Aboriginal and Other Social Statistics (DAOSS) Program

Relevance

The DAOSS Program contributes significantly to multiple areas of public interest and society's well-being and plays a strategic role in informing public debate on socioeconomic issues, in supporting social policy development and in guiding public and private decision making. The three focus components respond to needs that are specific to their subject-matter orientation and are relevant to the work of all levels of government and of many national and international groups and initiatives.

Overall, the DAOSS Program is in a good position to maintain its relevance; however, some areas of particular risks, challenges or gaps exist and need to be addressed. The need for consistent data disaggregated at a smaller geographical level (i.e. municipalities, on-reserve communities) or sub-groups level (i.e. Aboriginal groups, visible minorities) appears to be prominent. The improvement of the measures of interprovincial-territorial migration are linked to a growing area of interest of significant impact, hence, the Demography Division's efforts in this direction need to continue. In the scope of the GSS, the need for more analytical products related to historical series, to social trends occurring over time or to emerging social issues has been identified. In the field of statistics on Aboriginal people, the limited information for the on-reserve population is a gap that requires attention. Furthermore, there are concerns regarding the coherence, comparability and representativeness of the available data, which reduce the analytical power of the existing information on Aboriginal people.

The evaluation findings confirmed that the DAOSS Program is in line with the current federal government priorities and commitments as it supports multiple federal initiatives including legal and contractual obligations. It is also in line with the priorities and the role of Statistics Canada—not only because of its clear link to the agency's strategic outcomes, but also because each of the focus components is interrelated with other statistical programs and is a part of the entire national statistical structure. Further, the federal government appears to be in the best position to deliver the program as there is no other organization that has the capacity, the reputation, and the public trust to collect and process statistical information with comparable scope and quality. Other similar activities conducted by parties external to Statistics Canada are considered to be complementary rather than viewed as an alternative to the DAOSS Program.

Performance (efficiency and economy)

The evaluation findings confirm that the DAOSS Program has made significant investments (effort and reorganization) in efficiency-generating projects oriented toward: (1) reducing response burden by increasing the use of alternative data sources; (2) streamlining and consolidating the operations by using common toolsFootnote 1, and (3) incorporating new technologies in operations (e.g., e-questionnaire, interactive website, social media). The program is making progress in adopting a results-based culture; however, it is still too early to assess its implementation. Given the limitations of the financial system, it was not possible in this context to assess the cost of the inputs and whether optimization was reached. However, financial figures for the DAOSS Program as a whole provided some evidence that all variances between the planned budget and actual expenditures have not exceeded 4%, which indicates a good management of resources in general.

Population Estimates and Projections

Effectiveness

Overall, statistical products on population estimates and projections are accessible, although some concerns were expressed with the transition to the new format of CANSIM tables that resulted in inconsistencies and fewer content details in some sets, and that the search engine is not efficient enough to locate some specific information. The evaluation found that the population estimates and projections were released in a timely manner. The program has met the delivery dates and standards, and has improved its responsiveness to the increased demand of more frequent population projections by enhancing its microsimulation activities.

Evaluation findings suggest that the demographic statistical information empowers users by increasing their demography-related knowledge and capacity. A closer look at the interpretability, coherency and accuracy reveals that overall the population estimates and projections are coherent and accurate. The Demography Division has undertaken many initiatives to improve the coherency and accuracy of its data, including the harmonization of the administrative data and statistical products with other Statistics Canada divisions. Currently, the PE&P is the single internal provider of most demographic estimates. The division is also working on initiatives to address challenges around the coherency and accuracy of indicators for internal migration. This is a well-known issue in the domain of demography and is not unique to Canada. In terms of interpretability, findings suggest that there is some room for improvement. Some users are dissatisfied with the public availability and comprehension of up-to-date technical papers, metadata analysis and methodology papers, as well as accessibility through the website interface and availability of written explanations of the concepts in a format and language that a non-statistician could understand.

All lines of evidence confirm that the statistical information on population estimates and projections is widely used by governments, public institutions, academic researchers, private sector, and Canadians. Multiple examples of use were found in documents, and were provided by key informants and survey respondents. The bibliometric and webometric analysis confirmed that the information has been used in the domain of social sciences and humanities, and the natural sciences and engineering for academic research.

Efficiency

The PE&P is currently working on initiatives intended to improve its efficiency, such as the harmonization of the demographic content, improved documentation practices, the incorporation of new technology for dissemination and microsimulation, and the identification of new sources of administrative data. Efforts to increase the use of administrative data must continue. It is worth mentioning that population projections' projects contribute significantly to the revenue of the component (almost 50%), and the Demosim microsimulation activities alone generated 24% of the annual revenue in 2013/2014.

General Social Survey (GSS)

Effectiveness

Overall, the review found that the GSS statistical information is accessible. A wide range of GSS products are available to key stakeholders, the general public and researchers, through a variety of channels (e.g., the web, RDCs or customized services). The major GSS outputs are released in a timely manner, according to the planned schedules and deadlines. Areas of improvement identified are linked to the need to make the website more user-friendly to better acquire the information, and reviewing the frequency of cycles for some themes so that appropriate adjustments could be made in the future with the new GSS redesign.

By providing accurate, interpretable and coherent statistical information, the GSS is seen as a trusted source of social data and products which contribute to the enhancement of the knowledge in the social domain. While the efforts of the program toward achieving better results were acknowledged, there is still room for improvement in terms of interpretability (e.g., providing more methodological papers and explanations around the GSS changes) and coherency (e.g., harmonizing the content through cycles over time to ensure comparability and to allow the establishment of trends over time for some specific variables). Concerns were also expressed with regard to the challenges that the GSS is facing with declining response rates that may have quality implications.

Evaluation evidence provides strong confirmation of the fact that GSS statistical information is widely used, to inform a multitude of diverse activities carried out by clients such as policy-makers, governments, private and public institutions, academic institutions and social researchers, not-for-profit organizations, etc. The GSS information is also used by other statistical surveys, for comparability and validation, and to supplement the information.

Efficiency

Many initiatives were put in place to generate efficiencies and cost savings. For example, the implementation of e-questionnaires for data collection is expected to stabilize or increase response rates and improve the quality of data. However, it is still too early to analyze the real effect of this change on efficiency. The exploration of opportunities for the use of more administrative data sources in lieu of primary data is seen as a potential option for cost saving as well as for improving the quality and coherence of the data. It is also believed that an interdepartmental forum, which oversees horizontal social areas of statistical interest at the strategic level, is a much needed component of the governance structure of social statistics, in order to better identify priorities, avoid duplication, fill information gaps, and create synergies across policy departments.

Aboriginal Statistics component

Effectiveness

Overall, the statistical information on Aboriginal people is accessible. There are some concerns and uncertainties about the accessibility of on-reserve data collected by the First Nations Information Governance Centre. Statistical information on Aboriginal people is being released in a timely manner. Some preferences for more frequent collection and some issues with data being outdated have been expressed.

While the statistical information on Aboriginal people from the Aboriginal Peoples Survey (APS) is of good quality, a number of challenges and potential risks to maintaining the accuracy were identified. To mitigate the situation, the program is working towards increasing the use of administrative data. There are still concerns about the comparability and coherence of off-reserve/on-reserve data, data over time, and the comparability of statistical information on Aboriginal people with general population information. The interpretability of the statistics on Aboriginal people is good because metadata is available and Statistics Canada provides good support and response to custom tabulations and special analyses.

In the public sector, statistics on Aboriginal people are used for planning, policy development, decision making, reporting and funding allocation. The main key users are federal partners. The data also support the activities of National Aboriginal organizations and Aboriginal governments and communities. Some limitations to data use are a result of aggregating the data at a national level, which results in blending the specifics of the different Aboriginal groups, and restricts the analysis of smaller subgroups of the Aboriginal population. Despite some issues linked to the insufficient liaison capacity and the scope of training activities, evidence suggests that these program components are very much needed and useful, and they could create leverage for both Statistics Canada and the Aboriginal communities if their full potential is explored.

Efficiency

Between 61% and 80% of the activities of the Aboriginal Statistics component are cost-recovery funded. Consequently, the efficiency of statistics on Aboriginal people is highly influenced by factors that are beyond its direct control. The elimination of the training program in 2012 limited the opportunities for Statistics Canada to communicate with the Aboriginal communities. This might have an impact (yet unknown) on the quality of the data in the future, and in turn, on the effectiveness of the program. The internal governance mechanisms are solid and are working well; however, potential for improvement regarding coordination with the National Aboriginal Organizations and a need for a better definition of the roles and responsibilities with federal partners have been identified.

Recommendations

Recommendation 1 (DAOSS Program):

To maintain the continued relevance of data products to the needs of its clients, the DAOSS Program should establish a structured process for documenting, analyzing and responding to constructive feedback from the full range of its users. The process should include a systematic tracking and documentation of available information on these users.

Recommendation 2 (DAOSS Program):

The DAOSS Program in collaboration with Dissemination Division should consider actions to strengthen the analytical capacity of its data and products, and to increase their value to the users. To the extent possible:

  1. the Social and Aboriginal Statistics Division should look at appropriate alternatives, to increase the availability of disaggregated data for the General Social Survey and the Aboriginal Statistics components;
  2. the Social and Aboriginal Statistics Division and the Demography Division should ensure that metadata are comprehensive and interpretable by users with varied statistical proficiency, for the General Social Survey, the Aboriginal Statistics and the Population Estimates and Projections components; and
  3. the Dissemination Division should ensure that the implementation of Statistics Canada's New Dissemination Model will provide users with varied statistical proficiency effective search tools and streamlined access to products.

Recommendation 3 (General Social Survey):

The General Social Survey component should consider actions to strengthen its data and products and increase the value of the survey to users. To the extent possible, efforts should be focused on:

  1. expansion of the scope and the variety of analytical products, including both products that involve re-packaging and utilization of existing data, and products that address themes of emerging importance and interest to key users;
  2. revision of the frequency of the survey themes; and
  3. revision of existing fora and interdepartmental consultation mechanisms to identify potential horizontal social statistical interest.

Recommendation 4 (Population Estimates and Projections):

The Demography Division should continue its efforts in providing relevant demographic data particularly by focusing its developmental and research work on the increased use of administrative data to improve the estimations of different components of population change (e.g. interprovincial migration), and by exploring ways to effectively address users needs for population projections.

Management response and action plan

Recommendation 1

Focus: DAOSS Program

To maintain the continued relevance of data products to the needs of its clients, the DAOSS Program should establish a structured process for documenting, analyzing, and responding to constructive feedback from the full range of its users. The process should include a systematic tracking and documentation of available information on these users.

Statement of Agreement /Disagreement

Agree

Management Response

There are several processes by which the DAOSS Program systematically collects feedback from its various and varied users such as the Federal Provincial and Territorial statistical focal points, interdepartmental federal committees, bilateral and multilateral user consultations, cost-recovery clients, and advisory committees. The PE&P Program has also recently established a data user group composed of representatives of internal users at Statistics Canada including representatives from consultative services of Regional Offices who are at the forefront responding to data requests and users.

To that effect, the DAOSS Program intends to pursue its current efforts to maintain the relevance of its program and to collect information to identify gaps and emerging needs. The DAOSS Program will establish a central depository for the Branch in which all documentation and feedback received from key partners and data users will be stored and analyzed in terms of the ongoing relevance of the statistical programs and products. This information will serve the various statistical programs in the Branch, and be evaluated by the management team via an annual review as well as feed into field strategic long-term planning.

Table 1 Recommendation 1
Timeline Deliverable(s) Responsible Party
Establish the Branch Depository - December 2015

Review by Branch management – March 2016 and ongoing
Establish a Branch Depository of documentation and feedback received from data users; with annual or biannual review by Branch management team. Directors of Demography Division and Social and Aboriginal Statistics Division (SASD), and DG of Census Subject Matter, Social and Demographic Statistics Branch (CSMSDSB)

Recommendation 2

Focus: DAOSS Program

The DAOSS Program in collaboration with Dissemination Division should consider actions to strengthen its data and products, and to increase their value to the users. To the extent possible:

  1. the Social and Aboriginal Statistics Division should look at appropriate alternatives, to increase the availability of disaggregated data for the General Social Survey and the Aboriginal Statistics components;
  2. the Social and Aboriginal Statistics Division and the Demography Division should ensure that metadata are comprehensive and interpretable by users with varied statistical proficiency, for the General Social Survey, the Aboriginal Statistics and the Population Estimates and Projections components; and
  3. the Dissemination Division should ensure that the implementation of Statistics Canada's New Dissemination Model will provide users with varied statistical proficiency effective search tools and streamlined access to products.

Statement of Agreement /Disagreement

Agree

Management Response

Recommendation 2 (a): Social and Aboriginal Statistics Division (SASD) has begun to release more tables through CANSIM from the GSS Program, and will continue to release more products as the common tools such as GTAB and GEXPORT are implemented and adopted.

Recommendation 2 (b): SASD and the Demography Division provide a large amount of metadata with its released products. SASD will review the metadata currently provided in order to identify any inaccuracies and will make the appropriate updates. SASD will also review the user comments to identify the specific metadata needs, and assess the feasibility of expanding the metadata for its programs.

Demography Division recently completed the implementation of a set of recommendations made following an internal Audit on the Population Estimation Program which included the production of documentation on the systems and outputs. Demography Division will undertake the review of comments made on the state of metadata for its program and assess the feasibility and cost of expanding their metadata products if need be.

Recommendation 2 (c):

Statistics Canada is currently actively working to improve its dissemination strategy. It is seeking to modernize the way in which it organizes and publishes its statistical output. The objective of this "New Dissemination Model" project is to ensure relevant, user-friendly output which will meet the needs of a broad range of data consumers. It will feature a single output data repository to drive dynamically generated data tables, a simplified product line to ensure consistency in product availability, and a revised navigation strategy to make sure that Statistics Canada's information is easy to find. The model will also include an automated feed of official statistics to the Federal Government Open Data environment and Web Data Services will be introduced to allow for the retrieval of data directly from Statistics Canada output database. Dissemination Division and SASD will work together to integrate data and products into the New Dissemination Model.

Table 2 Recommendation 2
Timeline Deliverable(s) Responsible Party
2a) 2013 and continued with each data releases

Review by Branch management – March 2016
CANSIM tables for APS and GSS Director, SASD
2b) March 31, 2016 Feasibility report on expanded metadata; update of existing metadata Director, SASD ; Director of Demography Division
2c) March 31, 2015 New Dissemination Model Dissemination Division

Recommendation 3

Focus: General Social Survey

The General Social Survey component should consider actions to strengthen its data and products and increase the value of the survey to the users. To the extent possible, efforts should be focused on:

  1. Expansion of the scope and the variety of analytical products, including both products that involve re-packaging and utilization of existing data, and products that address themes of emerging importance and interest to key users;
  2. Revision of the frequency of the survey themes; and
  3. Review existing fora and interdepartmental consultation mechanisms to identify potential horizontal social statistical interest.

Statement of Agreement / Disagreement

Agree

Management Response

  1. GSS has begun the process of expanding analytical products over the past year, and brought in a full-time analyst to accomplish this. Analytical reports have been released in "Insights on Canadian Society" and for the new "Spotlight on Canadians: Results from the GSS" series which is specifically for GSS, and more are planned over the next year. In addition, GSS is producing Fact sheets and CANSIM tables, and plans to add infographics. In collaboration with key stakeholders, SASD will continue to work on developing an analytical plan for each GSS cycle to define the depth and breadth of analytical products and ensure their relevance.
  2. The survey themes were reviewed over the last year; this review was triggered because of two themes (Giving, volunteering and participating (GVP) and social identity (SI)) taking place in the same year (2013), and the recognition that there was an opportunity to introduce a new theme in 2016. The review of themes is intended to be a regular activity, in order to schedule the themes appropriately (since it is no longer possible to have all themes on a 5-year cycle) and to ensure that the content is relevant and gaps are being addressed. This review will take place with appropriate external consultation, considering stakeholder needs, operational capacity, and ability to monitor key social trends, level of effort to implement the next iteration of a theme, and implementation status for multimode. As a result of this review, a cycle topic plan until 2022 will be presented to the key stakeholders.
  3. The GSS Program has an extensive consultation process (bilateral and multilateral) with key stakeholders and user groups for each survey cycle. In addition, there is a GSS Steering Committee (cross-section of key departments), the Advisory Committee on Social Conditions, and the Federal-Provincial-Territorial Committee on Social Statistics, all of which provide input to the GSS Program. SASD will review and assess this governance structure, with the objective to define the mechanisms that should be put in place in order to advise the GSS program on horizontal and strategic issues. Possible elements of the plan include the establishment of objectives, the terms of reference and the best mechanisms to consult (for example, better use of existing forums or the creation of a new group).
Table 3 Recommendation 3
Timeline Deliverable(s) Responsible Party
December 31, 2015 Feasibility  plan to increase scope and variety of analytical products; Director, SASD
June 30, 2015 Development of  the Plan for GSS theme or topic until 2022; Director, SASD
March 31, 2016 Governance forum plan Director, SASD

Recommendation 4

Focus: Population Estimates and Projections

The Demography Division should continue its efforts in providing relevant demographic data particularly by focusing its developmental and research work on the increased use of administrative data to improve the estimations of different components of population change (e.g. interprovincial migration), and by exploring ways to effectively address users needs for population projections.

Statement of Agreement / Disagreement

Agree

Management Response

Demography Division in 2015-16 will work on a research and developmental plan to find ways to improve the components of population change, in particular focusing on ways to improve the estimates of inter provincial migration and emigration. The PE&P submitted a 10 year plan (Continuity and Quality Maintenance (CQM)) to fund the acceleration of the work pertaining to the increased use of administrative data and the use of the Canadian Statistical Demographic Database (CSDD) research results in the PE&P development work. The plan would consist of conducting research on the use of current and new administrative data to improve the quality of the estimation of the demographic component and therefore reduce the error of closure. This work would also build on the efforts of other partner divisions, namely the Administrative Data Division which conducts the CSDD and that of the Tourism and the Centre for Education Statistics Division (TCESD) which is currently looking at ways of accessing Canada Border Services Agency (CBSA) exit and entry data. The PE&P will also be launching a series of workshops with the provinces and territories in 2015-2016 to explore how the PE&P and the provinces and the territories can work together to improve the estimation of intra-provincial data and interprovincial migration. This latter set of activities would launch the second phase of the CQM plan to improve the quality of population estimates.

The proposed plan will consist of multiple activities occurring over the next 10 years. Activities listed below are the ones taking place in the short-term.  Pending the outcomes of these activities, other specific and relevant activities will be proposed.

Table 4 Recommendation 4
Timeline Deliverable(s) Responsible Party
January - December 2015 Develop a long-term strategy for developmental and research work for Demosim and PE&P Programs Director Demography
Spring/Fall 2015 Initiation of workshops with provinces and territories, to seek input regarding future research plans Director Demography
March 2016 Propose implementation alternatives for the PE&P Program according to the level of funding in place. Director Demography
December 2015 Assist TCESD and ADD in negotiating access to CBSA data (completion by December 2015) Directors of TCESD, ADD and Demography Division

The full report is available upon request. Please contact: AEB-Professional-Practices@statcan.gc.ca

Notes:

Footnote 1

The program has made the transition to common tools for collection and dissemination, harmonizing content and eliminating duplication, or enhancing quality assurance.

Return to footnote 1 referrer

Date modified: