Evaluation of the Labour, Education, Income and Tourism Programs
2010-2011 to 2012-2013

Executive Summary and Management Response and Action plan

December 2014

Executive Summary

This report presents the results of the evaluation of Statistics Canada's Labour, Education, Income and Tourism Programs (LEIT). It covers three fiscal years, from 2010-2011 to 2012-2013. The evaluation was undertaken by the Evaluation and Performance Measurement Division. The evaluation study was conducted over the period June 2013 to August 2014.

This report was approved by the Departmental Evaluation Committee and the Chief Statistician on December 17, 2014.

In accordance with the accountability requirements in the Treasury Board 2009 Policy on Evaluation and its Directive, this report is available to the public with the Executive Summary and Management Response and Action Plan being posted on the departmental website in both official languages. Statistics Canada also shared this report with its program delivery partners and key stakeholders, including the National Statistical Council.

Evaluation scope, purpose and methodology

This evaluation covers all four LEIT Programs with a particular focus on two programs: income statistics and tourism statistics. The programs were identified based on consultation with LEIT representatives during the planning phase, as well as additional planning criteria, such as the different constraints that the evaluation faced, and the complexity and the structure of LEIT.

In accordance with the Government of Canada's Policy on Evaluation, the purpose of the evaluation is to provide an evidence-based, neutral assessment of its value for money, and more specifically its relevance and performance. It addresses a number of questions related to the continued need for the programs, their alignment with government priorities, their consistency with federal roles and responsibilities, the achievement of their expected outcomes, and the extent to which they demonstrates efficiency and economy. It was included in the Departmental Risk-based Audit and Evaluation Plan for 2012/2013 to 2016/2017, which was approved by the Departmental Evaluation Committee in March, 2013.

Information from multiple sources was used to address the evaluation questions, including document and literature reviews; a review of financial and administrative data; a series of key informant interviews with program representatives, and internal and external stakeholders; a survey of data users; and a bibliometric and webometric analysis.

Overall Conclusions about LEIT

The evaluation found, overall, that all four LEIT Programs are relevant. Each one responds to specific legislative obligations and/or international commitments, they provide important statistics about Canada on a national level, and their activities are consistent with the federal government's roles and responsibilities. Each of the four LEIT programs is important to Canada for different reasons, and there will continue to be a demand for national statistics in all four areas.

Even though some opportunities for improving performance were identified, LEIT's income, labour and education programs are achieving their intended outcomes in terms of accessibility, timeliness, accuracy, interpretability, and coherence of data. The quality of international tourism data is an issue due to low return rates to surveys, and domestic tourism statistics are at risk of losing cost-recovery partners. As a result, the sustainability of the Tourism Statistics Program is at risk.

The evaluation examined the efficiency and economy of the income and tourism components and found that steps have been taken to reduce costs and increase efficiency.

Income Statistics Program (ISP)

Relevance

Statistical information produced by ISP responds to legislative and policy requirements for information to support, among others, the national macroeconomic accounts, the Consumer Price Index calculations, and calculations of provincial transfers. These are used by federal government and external stakeholders as a basis for research; policy development; design and assessment of programs; and to fulfill national and international commitments. ISP responds to the Government of Canada's needs by providing information on income, pensions, and wealth. Information on what Canadians earn, spend and save is used to inform policy development, and to conduct research on the social and economic conditions of families and individuals. No other organization provides income statistics of the same scope, level of detail, and quality as does Statistics Canada.

Given the importance of income statistics in social and economic policy-making and business decision-making, there will continue to be an interest in conducting research that uses income statistics. Demographic changes in the Canadian population, specifically the aging population will drive interest in certain types of data and analyses, for example, pensions. Economic cycles also create interest in certain types of data, for example, income distribution during recession and post-recession periods. In addition to those topics, there are several areas of future research interest, such as: growing income disparities, debt, and the economic performance of Canadian companies in global markets. All these factors support the conclusion that there is a strong continued need for ISP.

Performance (Achievement of Outcomes)

Overall, accessibility and timeliness are satisfactory among users. There may be opportunities to improve the timeliness of periodic/one-time surveys and accessibility issues related to Personal Income Tax data were noted. It should be noted that in 2013-14, the Longitudinal Administrative Database (LAD) was placed in the Federal Research Data Centre as a pilot project to assess the feasibility of allowing access to RDC researchers. The LAD consists of a 20% longitudinal sample of Canadian tax filers and provides researchers and analysts with a tool for studying the changes in income experienced by individuals and their families. The pilot project is currently being evaluated and the next phase will be to extend the pilot to another RDC.

A strong majority of users are satisfied with the accuracy, coherence and interpretability of the program's statistical information. There are concerns raised by users. These relate to information on changes to methodologies and the loss of longitudinal surveys. These concerns could be addressed through more communication between ISP and certain of its product users.

Income statistics have been used extensively to inform debate, research and decision-making. Users, which include provinces and territories, non-government organizations, policy think tanks, financial institutions, researchers, and academics, use ISP information for research purposes. Examples of research include: income disparities (including gender disparities), poverty analyses, and determinants of personal income.

Performance (Efficiency and Economy)

ISP has increased the economy and efficiency of its operations. A number of initiatives have been undertaken to reduce costs and increase efficiency.

ISP continually investigates new data sources to support the program. Users indicated that these new directions should continue to be examined.

Tourism Statistics Program (TSP)

Relevance

Tourism is an important sector in supporting the Canadian economy, and tourism statistics contribute to several important travel market analyses and provide key information on this industry to the National Accounts. In 2010, tourism was responsible for $73.4 billion in revenues and represented approximately 2 percent of Canada's overall gross domestic product. It is a dynamic and complex sector, characterized by multiple modes of transportation, multiple points of entry, many geographical areas covered, many activities, and many users. Tourism data are important to users such as academics and businesses to conduct comparative analyses across Canadian jurisdictions and other travel destinations markets. The relevance of tourism statistics is reinforced by the fact that the TSP receives additional requests, on a cost-recovery basis.

While there are other sources of tourism statistics, their products would not meet all the needs of TSP users and, in particular, TSP's internal Statistics Canada users.

The complexity of tourism and the considerable economic impact of the tourism industry suggest that there will be a continued need for these statistics by a wide range of users, including the Government of Canada.

Performance (Achievement of Outcomes)

Accessibility of tourism data is satisfactory. Timeliness has been an issue, although improvements have been noted in this area. TSP's products are indeed contributing some level of knowledge.

Evaluation findings indicate that TPS has been challenged with data quality issues, primarily due to low return rates to its international travel survey. Funding reductions at one of TSP's primary clients in 2012 led to a significant reduction in the funding it provides to TSP. This revenue reduction led to a 50% reduction in sampling for the domestic tourism survey by TSP. As a consequence, the remaining funding partners for the domestic survey are now dissatisfied, which could lead to further funding withdrawals and further sampling reduction. Under the current resourcing model, if partners withdraw their funding, the TSP will no longer have the resources necessary to collect the national tourism data that it needs for the federal government's and Statistics Canada's purposes.

Performance (Efficiency and Economy)

There is evidence of efficient and economical elements within the program. The program has strived to increase economy by relying on other organizations to collect data. Funding received from cost-recovered activities also helps meet the needs of its internal users by funding the provision of data to internal clients. This approach demonstrates economy. However, it increases the risk to Statistics Canada, since TSP's ability to perform core activities depends on its success at finding external funding sources and meeting the needs of cost-recovery clients.

Labour Statistics Program (LSP)

Relevance

Data from the Labour Statistics Program are important sources of information for assessing the economic health of the nation. LSP data are important for organizations such as the Bank of Canada and programs such as Employment Insurance. These data are used to meet legal requirements and international obligations. LSP outputs are an important contributor to the federal government's statutory obligations and policy framework, and to Canada's ability to conduct economic analyses. Because of the government's and country's need for high quality statistics in these topic areas, LSP's involvement in producing these data is fully consistent with federal roles and responsibilities. There will continue to be a demand for labour statistics on the part of governments, the private sector, academia, the not-for-profit sector, and Canadians interested in the economy. Continued interest in the state of the nation's economic health and the high rate of use of LSP data indicate that LSP's products will continue to make an important contribution.

Performance (Achievement of Outcomes)

LSP is achieving its outcomes. Labour statistics were considered to be accessible and timely. The quality of labour statistics data is high. A large majority of users are satisfied with the accuracy, interpretability and coherence of the data. LSP statistics are used to inform debate, research, and decision-making. Its information has been and continues to be used for purposes as varied as economic monitoring and forecasting, program delivery and monitoring, wage settlement, contract escalation, as well as to inform the general public.

Education Statistics Program (ESP)

Relevance

ESP provides information related to education, training and learning as well as a number of national indicators and some internationally comparable pan-Canadian education indicators. It also provides information that contributes to other Statistics Canada programs and fulfils international obligations. ESP plays a key role by aggregating and harmonizing information, thereby making inter-jurisdictional and international comparisons possible. Internationally, organization like the OECD and United Nations Statistics Division rely on data produced by ESP. Given the importance of education internationally, the demand for information on education statistics is expected to continue. Also, factors such as low population growth, an aging labour force, and rising demand for skilled workers will likely drive continued research into educational matters, furthering the demand for these statistics.

Performance (Achievement of Outcomes)

Users are satisfied to a large extent with the accessibility and timeliness of Education Statistics Program data. They are also satisfied with the accuracy, interpretability and coherence of the data. ESP data are useful to a range of organizations domestically and internationally. Thousands of pages on the Web and documents in scientific literature have cited ESP products.

Recommendations

Recommendation 1 (Income Statistics Program):

The Income Statistics Program should increase its communication with certain of its user groups, to address concerns related to information on changes to methodologies and quality of some data, and resolve issues to the extent possible. This communication could be achieved using existing committee structures, memoranda of understanding, service-level agreements, and other suitable mechanisms.

Recommendation 2 (Tourism Statistics Program):

  1. In light of a low response rate for the International Travel Survey, the program should address issues related to methodologies and collection approaches to the Travel Statistics Program.
  2. Considering the relative importance of cost recovery revenues for the Tourism Statistics Program, it is important to ensure that a reduction in cost recovery does not prevent the program from producing the statistics needed by Statistics Canada. LEIT management should develop a contingency plan for action if it were to lose a significant volume of revenue-generating clients. The plan could include steps such as the reallocation of resources and working with internal clients to prioritize the statistical inputs required to support Statistics Canada programs.

Management Response and Action Plan

Recommendation 1

Focus: Income Statistics Program:

The Income Statistics Program should increase its communication with certain of its main user groups, to address concerns related to information on changes to methodologies and quality of some data, and resolve issues to the extent possible. This communication could be achieved using existing committee structures, memoranda of understanding, service-level agreements, and other suitable mechanisms.

Statement of Agreement / Disagreement

Management agrees with the proposed recommendation.

Management Response

Work is currently underway to consult more widely with user groups for particular surveys for which ISD is responsible. For example, a steering committee for key Statistics Canada users of the Survey of Household Spending (SHS) has been established and meets approximately every 3 to 4 months to discuss SHS survey results and design questions. In addition, a number of bi-lateral meetings have taken place with key internal SHS stakeholders to discuss and prioritize changes to the survey design. Further, external consultations have also taken place with key external SHS users to discuss the SHS design (e.g., Finance Canada, Bank of Canada (upcoming), System of National Accounts and Consumer Price Index external advisory committees (upcoming)), and will continue in the future.

For other surveys within ISD, steering committees (e.g., for the Longitudinal and International Study of Adults), as well as other forms of communication with clients, stakeholders and key users will continue as required. Should a major survey redesign be undertaken, a consultation process, where applicable, will be conducted.

Table 1 Recomendation 1
Timeline Deliverable(s) Responsible Party
September 2015 A review of the ISP steering committee structure/survey consultation process (both internal and external) will be conducted in 2015 to ensure the proper governance mechanisms are in place for the division. Director General, ELISB

Director, Income Statistics Division

Recommendation 2

Focus: Tourism Statistics Program:

  1. In light of a low response rate for the International Travel Survey, the program should address issues related to methodologies and collection approaches to the Travel Statistics Program.
  2. Considering the relative importance of cost recovery revenues for the Tourism Statistics Program, it is important to ensure that a reduction in cost recovery does not prevent the program from producing the statistics needed by Statistics Canada. LEIT management should develop a contingency plan for action if it were to lose a significant volume of revenue-generating clients. The plan could include steps such as the reallocation of resources and working with internal clients to prioritize the statistical inputs required to support Statistics Canada programs.

Statement of Agreement / Disagreement

Management agrees with the proposed recommendation.

Management Response

Various initiatives have been initiated in 2014-2015 to mitigate risks associated with the potential loss of additional funding partners, but also to seek out alternative methodologies and collection approaches to the Travel Statistics Program.

  1. Three-year funding was received from Statistics Canada's planning process to examine alternate collection approaches to tourism statistics including identification of key data needs for internal purposes and feasibility of merging survey tools for international and domestic travel by Canadians.
  2. The process flow for the capture and processing of border clearance cards (E-311) from the Canadian Border Services Agency (CBSA) is being examined to identify opportunities for streamlining in partnership with CBSA.
  3. The process flow of the monthly frontier counts component for the International Travel Program is being examined to identify potential automation.
Table 2 Recomendation 2
Timeline Deliverable(s) Responsible Party
June 2015 Feasibility study for the merging of the International Travel Survey (ITS - Canadian travellers) and the Travel Survey of Residents to Canada (TSRC) Director General – ELISB / Director TCESD / Chief, Tourism Statistics Program
August 2015 New contract for the capture and processing of the border clearance cards taking into account revised requirements Director General – ELISB / Director TCESD/ Chief, Tourism Statistics Program in collaboration with CBSA and Services Canada
April 2017 Collection and processing tools development of the new combined survey completed

Automation of the monthly frontier counts process
Director General – ELISB / Director TCESD/ Chief, Tourism Statistics Program
December 2017 Testing of the new household tourism survey completed including a 3-month field test Director General – ELISB / Director TCESD/ Chief, Tourism Statistics Program

The full report is available upon request. Please contact: AEB-Professional-Practices@statcan.gc.ca

Date modified: