Publications

Note to Users of Data from the 2012 Canadian Income Survey

Note to Users of Data from the 2012 Canadian Income Survey

[Full article in PDF]

Skip to text

Text begins

Introduction

Governments use income statistics to develop income support programs and social services, such as the Canada Child Tax Benefit, Employment Insurance, provincial income supplements, and welfare payments.

Private sector and public sector researchers as well as academics also use data on income, low income, income inequality and earnings from income surveys to study labour markets, industrial patterns or changes in family situations.

Starting with the 2012 reference year, annual individual and family income data is produced by the Canadian Income Survey (CIS). The CIS is a new cross-sectional survey developed to provide information on the income and income sources of Canadians, along with their individual and household characteristics.

The CIS reports on many of the same statistics as the Survey of Labour and Income Dynamics (SLID), which last reported on income for the 2011 reference year. Prior to SLID, income data came from the Survey of Consumer Finances (SCF) Note 1.

The 2012 CIS uses a different methodology compared to that used in SLID . Comparisons of CIS and SLID reveal differences in estimates between 2011 and 2012 which are attributable to the two surveys having different methods, rather than a true change in the characteristics of the population.

Statistics Canada will release, in July 2015, results from the 2013 CIS. This will be a first step towards the generation of a new income series using CIS data.

As annual statistics on income are more informative when comparisons can be made over time, by December 2015, Statistics Canada will release a revised series of income statistics which will allow for the comparison of 2012 and 2013 data to earlier years.

Until revised historical statistics are prepared and analysed to ensure that they are as comparable as possible to the current CIS results, the results of the Canadian Income Survey should not be compared to those produced by the Survey of Labour and Income Dynamics or other previous income surveys.

It is valid to compare estimates for different subpopulations within the CIS, such as estimates for different provinces or demographic groups.

This note describes the CIS methodology, as well as the main differences in survey objectives, methodology and questionnaires between CIS and SLID. Indicators of data quality are compared where appropriate.

Methodology of CIS

The CIS is an annual supplement to the Labour Force Survey (LFS), with the first collection undertaken in 2013 for the 2012 reference year. For four consecutive months, LFS respondents in their last month of LFS collection were selected for the CIS. The CIS is a live LFS supplement, meaning it consists of a brief interview conducted shortly after the LFS interview.

In order to reduce burden for respondents, as well as improve the accuracy of the data, CIS does not ask respondents questions on every aspect of their income. Rather, CIS retrieves this information from tax records. CIS respondents are informed of these plans during the interview, a practice which is called informed replacement. As a result, respondents were asked only a minimum number of questions related to income, as well as other questions from related subjects such as housing and disability.

This interview information is combined with information from the LFS, as well as information obtained from the respondent’s income tax records to produce estimates of income for the individual and family.

Unlike the LFS, the CIS is a voluntary survey, meaning that some LFS respondents may choose not to participate in the CIS. In total, information for Canadians in 25,536 households was collected for the 2012 CIS.

The CIS target population consists of all persons living in Canada, excluding persons living in the territories or persons living on reserves and other Aboriginal settlements in the provinces, full-time members of the Canadian Forces living in military camps and persons living full-time in institutions. CIS information is collected for all household members 16 years of age and older.

Differences between CIS and SLID

Survey objectives

The objective of the CIS is to produce annual income statistics on the Canadian population. Like most surveys, CIS is a cross-sectional survey in which respondents are interviewed only once, and each year new respondents are selected and interviewed.

Like CIS, SLID had the objective to produce annual income statistics. SLID also had a second key objective which was to answer a broad range of research questions such as: how do the economic situations of individuals and families change over time with changes in their paid work, family make-up, receipt of government transfers or other factors? To provide the data to answer complex questions such as this, SLID interviewed the same respondents each year for a six year period. This type of survey is called a longitudinal survey.

Because of the need to satisfy both cross-sectional and longitudinal purposes, SLID had a more complex methodological design compared to the CIS. Correspondingly, having a single objective and a simple cross-sectional design is an advantage of CIS for the production of annual estimates. Some of the major differences in the methodology are discussed in the following sections.

Design and Collection

Both the SLID and CIS use the LFS sample for its frame. As a live supplement to the LFS, the CIS is collected shortly after the LFS interview.

SLID, on the other hand, was a stand-alone survey and respondents were contacted several months after their last LFS interview, and then re-contacted for a new interview each year for a total of six years.

SLID selected a new group of respondents every three years, so at any point in time, half of the sample was relatively new. Nevertheless, in any longitudinal survey, there is attrition, which is the tendency for respondents in longitudinal surveys to drop out of the survey over time. Attrition can be caused by difficulties in re-contacting respondents year after year or by respondent fatigue. Attrition could cause SLID to be less representative of the Canadian population than would be the case in the cross-sectional CIS design.

The longitudinal design also meant that there was undercoverage of recent immigrants in SLID, specifically immigrants who arrived between the time the sample was selected and the interview was conducted. This characteristic of SLID is not present in CIS.

Differences in design and collection of CIS and SLID mean that it is impossible to define final response rates to the two surveys in a comparable way. What is possible is to compare “collection” response rates — these are the share of households in the sample sent for collection with responses for at least one household member. This rate could be seen as an indicator of Statistics Canada’s ability to contact desired respondents, as well as these respondents’ willingness to participate in the survey. The collection response rate for CIS was 76.4% compared to 69.4% among new respondents to SLID in 2011Note 2.

Informed Income Replacement

The method of collection of income information differed between the two surveys. CIS respondents were informed during the interview of plans to combine their survey data with tax data, a practice which is called informed replacement. Since information from the tax files would be used more often, respondents were asked only a minimum number of questions related to income Note 3.

In SLID, respondents were asked permission to link to their income tax records. If permission was not granted, then income information was collected directly from survey respondents though survey questions.

The approach used in CIS results in reduced respondent burden and further improvements in the accuracy of the data. The differences in these strategies meant that in 2012, tax data was used in 89.0% of CIS cases, compared to 83.4% of SLID cases in 2011, 81.5% in 2010 and 79.3% in 2009.

Other Content and Concepts

As with other live LFS supplements, the length of the CIS interview is limited. Consequently, some questions that had been asked in SLID had to be modified or dropped altogether from the CIS. At the same time, key background variables such as age, educational attainment, family structure and main job characteristics now come directly from the LFS portion of the interview, and the new CIS can focus on collecting information needed to produce quality estimates of income.

Some of the main differences in content and concepts between CIS and SLID are listed below:

  • In CIS, the “main job” refers to the main job during the week prior to the interview, whereas in SLID, the main job referred to the main job during the reference year.
  • CIS obtains annual labour force activity variables using different questions than SLID.
  • Most housing content is similar between SLID and CIS. However, because information for renters comes from the LFS rent component, the questions related to subsidized housing are different.
  • CIS relies on LFS information to determine family types which, in some cases, does not allow for the same precision as was available in SLID.

Income Imputation

Imputation processes rely on knowing some characteristics of respondents that can be used to estimate other characteristics that are missing due to non-response. CIS was designed to use administrative data as its main source of income data, and as a result less imputation was required. In CIS in 2012, income information was fully or partially imputed for respondents 12.3% of the time. In SLID in 2011, income information was fully or partially imputed 21.2% of the time.

When imputation was required, the income imputation generally followed the same strategy for both the CIS and SLID, with a few differences. For example, the imputation of Employment Insurance, social assistance and Workers’ Compensation income was handled differently for the two surveys. In SLID, respondents were asked detailed questions regarding whether a respondent received any of these three sources of income. If the respondent answered affirmatively, then a non-zero value was imputed for records showing zero tax-value for these sources. For the CIS, respondents were only asked about social assistance and, as result, imputation was only undertaken for social assistance. In CIS, values for Employment Insurance and Workers’ Compensation were taken exclusively from the tax data.

Weighting

Due to the longitudinal nature of SLID, a more complex weighting strategy was required. Two areas where weighting steps were different between the two surveys are non-response adjustment and calibration.

In non-response adjustment, weights of respondents are adjusted based upon information gathered from non-respondents. SLID and CIS both used information collected during the respondent’s LFS interview for non-response adjustment. Some differences arose from the facts that (1) for SLID, the LFS interview information was less current than for CIS and (2) SLID additionally used some paradata (information collected during the SLID collection period about non-respondents) which was not available for the CIS.

In calibration, weights are adjusted so that weighted estimates from the survey match the estimates from other reliable sources. Some improvements to the calibration method were adopted with the new CIS design which would make weighted population totals from CIS match better with totals for other sources such as the Census Note 4. For example, calibration to counts of households of size 3+ as well as total population counts in Montreal, Toronto, Winnipeg, Calgary, Edmonton and Vancouver was introduced for the CIS.

One important quality indicator of a survey is commonly referred to as slippage. Slippage is a measure of survey coverage error. It is defined as the percentage difference between control totals (postcensal population estimates) and weighted sample counts. In 2012, CIS covered 90.6% of its target population compared to 87.4% for SLID in 2011.

Summary

The Canadian Income Survey, first available for reference year 2012, produces high quality annual statistics on individual and family income using a robust methodology. Statistics available from the CIS include the most used and accessed statistics on annual income at the national, provincial and Census Metropolitan Area level, such as:

  • market, total, and after-tax income for economic families, census families and households;
  • income from provincial and federal government transfers;
  • federal and provincial taxes paid;
  • low income statistics according to LIM, LICO and MBM based thresholds Note 5;
  • individual market earnings.

The CIS reports on many of the same statistics as the Survey of Labour and Income Dynamics (SLID), which last reported on income for the 2011 reference year.

This document described some of the key methodological differences between CIS and SLID which users should be aware of when using the new data. As annual statistics on income are more informative when comparisons can be made over time, by December 2015, Statistics Canada will release a revised series of income statistics which will allow for the comparison of 2012 and 2013 data to earlier years.

Until revised historical statistics are prepared and analysed to ensure that they are as comparable as possible to the current CIS results, the results of the Canadian Income Survey should not be compared to those produced by the Survey of Labour and Income Dynamics or other previous income surveys.

Notes

Date modified: