The 2012 Canadian Survey on Disability (CSD) includes a set of disability screening questions that were used for the first time to identify persons with a disability in Canada. Although some data users may be seeking to compare the prevalence of disability between surveys, and particularly with the CSD's predecessor – the 2006 Participation and Activity Limitation Survey (PALS) – there are many reasons why this is not possible.
The concepts and methods used to measure disability in the 2012 CSD represent a significant change from those used in the 2006 PALS. The most important change is that the two surveys used a different definition of disability. In the CSD, the definition was applied by using the new set of disability screening questions (DSQ). These screening questions reflect a fuller implementation of the social model of disability, greater consistency in disability identification by type, and improved coverage of the full range of disability types, especially mental/psychological and cognitive (learning and memory) disabilities.Note 1 Differences are discussed in more detail below.
Because of the major differences in concepts and methods between the 2006 PALS and the 2012 CSD, it is neither possible nor recommended to compare the prevalence of disability over time between these two sources.
New method of screening for disability
In contrast to the PALS screening questions which used a hybrid approach—a social model for identifying some types of disabilities and a medical model for other types— the CSD screening questions (DSQ) were designed to provide greater consistency in disability identification by type.
Based on their responses to the DSQ, respondents are identified as having a disability only if their daily activities are limitedNote 2 as a result of an impairment or difficulty with particular tasks. For a more detailed explanation of how disability is defined in the CSD, please refer to Section 2.1 of the Canadian Survey on Disability, 2012: Concepts and Methods Guide (Catalogue No. 89-654), forthcoming.
The CSD (by adopting the DSQ) allows respondents to determine whether they face activity limitations as a result of these difficulties or impairments. Some people who indicate that they have some difficulty with certain tasks or have an impairment of some type go on to indicate that this never interferes with their daily activities. In PALS, these individuals were considered to have a disability, but in the CSD, they are not.
This change will have the greatest impact on the identification of persons with sensory and physical disabilities because the PALS identified disabilities in these areas solely on the basis of an indication of some difficulty. At the same time, for certain non-physical disability types, PALS is closer to the CSD because it did have the added requirement of a limitation of activities.
Other changes to screening questions may also have an impact on results. For example, the questions regarding mental/psychological disabilities have been altered somewhat by including examples of the more prevalent conditions (such as depression, anxiety, and bipolar disorder) and excluding examples of less prevalent conditions which are also more highly stigmatized (such as schizophrenia). Changes were also made to the list of examples in questions pertaining to learning disabilities and memory disabilities.
An additional difference between the surveys involves the identification of communication disabilities which was done in PALS but not in the CSD. For the DSQ, no question could be found in successive rounds of qualitative testing to properly identify persons with communication disabilities. Most iterations of a question to identify this small group (including the PALS question itself) yielded difficulties due to people having neither English nor French as their first language or to cultural difficulties (for example, not understanding colloquial references). As well, the advent of social media as a form of communication appears to have added new complexities to the concept of communication for the Canadian population.
Finally, changes were made to the concept of "agility" used by PALS. In the DSQ, this type of disability was split into two types: flexibility and dexterity, since qualitative tests showed that people find that these two tasks are quite different from each other and relate to different underlying conditions. This split was considered an improvement in the identification of different physical disabilities and was also in response to requests made by disability data users and by the Employment and Social Development Canada Persons with Disabilities Technical Advisory Group (TAG).
Other changes to CSD content
Just as the prevalence of disability and the prevalence of certain types of disabilities cannot be compared across surveys, data regarding other content of the CSD cannot be compared with PALS either. The content of the CSD has been streamlined and updated to a large extent. Some of the content from the PALS was cut due to operational constraints; however, every effort was made to ensure that most of the cuts were restricted to content that had been less informative and less utilized.
Survey questions were also updated to better reflect current realities and to correct known weaknesses in the PALS. For example, the section on aids and assistive devices has undergone major changes. Many of the items contained within the PALS were considered out of date in terms of their current usage by people with disabilities. Similarly, new items were added to better reflect technological advancements that have happened since the PALS questions on aids and assistive devices were originally developed in the late 1990s.
Efforts were also made to streamline the method by which CSD respondents were asked about their requirements and unmet needs. For example, questions regarding the need for and use of household fixtures (such as grab bars, etc.) by persons with certain types of physical disabilities are now combined with other aids and devices for the same disability type. The PALS separated questions about aids and devices that were portable and attached to the persons themselves from those attached to homes. This separation was unnatural for many individuals who wanted to indicate a need for items such as grab bars in the earlier section on aids/devices (often reported under "other") and then reported them again in a later section when specifically prompted, possibly leading to some double counting.
These changes to questionnaire wording and flows mean that comparisons should not be made between the PALS and the CSD data.
NHS filter questions
While the CSD and the DSQ are considered to be a big step forward in improving the measurement of disability using the social model, it should be noted that the CSD sample was pre-filtered using the same filter questions on the 2011 National Household Survey (NHS) as those used on the 2006 Census long form for the PALS. Follow up studies have shown that these filter questions do not adequately identify people with mental/psychological or cognitive disabilities. This means that the CSD continues to have some of the weaknesses that the PALS had with respect to undercoverage of some disability types. Nevertheless, of those screened in by the NHS, the new method of screening on the CSD will help improve the identification of persons with mental/psychological, cognitive and "other" types of disabilities because they can now be better identified.
Change in lag time from the filtering survey
As mentioned above, both PALS and CSD derived their sample frames from the answers to the long-form Census in 2006 and the NHS in 2011. The PALS questionnaire was administered between six and nine months after Census data were collected. The CSD, on the other hand, was in the field 16 to 20 months after the NHS data were collected. This difference in lag time not only made it more difficult to track selected respondents who had moved, but it also increased the possibility that a respondent who had reported an activity limitation at the time of the NHS may no longer have a disability, may have been institutionalized or may have died during that time (see next section "Other methodological changes"). In addition, some information appended from the NHS to the CSD file (for example, information on income) may have changed during the period of time between the two surveys.
Other methodological changes
The CSD sampling frame was built based on answers to the 2011 NHS, while that of PALS was taken from the 2006 long-form Census. Although every effort was made by the NHS to minimize impacts due to a lower response rate, the CSD results may have been impacted by this change. For a full discussion of data quality for the NHS, please refer to the National Household Survey User Guide.
A second methodological difference of the CSD compared to the PALS involved a change in the weighting strategy applied to the CSD to compensate for the longer time lag between the NHS collection and the CSD collection. As mentioned earlier, this time lag increased the likelihood of non-response in the CSD due to death or institutionalisation. As many of these cases may have been persons with a disability, it was important to ensure that disability prevalence not be underestimated. The weights of the population who said NO to the NHS filter questions were, therefore, adjusted to take into account deaths and institutionalizations that would have occurred between the NHS collection and CSD collection. This required a calibration of the weights to population estimates adjusted for net undercoverage, which was not done in the 2006 PALS.
For further details on the methodology used in the CSD, please refer to the 2012 Canadian Survey on Disability: Concepts and Methods Guide.
Summary and recommendation
As discussed above, the main differences between the PALS and the CSD can be summarized as follows:
- The definition of disability used in the CSD is different from PALS. The CSD has adopted the newly developed DSQ which is being used for the first time to identify disability in Canada.
- Screening questions in the CSD more closely reflect a social model of disability than do the PALS screening questions. They are also consistent across all types of disabilities, unlike the PALS questions.
- Questionnaire content has been streamlined and updated to reflect current technology and to correct weaknesses in question wording.
- The longer lag time between the NHS and CSD follow-up increased the possibility that selected respondents no longer had a disability, were institutionalized or died during that time. This required a different method for calibration of weights which was not done in the 2006 PALS.
- Finally, the sampling frame for the CSD was derived from the 2011 NHS rather than the 2006 Census and so CSD results may have been impacted by this change.
All of these changes should be assumed to affect comparability of the surveys. Comparison of CSD data to PALS data is, therefore, neither possible nor recommended.
Notes
- This is true of the DSQ module on its own; however some constraints are associated with it being administered as a component of the 2012 CSD, a post-censal survey. See section "NHS filter questions".
- The only exception to this is for developmental disabilities where a person is considered to be disabled if the respondent has been diagnosed with this condition.