Building on Our Competencies: Canadian Results of the International Adult Literacy and Skills Survey: Introduction

Literacy, numeracy and problem solving are essential to function in today's world. For individuals, they are the key to realizing their full economic and social potential and the foundation upon which they acquire additional knowledge and skills throughout adulthood. They are strongly associated with individual outcomes and enable people to participate in their communities, make wise consumer decisions, and construct social networks.

Proficiency in literacy, numeracy and problem solving enhance the capacity of individuals to participate in lifelong learning and to improve productivity. In a global economy, a highly skilled population is an asset and fundamental to economic growth. Factors such as globalization, the impact of new technologies and the move to a knowledge economy have led the occupational composition in Canada to shift towards more highly skilled occupations requiring higher levels of education. A knowledge economy requires workers who can adapt quickly to the changing skills requirements in the labour market.

The social implications of literacy, numeracy and problem solving are no less important. Understanding the link between these competencies and civic participation, for example, has important implications for building strong communities and social institutions. Inadequate proficiencies in literacy, numeracy or problem solving increase the risk of exclusion for particular societal groups.

Canada, as well as many other countries, invests heavily in compulsory education in order to build a strong foundation of literacy, numeracy and problem solving. Measuring the distribution of these competencies in the population can assist individuals, employers, communities and governments in making decisions that are crucial to achieving high rates of economic growth, reducing inequalities in economic and social outcomes, and promoting social inclusion.

Previous international research has already shown that most of the differences in the level and distribution of skill can be explained by social background, education and a range of factors that reflect how adults lead their lives. Further, differences in the level and distribution of skill have been found to be associated with large differences in outcomes in multiple facets of life - work, education, home and the community. And finally, large differences in skills exist both within and between countries.

This report presents the results of the 2003 International Adult Literacy and Skills Survey (IALSS) that aimed to measure the proficiencies in literacy, numeracy and problem solving of the Canadian population. It focuses on differences within Canada rather than those observed between countries. The IALSS sample is large enough to present the skills distributions of the population of each of the ten provinces and three territories and of specific subpopulations, such as immigrants, Aboriginal people and minority language groups. The report also analyses the relationships between socio-demographic characteristics, such as age, education, type of work and income, and performance in literacy, numeracy and problem solving.

Text Box A

Literacy is a continuum

The 2003 International Adult Literacy and Skills Survey, like its predecessor, the 1994 International Adult Literacy Survey is not a survey aimed at distinguishing those who are "literate" from those who are "illiterate". There is no arbitrary standard used in the IALSS to distinguish adults who have proficiency from those who do not. The IALSS measures literacy and numeracy along a continuum of proficiency that indicates how well adults use information in today's society.

What is the IALSS?

Conducted in 2003, the International Adult Literacy and Skills Survey (IALSS) is the Canadian component of the Adult Literacy and Life Skills program (ALL). The ALL program is a ". large-scale co-operative effort undertaken by governments, national statistical agencies, research institutions and multi-lateral agencies" that provides internationally comparable measures in four domains: prose and document literacy, numeracy and problem solving (OECD and Statistics Canada, 2005). Over 23,000 individuals aged 16 and over from across the ten provinces and three territories responded to the Canadian IALSS.

Text Box B

The Adult Literacy and Life Skills Program (ALL)

The development and management of the ALL study were co-ordinated by Statistics Canada and the Educational Testing Services (ETS, Princeton, United States) in collaboration with the National Center for Education Statistics (NCES) of the United States Department of Education, the Organization for Economic Co-operation and Development (OECD) and the Institute for Statistics (UIS) of the United Nations Educational, Scientific and Cultural Organization (UNESCO).

The ALL survey, undertaken during the first half of 2003, required all participating countries¹ to collect data from a nationally representative sample of at least 3,000 respondents aged 16 to 65 for each language tested - English and French in the case of Canada. The minimum sample requirements for the ALL survey were exceeded in Canada because several federal agencies and provincial governments funded the collection of additional cases so as to ensure high reliability in the estimation of data values for small population groups. Moreover, unlike the 1994 IALS,² the 2003 Canadian IALSS also benefited from contributions made by territorial governments. As a result, the number of respondents is sufficient to provide accurate estimates for the Yukon, Northwest Territories and Nunavut (a complete list of Partners to the IALSS can be found in Annex D). Finally, as with the 1994 IALS, the 2003 IALSS added Canadians over the age of 65 to the sample. Over 23,000 individuals from across Canada spent an average of two hours responding to the IALSS. Annex Table I.3 shows the actual and weighted distributions of respondents from across Canada.

Every respondent was first given a common questionnaire seeking information about demographic characteristics and variables such as educational attainment, occupation, income, and engagement in adult learning and community activities. The respondents were then given an internationally validated psychometric³ instrument designed to measure proficiency in four domains:

The IALSS builds on its predecessor, the 1994 International Adult Literacy Survey (IALS). The IALSS numeracy scale expands the quantitative literacy domain measured in 1994 and is a broader, more inclusive measure of mathematics skills and knowledge. Problem solving is a new domain in 2003. The prose and document literacy scales used in 2003, however, are identical to those carried by the IALS in 1994. Thus, for several countries including Canada, it is now possible to examine both the current distributions of prose and document literacy and how these have evolved between 1994 and 2003.

Text Box C

The four domains: Prose and document literacy, numeracy and problem solving

The same prose and document literacy scales are used in both the 1994 IALS and the 2003 IALSS. For both domains, the proficiency scales from the two surveys were linked through the inclusion of a subset of test items originally used in 1994. Thus, for several countries including Canada, the current distributions of prose and document literacy can be compared to those in 1994 to see how these have evolved.

The 2003 IALSS numeracy scale builds on the quantitative literacy domain measured in 1994, providing a broader, more inclusive measure of mathematics skills and conceptual mathematical knowledge. This expanded scale measures more than the ability to perform mathematical operations on numbers embedded in text by including many tasks that require no or little reading.

Finally, the IALSS carried tasks to assess proficiency in problem solving. This new domain was validated through rigorous testing and displays unique characteristics not found in the other measures. To some extent, it requires the integration of the knowledge and skills measured by the literacy and numeracy domains and their application to new situations. It also implicates basic logical tools needed to provide effective solution strategies to the problems presented in everyday life. These include the ability to order, evaluate and prioritize a series of factors and to discriminate, plan, analyze and reason through a variety of choices in order to arrive at an effective solution to a given problem.

The conceptualization and definitions of the four domains as well as examples of actual test items used in the assessment are presented in Annex B. Readers requiring additional technical information on the psychometric aspects of the study are referred to Statistics Canada (2004).

How to interpret the IALSS

Like the IALS before it, the 2003 IALSS conceptualizes proficiency along a continuum that denotes how well adults use information to function in society and the economy. The IALSS does not measure the absence of competencies rather it measures knowledge and skills in the four domains along a broad range of ability. Consequently, the results cannot be used to classify population groups as either "literate" or "illiterate".

Proficiency in each domain is measured on a continuous scale. Each scale starts at zero and increases to a theoretical maximum of 500 points. Scores along the scale denote the points at which a person with a given level of performance has an 80 percent probability of successfully completing a task at that level of difficulty (see Text Box D).

Useful summary statistics can be derived that describe the competencies of populations such as their overall average score. Populations with similar average scores, however, may have quite different numbers of low or high performing adults. Thus, one can also look at how the scores are distributed within populations by using percentile scores. Percentile scores are the scores below which a specified percentage of adults are found. Thus, for example, the 5th percentile score is the one below which we find 5 percent of adults in a particular population. Differences in percentile scores tell us something about the degree of equality in proficiency across populations.

The IALSS scores are also grouped into proficiency levels representing a set of tasks of increasing difficulty (see Table I.1). For the prose and document literacy domains as well as the numeracy domain, experts have defined five broad levels of difficulty, each corresponding to a similar range of scores. For the problem solving domain, experts have defined four broad levels of difficulty. In each domain, Level 1 denotes the lowest proficiency level and Level 4/5 the highest.

It is important, for analytical as well as operational reasons, to define a "desired level" of competence for coping with the increasing skill demands of the emerging knowledge and information economy. Level 3 performance is generally chosen as a benchmark because in developed countries, performance above Level 2 is generally associated with a number of positive outcomes. These include increased civic participation, increased economic success and independence, and enhanced opportunities for lifelong learning and personal literacy (Kirsch, I., et al., 1993; Murray, T.S. et al., 1997; Tuijnman, A., 2001). Whereas individuals at proficiency Levels 1 and 2 typically have not yet mastered the minimum foundation of literacy needed to attain higher levels of performance (Strucker, J., Yamamoto, K. 2005).

Secondary analysis of the 1994 IALS data has yielded consistent evidence that the performance difference between Level 2 and Level 3 on the prose, document and quantitative literacy scales is substantive and corresponds to a significant difference in measurable benefits accruing to citizens in OECD countries (OECD and HRDC, 1997). Results of preliminary analysis of the IALSS data, including the new numeracy scale, are consistent with this finding. For this reason, some of the analyses contained in this report anchor the scales at the cut point between Levels 2 and 3, thus highlighting the distributions above and below this threshold for the prose, document and numeracy domains. In contrast, interpretation of the problem solving domain (see Table I.2) is more complex and no single "desirable" threshold has yet been set.

Thus, the tables and charts included in this report provide multiple ways to examine how the distributions of competencies differ across Canada.

Text Box D

Measuring proficiency

For IALSS, each proficiency scale starts at zero and increases to a theoretical maximum of 500 points. Scores along the scale denote the points at which a person with a given level of performance has an 80 percent probability of successfully completing a task at that level of difficulty. For instance, a person with an assessed performance at 250 points has an 80 percent probability of correctly answering a task with an estimated difficulty level of 250. The same individual would have an "80 percent plus" probability of correctly answering a simpler task (about 95 percent for a task with a complexity of 200) and a diminished probability (less than 80 percent) of successfully completing a more difficult task (about 40 percent for a task with a complexity of 300) (Kirsch, Jungeblut and Campbell, 1992).

Interestingly, while the probability of a correct response may approach zero as the tasks become more difficult, it can never quite reach it because there is always some chance, however small, that a correct answer will be provided regardless of ability. Accordingly, the results presented in this report measure performance along a proficiency continuum. The scales do not measure the absence of a competence, and thus cannot distinguish those who have from those who lack a specific competency.

The proficiency levels used for IALSS are useful in summarizing the results but also have some limitations. First, the relatively small proportions of respondents who actually reach Level 5 do not always allow for accurate reporting. For this reason, whenever results are presented by proficiency level, Levels 4 and 5 are combined. Second, as shown in Tables I.1 and I.2, the levels indicate specific sets of abilities and, therefore, the thresholds for the levels are not equidistant. The ranges of scores in each level are therefore not identical. In fact, for all four domains, Level 1 captures almost half of the scale. The thresholds for the problem solving domain are set somewhat differently and Level 1 covers precisely half of the scale. Level 1 includes all basic abilities required to attain higher levels. In other words, the ability to read may lie somewhere in Level 1, but the ability to understand and use what has been read comes in gradations of complexity from Level 1 to Level 5. The upshot of the relatively large ranges of scores in Level 1 on each of the scales is that there are multiple sub-levels of proficiency within this level. The range includes those who can barely read at all as well as those who read poorly or inattentively.⁴

Organization of the report

Chapter 1 presents international, provincial and territorial comparisons of performance in the four domains measured (prose literacy, document literacy, numeracy, and problem solving). This chapter also presents analysis exploring changes in literacy performance over time. Chapter 2 examines variations in the population distributions of proficiency by various demographic characteristics, including age, gender and education. Chapter 3 analyses the proficiency distributions of population groups of special interest, including Aboriginal and immigrant populations. Chapter 4 analyses the relationships between proficiency and economic outcomes such as labour force participation and income inequality. Chapter 5 presents analytical results concerning the magnitude of the effects of proficiency on social outcomes, notably health and civic engagement and investigates patterns in the relationships between proficiency and the use of information and communication technologies (ICT) in Canada. The key findings are summarized and discussed in the Conclusion.

Data underlying the graphs and figures in the body of the report can be found, often with more details, in the tables of Annex A. A detailed overview of the IALSS proficiency scales - how they are defined, how they were measured, how proficiency was summarized and how proficiency estimates should be interpreted can be found in Annex B. Annex C documents key methodological aspects of the study and, finally, Annex D identifies the various federal, provincial, territorial and international agencies responsible for the IALSS.

Note to readers

The proficiency results are generally reported separately for the four measured scales - prose literacy, document literacy, numeracy, and problem solving.

Given the comparative nature of IALSS, every effort was made to establish the validity, reliability, comparability and interpretability of estimates, and to control and quantify errors that might interfere with or bias interpretation. Notes to figures and tables are used to alert readers whenever errors might affect interpretation.

The data presented in this report are estimated from representative but complex samples of adults in Canada. Additionally, there is a degree of error associated with the measurement of skills because they are estimated on the basis of responses to samples of test items. In the report's data tables, standard errors, in parenthesis next to the actual estimates, express the degree of uncertainty associated with both sampling and measurement errors.

When comparing 2003 IALSS results for Canada to either 2003 data for other countries or to the 1994 IALS, it is necessary to use a sub-sample of the IALSS. On one hand, the international comparisons need to be restricted to the 16 to 65 age group, since Canada is the only country that collected data for the over 65 population. On the other hand, the comparisons to the 1994 IALS need to be restricted to the provinces since the northern territories were not part of the 1994 survey. Each of these sub-samples of the 2003 IALSS displays somewhat different proficiency profiles, which needs to be kept in mind.

Endnotes

1. Participating countries included Bermuda, Canada, Italy, Norway, Switzerland, the United States and the Mexican state of Nuevo Leon.

2. The International Adult Literacy Survey (IALS) was fielded between 1994 and 1998 in 22 countries. Data for Canada were collected in 1994. Results are reported in three volumes, see OECD and Statistics Canada (1995), OECD and HRDC (1997), and OECD and Statistics Canada (2000).

3. Psychometrics refers to the branch of psychology that deals with the design, administration, and interpretation of quantitative tests for the measurement of variables such as intelligence, aptitude, and personality traits.

4. The International Survey of Reading Skills is a follow-up to the 2003 IALSS that will provide more information about respondents at Level 1. Results are expected in 2006.

References

Kirsch, I.S., Jungeblut, A., and Campbell, A. (1992), Beyond the school doors: The literacy needs of job seekers served by the US Department of Labor. Princeton, New Jersey : Educational Testing Service (ETS), U.S. Department of Labor, Employment and Training Administration.

Kirsch, I., Jungeblut, A., Jenkins, L., and Kolstad, A. (Eds.) (1993). Adult Literacy in America : A first look at the National Adult Literacy Survey. National Center for Education Statistics, US Department of Education, Washington, DC.

Murray, T.S., Kirsch, I., and Jenkins, L. (Eds.) (1997) Adult Literacy in the OECD Countries: Technical Report of the First International Adult Literacy Survey. National Center for Education Statistics, US Department of Education, Washington, DC.

OECD and Statistics Canada (1995), Literacy, economy and society: Results of the first International Adult Literacy Survey. Paris and Ottawa : Authors.

OECD and HRDC (1997), Literacy skills for the knowledge society: Further results of the International Adult Literacy Survey. Paris and Hull : Authors.

OECD and Statistics Canada (2000), Literacy in the information age: Final results of the International Adult Literacy Survey. Paris and Ottawa : Authors.

OECD and Statistics Canada (2005), Learning a living: First results of the Adult Literacy and Life Skills Survey. Paris and Ottawa : Authors.

Statistics Canada (2004), Measuring Adult Literacy and Life Skills: New Frameworks for Assessment.

Strucker, J., and Yamamoto, K. (2005). Component Skills of Reading : Tipping Points and Five Classes of Adult Literacy Learners, (unpublished).

Tuijnman, A. (2001). Benchmarking Adult Literacy in North America : An International Comparative Study. Ottawa : Statistics Canada and Human Resources Development Canada.

Archived Content

Introduction