Publications

    Longitudinal and International Study of Adults Research Paper Series

    Using the Program for International Assessment of Adult Competencies direct measures of skills in the Longitudinal and International Study of Adults

    Using the Program for International Assessment of Adult Competencies direct measures of skills in the Longitudinal and International Study of Adults

    Warning View the most recent version.

    Archived Content

    Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please "contact us" to request a format other than those available.

    Introduction

    In this paper, we demonstrate the use of the Program for International Assessment of Adult Competencies (PIAAC) direct measures of skills that are available in the Longitudinal and International Study of Adults (LISA) data.

    The PIAAC direct measures of skills are available in the LISA data because the two surveys had coordinated collection. The PIAAC direct measures of skills are reflected in a set of 10 plausible values (PVs) for each of the three domains measured by PIAAC: literacy, numeracy, and problem solving in technology-rich environments (PSTRE).

    The first part of this study describes and demonstrates the use of the PVs to obtain group-level estimates and estimates of the standard error. The second part of this study compares the LISA estimates to the PIAAC estimates to assess differences—as only a subset of the PIAAC respondents are included in LISA.

    Part I: PIAAC direct measures of skills in LISA

    The PIAAC test aimed to cover a large volume of content in a limited time. As a result, not all respondents were asked all test questions in the direct assessment. “The test design for PIAAC was based on a variant of matrix sampling (using different sets of items, multistage adaptive testing, and different assessment modes) where each respondent was administered a subset of items from the total item pool.” (Organisation for Economic Co-operation and Development [OECD], 2013b, p. 408)

    The responses to the subset of test items are combined with other background information (provided by the respondent) and model parameters to produce a set of 10 plausible values (PVs). These PVs can be used to produce group-level estimations of proficiency values in literacy, numeracy, and PSTRE.

    Wu and Adams (2002) describe plausible values in a simple way as follows:

    • The simplest way to describe plausible values is to say that plausible values are a representation of the range of abilities that a student might reasonably have. (...). Instead of directly estimating a student’s ability θ,..., a range of  possible values for a student’s θ, with an associated probability for each of these values is estimated. Plausible values are random draws from this (estimated) distribution for a student’s θ.

    A single set of plausible values can be used during exploration of the data; however, to get the best estimate of the statistic and properly capture the variability in the proficiency values, all 10 sets of plausible values should be used—following the instructions provided in the OECD documentation.

    OECD (2013b, Chapter 18.3) provides the following steps to calculate any estimate T of the proficiency values θ using PVs and to calculate an estimate of the variance of T:

    1. Using the first vector of plausible values for each respondent, evaluate T as if the plausible values were the true values of θ. Denote the result T1.
    2. In the same manner as in step 1 above, evaluate the sampling variance of T, or Var(T1), with respect to respondents’ first vectors of plausible values. Denote the result Var1.
    3. Carry out steps 1 and 2 for the second through all 10 vectors of plausible values, thus obtaining Tu and Varu for u=2, . . .,10.
    4. The best estimate of T obtainable from the plausible values is the average of the 10 values obtained from the different sets of plausible values: T . = u T u 10 MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr 4rNCHbGeaGqiVu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq=Jc9 vqaqpepm0xbba9pwe9Q8fs0=yqaqpepae9pg0FirpepeKkFr0xfr=x fr=xb9adbaqaaeGaciGaaiaabeqaamaabaabaaGcbaaeaaaaaaaaa8 qacaWGubWdamaaBaaaleaapeGaaiOlaaWdaeqaaOWdbiabg2da9maa laaapaqaa8qadaqfqaqabSWdaeaapeGaamyDaaqab0WdaeaapeGaey yeIuoaaOGaamiva8aadaWgaaWcbaWdbiaadwhaa8aabeaaaOqaa8qa caaIXaGaaGimaaaaaaa@4033@
    5. An estimate of the variance of T is the sum of two components: an estimate of Var(Tu) obtained as in step 4 and the variance among the Tus: Var( T . )= n Va r u 10 +( 1+ 1 10 ) u ( T u T . ) 2 101 MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr 4rNCHbGeaGqiVu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq=Jc9 vqaqpepm0xbba9pwe9Q8fs0=yqaqpepae9pg0FirpepeKkFr0xfr=x fr=xb9adbaqaaeGaciGaaiaabeqaamaabaabaaGcbaaeaaaaaaaaa8 qacaWGwbGaamyyaiaadkhadaqadaWdaeaapeGaamiva8aadaWgaaWc baWdbiaac6caa8aabeaaaOWdbiaawIcacaGLPaaacqGH9aqpdaWcaa WdaeaapeWaaubeaeqal8aabaWdbiaad6gaaeqan8aabaWdbiabggHi LdaakiaadAfacaWGHbGaamOCa8aadaWgaaWcbaWdbiaadwhaa8aabe aaaOqaa8qacaaIXaGaaGimaaaacqGHRaWkdaqadaWdaeaapeGaaGym aiabgUcaRmaalaaapaqaa8qacaaIXaaapaqaa8qacaaIXaGaaGimaa aaaiaawIcacaGLPaaadaWcaaWdaeaapeWaaubeaeqal8aabaWdbiaa dwhaaeqan8aabaWdbiabggHiLdaakmaabmaapaqaa8qacaWGubWdam aaBaaaleaapeGaamyDaaWdaeqaaOWdbiabgkHiTiaadsfapaWaaSba aSqaa8qacaGGUaaapaqabaaak8qacaGLOaGaayzkaaWdamaaCaaale qabaWdbiaaikdaaaaak8aabaWdbiaaigdacaaIWaGaeyOeI0IaaGym aaaaaaa@5B94@
      The first component in Var(T.) reflects uncertainty due to sampling from the population; the second component reflects uncertainty because the respondents' proficiencies θ are only indirectly observed... (p. 457)

    Literacy skills in LISA

    Following the above steps from OECD (2013b), the estimated average of the literacy proficiency values is shown below. Notice that the statistic calculated using only 1 plausible value is a close approximation of the best estimate using all 10 plausible values; however, the variance of the estimate is underestimated with only 1 plausible value because the variability corresponding to the uncertainly of the respondents’ proficiency is not reflected.

    Step 1 to 3: Calculating the statistic for each of the 10 plausible value sets, and calculate the variance estimate for each statistic using the bootstrap weights. Table 1 shows the results. Annex A contains the SAS code used to obtain the values in this table.

    Table 1 Mean literacy score for 10 plausible value sets
    Table summary
    This table displays the results of Table 1 Mean literacy score for 10 plausible value sets. The information is grouped by Plausible value set i (appearing as row headers), Ti and Variance of Ti (appearing as column headers).
    Plausible value set i Ti Variance of Ti
    1 272.2 0.40
    2 271.3 0.38
    3 271.9 0.38
    4 271.7 0.41
    5 272.0 0.41
    6 271.9 0.39
    7 272.0 0.38
    8 272.2 0.37
    9 271.7 0.38
    10 272.0 0.39

    Step 4: Calculate the best estimate: T . = u T u 10 =271.9 MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr 4rNCHbGeaGqiVu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq=Jc9 vqaqpepm0xbba9pwe9Q8fs0=yqaqpepae9pg0FirpepeKkFr0xfr=x fr=xb9adbaqaaeGaciGaaiaabeqaamaabaabaaGcbaaeaaaaaaaaa8 qacaWGubWdamaaBaaaleaapeGaaiOlaaWdaeqaaOWdbiabg2da9maa laaapaqaa8qadaqfqaqabSWdaeaapeGaamyDaaqab0WdaeaapeGaey yeIuoaaOGaamiva8aadaWgaaWcbaWdbiaadwhaa8aabeaaaOqaa8qa caaIXaGaaGimaaaacqGH9aqpcaaIYaGaaG4naiaaigdacaGGUaGaaG yoaaaa@44E6@

    Step 5: Calculate the estimate of the variance of T.:

    Var( T . )= n Va r u 10 +( 1+ 1 10 ) u ( T u T . ) 2 101 MathType@MTEF@5@5@+= feaagKart1ev2aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr 4rNCHbGeaGqiVu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq=Jc9 vqaqpepm0xbba9pwe9Q8fs0=yqaqpepae9pg0FirpepeKkFr0xfr=x fr=xb9adbaqaaeGaciGaaiaabeqaamaabaabaaGcbaaeaaaaaaaaa8 qacaWGwbGaamyyaiaadkhadaqadaWdaeaapeGaamiva8aadaWgaaWc baWdbiaac6caa8aabeaaaOWdbiaawIcacaGLPaaacqGH9aqpdaWcaa WdaeaapeWaaubeaeqal8aabaWdbiaad6gaaeqan8aabaWdbiabggHi LdaakiaadAfacaWGHbGaamOCa8aadaWgaaWcbaWdbiaadwhaa8aabe aaaOqaa8qacaaIXaGaaGimaaaacqGHRaWkdaqadaWdaeaapeGaaGym aiabgUcaRmaalaaapaqaa8qacaaIXaaapaqaa8qacaaIXaGaaGimaa aaaiaawIcacaGLPaaadaWcaaWdaeaapeWaaubeaeqal8aabaWdbiaa dwhaaeqan8aabaWdbiabggHiLdaakmaabmaapaqaa8qacaWGubWdam aaBaaaleaapeGaamyDaaWdaeqaaOWdbiabgkHiTiaadsfapaWaaSba aSqaa8qacaGGUaaapaqabaaak8qacaGLOaGaayzkaaWdamaaCaaale qabaWdbiaaikdaaaaak8aabaWdbiaaigdacaaIWaGaeyOeI0IaaGym aaaaaaa@5B93@ =0.39+( 1+ 1 10 ) 0.68 101 MathType@MTEF@5@5@+= feaagKart1ev2aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr 4rNCHbGeaGqiVu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq=Jc9 vqaqpepm0xbba9pwe9Q8fs0=yqaqpepae9pg0FirpepeKkFr0xfr=x fr=xb9adbaqaaeGaciGaaiaabeqaamaabaabaaGcbaaeaaaaaaaaa8 qacqGH9aqpcaaIWaGaaiOlaiaaiodacaaI5aGaey4kaSYaaeWaa8aa baWdbiaaigdacqGHRaWkdaWcaaWdaeaapeGaaGymaaWdaeaapeGaaG ymaiaaicdaaaaacaGLOaGaayzkaaWaaSaaa8aabaWdbiaaicdacaGG UaGaaGOnaiaaiIdaa8aabaWdbiaaigdacaaIWaGaeyOeI0IaaGymaa aaaaa@4706@ =0.47 MathType@MTEF@5@5@+= feaagKart1ev2aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr 4rNCHbGeaGqiVu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq=Jc9 vqaqpepm0xbba9pwe9Q8fs0=yqaqpepae9pg0FirpepeKkFr0xfr=x fr=xb9adbaqaaeGaciGaaiaabeqaamaabaabaaGcbaaeaaaaaaaaa8 qacqGH9aqpcaaIWaGaaiOlaiaaisdacaaI3aaaaa@3A07@ SD( T . )=0.69 MathType@MTEF@5@5@+= feaagKart1ev2aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr 4rNCHbGeaGqiVu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq=Jc9 vqaqpepm0xbba9pwe9Q8fs0=yqaqpepae9pg0FirpepeKkFr0xfr=x fr=xb9adbaqaaeGaciGaaiaabeqaamaabaabaaGcbaaeaaaaaaaaa8 qacaWGtbGaamiramaabmaapaqaa8qacaWGubWdamaaBaaaleaapeGa aiOlaaWdaeqaaaGcpeGaayjkaiaawMcaaiabg2da9iaaicdacaGGUa GaaGOnaiaaiMdaaaa@3F53@

    Numeracy skills in LISA

    Numeracy skills are calculated in a similar way as literacy skills. The estimate for the average numeracy proficiency value is shown below.

    Step 1 to 3: Calculating the statistic for each of the 10 plausible value sets. Table 2 shows the results.

    Table 2 Mean numeracy score for 10 plausible value sets
    Table summary
    This table displays the results of Table 2 Mean numeracy score for 10 plausible value sets. The information is grouped by Plausible value set i (appearing as row headers), Ti and Variance of Ti (appearing as column headers).
    Plausible value set i Ti Variance of Ti
    1 264.0 0.48
    2 262.8 0.48
    3 263.2 0.49
    4 263.3 0.49
    5 263.6 0.48
    6 263.3 0.49
    7 263.7 0.45
    8 262.9 0.46
    9 263.6 0.47
    10 263.4 0.48

    Step 4: Calculate the best estimate: T . = u T u 10 =263.4 MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr 4rNCHbGeaGqiVu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq=Jc9 vqaqpepm0xbba9pwe9Q8fs0=yqaqpepae9pg0FirpepeKkFr0xfr=x fr=xb9adbaqaaeGaciGaaiaabeqaamaabaabaaGcbaaeaaaaaaaaa8 qacaWGubWdamaaBaaaleaapeGaaiOlaaWdaeqaaOWdbiabg2da9maa laaapaqaa8qadaqfqaqabSWdaeaapeGaamyDaaqab0WdaeaapeGaey yeIuoaaOGaamiva8aadaWgaaWcbaWdbiaadwhaa8aabeaaaOqaa8qa caaIXaGaaGimaaaacqGH9aqpcaaIYaGaaGOnaiaaiodacaGGUaGaaG inaaaa@44E2@

    Step 5: Calculate the estimate of the variance of T.:

    Var( T . )= n Va r u 10 +( 1+ 1 10 ) u ( T u T . ) 2 101 MathType@MTEF@5@5@+= feaagKart1ev2aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr 4rNCHbGeaGqiVu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq=Jc9 vqaqpepm0xbba9pwe9Q8fs0=yqaqpepae9pg0FirpepeKkFr0xfr=x fr=xb9adbaqaaeGaciGaaiaabeqaamaabaabaaGcbaaeaaaaaaaaa8 qacaWGwbGaamyyaiaadkhadaqadaWdaeaapeGaamiva8aadaWgaaWc baWdbiaac6caa8aabeaaaOWdbiaawIcacaGLPaaacqGH9aqpdaWcaa WdaeaapeWaaubeaeqal8aabaWdbiaad6gaaeqan8aabaWdbiabggHi LdaakiaadAfacaWGHbGaamOCa8aadaWgaaWcbaWdbiaadwhaa8aabe aaaOqaa8qacaaIXaGaaGimaaaacqGHRaWkdaqadaWdaeaapeGaaGym aiabgUcaRmaalaaapaqaa8qacaaIXaaapaqaa8qacaaIXaGaaGimaa aaaiaawIcacaGLPaaadaWcaaWdaeaapeWaaubeaeqal8aabaWdbiaa dwhaaeqan8aabaWdbiabggHiLdaakmaabmaapaqaa8qacaWGubWdam aaBaaaleaapeGaamyDaaWdaeqaaOWdbiabgkHiTiaadsfapaWaaSba aSqaa8qacaGGUaaapaqabaaak8qacaGLOaGaayzkaaWdamaaCaaale qabaWdbiaaikdaaaaak8aabaWdbiaaigdacaaIWaGaeyOeI0IaaGym aaaaaaa@5B93@ =0.48+( 1+ 1 10 ) 1.36 101 MathType@MTEF@5@5@+= feaagKart1ev2aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr 4rNCHbGeaGqiVu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq=Jc9 vqaqpepm0xbba9pwe9Q8fs0=yqaqpepae9pg0FirpepeKkFr0xfr=x fr=xb9adbaqaaeGaciGaaiaabeqaamaabaabaaGcbaaeaaaaaaaaa8 qacqGH9aqpcaaIWaGaaiOlaiaaisdacaaI4aGaey4kaSYaaeWaa8aa baWdbiaaigdacqGHRaWkdaWcaaWdaeaapeGaaGymaaWdaeaapeGaaG ymaiaaicdaaaaacaGLOaGaayzkaaWaaSaaa8aabaWdbiaaigdacaGG UaGaaG4maiaaiAdaa8aabaWdbiaaigdacaaIWaGaeyOeI0IaaGymaa aaaaa@4702@ =0.64 MathType@MTEF@5@5@+= feaagKart1ev2aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr 4rNCHbGeaGqiVu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq=Jc9 vqaqpepm0xbba9pwe9Q8fs0=yqaqpepae9pg0FirpepeKkFr0xfr=x fr=xb9adbaqaaeGaciGaaiaabeqaamaabaabaaGcbaaeaaaaaaaaa8 qacqGH9aqpcaaIWaGaaiOlaiaaiAdacaaI0aaaaa@3A06@ SD( T . )=0.80 MathType@MTEF@5@5@+= feaagKart1ev2aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr 4rNCHbGeaGqiVu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq=Jc9 vqaqpepm0xbba9pwe9Q8fs0=yqaqpepae9pg0FirpepeKkFr0xfr=x fr=xb9adbaqaaeGaciGaaiaabeqaamaabaabaaGcbaaeaaaaaaaaa8 qacaWGtbGaamiramaabmaapaqaa8qacaWGubWdamaaBaaaleaapeGa aiOlaaWdaeqaaaGcpeGaayjkaiaawMcaaiabg2da9iaaicdacaGGUa GaaGioaiaaicdaaaa@3F4C@

    PSTRE skills in LISA

    PSTRE proficiency values require additional consideration than that given to literacy and numeracy proficiency values because a group of respondents did not do the PSTRE assessment. The PSTRE assessment was not done if the person had insufficient computer skills, or the person opted to do a paper-and-pencil-based assessment, or the person did not do the computer assessment for literacy-related reasons. As a result, there are missing values for PSTRE proficiency values that are not addressed through imputation or weighting—as their characteristics are different from those that did complete the assessment.

    Statistics Canada (2013, p. 19) recommends two approaches to conveying the volume of missing values:

    1. Displaying the proportion of the population with a proficiency value when reporting any statistics calculated on the PSTRE values.
    2. Displaying the proportion with missing values when reporting on the full distribution of the population by proficiency levels.

    The example below shows the calculation of the proportion of the population in the different proficiency levelsNote 1 of PSTRE.

    Step 1 to 3: Calculating the statistics for each of the 10 plausible value sets. Table 3 shows the results.

    Table 3 Proportion of population by PSTRE levels for 10 plausible value sets
    Table summary
    This table displays the results of Table 3 Proportion of population by PSTRE levels for 10 plausible value sets. The information is grouped by Plausible value set i (appearing as row headers), Ti,below level 1, Variance of Ti, below level 1, Ti, level 1, Variance of Ti, level 1, Ti, level 2, Variance of Ti, level 2, Ti, level 3 and Variance of Ti, level 3 (appearing as column headers).
    Plausible value set i Ti,below level 1 Variance of Ti, below level 1 Ti, level 1 Variance of Ti, level 1 Ti, level 2 Variance of Ti, level 2 Ti, level 3 Variance of Ti, level 3
    1 0.157 0.000026 0.308 0.000043 0.288 0.000037 0.064 0.000012
    2 0.155 0.000024 0.309 0.000041 0.283 0.000034 0.071 0.000013
    3 0.162 0.000025 0.296 0.000036 0.289 0.000036 0.070 0.000012
    4 0.159 0.000026 0.303 0.000041 0.289 0.000037 0.066 0.000012
    5 0.156 0.000024 0.309 0.000043 0.279 0.000035 0.073 0.000013
    6 0.155 0.000026 0.307 0.000044 0.287 0.000043 0.068 0.000012
    7 0.157 0.000027 0.300 0.000039 0.295 0.000039 0.065 0.000012
    8 0.158 0.000025 0.303 0.000040 0.284 0.000036 0.072 0.000012
    9 0.159 0.000026 0.300 0.000040 0.295 0.000037 0.064 0.000010
    10 0.163 0.000025 0.299 0.000042 0.285 0.000038 0.070 0.000012

    Step 4: Calculate the best estimate. Table 4 shows the results.

    Table 4 Proportion of population by PSTRE levels
    Table summary
    This table displays the results of Table 4 Proportion of population by PSTRE levels. The information is grouped by PSTRE level (appearing as row headers), Proportion of population (appearing as column headers).
    PSTRE level Proportion of population
    T.,below level 1 0.158
    T., level 1 0.303
    T., level 2 0.288
    T., level 3 0.068

    Step 5: Calculate the estimate of the variance of T.. Table 5 shows the results.

    Table 5 Variance estimates by PSTRE levels
    Table summary
    This table displays the results of Table 5 Variance estimates by PSTRE levels. The information is grouped by PSTRE level (appearing as row headers), Variance of T. and Standard deviation of T. (appearing as column headers).
    PSTRE level Variance of T. Standard deviation of T.
    T.,below level 1 0.000034 0.0058
    T., level 1 0.000065 0.0081
    T., level 2 0.000064 0.0080
    T., level 3 0.000024 0.0049

    Missing PSTRE proficiency values are directly observed—unlike the proficiency values which are indirectly observed. Thus, estimates related to the population without PSTRE proficiency can be estimated using only one set of plausible values instead of all ten sets—as there is no uncertainty related to the fact that the PSTRE proficiency value is not available. The proportion without PSTRE proficiency values is 0.183, and its standard error is 0.0054.

    Part II: Direct measures of skills in LISA versus PIAAC

    LISA included a subsample of PIAAC respondents. The PIAAC sample covered all provinces and territories in Canada; it included supplementary samples for youths, official language minorities, immigrants, Métis, and Aboriginals in select provinces. The final number of respondents obtained in PIAAC is 27,285.

    The subsample of PIAAC selected for LISA was selected using systematic sampling of the PIAAC sample excluding the supplementary samples and sampled households in the territories. PIAAC respondents also had to complete the questionnaire up to the point where an assessment score could be assigned to be considered a PIAAC respondent in LISA (see Box 1 for more details). The final number of PIAAC respondents in LISA with PIAAC assessment scores is 8,598. Countries involved in PIAAC were asked to have a minimum of 5,000 cases; thus, the LISA subsample is as large as the complete PIAAC sample of many other countries.

    The advantage of using LISA over PIAAC is the availability of additional variables and administrative data with which proficiency scores can be analyzed. However, LISA’s smaller sample size compared to PIAAC results in a higher variability in the estimate; and thus, also less power in statistical tests for significance. Literacy score estimates are provided below to give a flavour of the estimates from LISA compared to PIAAC. Numeracy and PSTRE estimates can be expected to behave in a similar fashion.

    Table 6 shows the estimates of mean literacy proficiency scores by age group using LISA compared to PIAAC data. The standard error is about 20% larger for the LISA estimates compared to the PIAAC estimates.Note 2 Nevertheless, the LISA literacy mean estimates are reliable and also match the relative rankings observed with the PIAAC estimates.

    Table 6 Mean literacy score by age, for LISA and PIAAC, for Canada excluding territories
    Table summary
    This table displays the results of Table 6 Mean literacy score by age. The information is grouped by Age group (appearing as row headers), Literacy, LISA, PIAAC, Average, Standard error and Coefficient of variation, calculated using % units of measure (appearing as column headers).
    Age group Literacy
    LISA PIAAC
    Average Standard error Coefficient of variation Average Standard error Coefficient of variation
      %   %
    16 to 24 years 273.1 1.9 0.7 275.9 1.3 0.5
    25 to 34 years 284.6 1.7 0.6 285.2 1.3 0.4
    35 to 44 years 278.5 1.7 0.6 279.7 1.4 0.5
    45 to 54 years 265.2 1.7 0.6 268.0 1.3 0.5
    55 to 65 years 259.5 1.5 0.6 260.4 1.1 0.4
    Total, 16 to 65 years 271.9 0.7 0.3 273.6 0.6 0.2

    Table 7 shows the distribution of literacy scores by level of proficiency by age group. As to be expected with a smaller sample size in LISA compared to PIAAC, the coefficients of variation of the estimates in LISA are higher and in some instances, unreliable. Nevertheless, the estimated proportion of people in most levels is reliable and the relative rankings between the levels matches what is observed in PIAAC.

    Table 7 Distribution of literacy score by age group and level, for LISA and PIAAC, for Canada excluding territories
    Table summary
    This table displays the results of Table 7 Distribution of literacy score by age group and level. The information is grouped by Age group (appearing as row headers), Literacy level, LISA, PIAAC, Proportion, Standard error and Coefficient of variation, calculated using % units of measure (appearing as column headers).
    Age group Literacy level LISA PIAAC
    Proportion Standard error Coefficient of variation Proportion Standard error Coefficient of variation
    %
    16 to 24 years Below level 1 3.1Note E: Use with caution 0.8 26.1 2.6Note E: Use with caution 0.5 19.0
    Level 1 12.3 1.4 11.3 10.7 0.9 8.4
    Level 2 33.4 2.1 6.4 32.9 1.7 5.1
    Level 3 40.1 2.0 4.9 41.9 1.5 3.6
    Level 4 10.7 1.4 13.4 11.4 1.0 8.7
    Level 5 Note F: too unreliable to be published Note F: too unreliable to be published 52.3 Note F: too unreliable to be published Note F: too unreliable to be published 40.6
    25 to 34 years Below level 1 2.3Note E: Use with caution 0.7 30.9 1.9Note E: Use with caution 0.4 20.5
    Level 1 9.6 1.2 12.3 9.2 0.9 9.6
    Level 2 27.7 1.8 6.4 28.7 1.7 5.8
    Level 3 40.8 2.0 4.9 40.1 2.1 5.1
    Level 4 18.2 1.5 8.2 18.3 1.4 7.5
    Level 5 Note F: too unreliable to be published Note F: too unreliable to be published 38.0 1.7Note E: Use with caution 0.5 27.1
    35 to 44 years Below level 1 3.0Note E: Use with caution 0.6 20.4 3.3 0.5 15.8
    Level 1 11.7 1.3 11.4 11.0 1.0 9.1
    Level 2 29.0 1.8 6.1 28.1 1.5 5.2
    Level 3 40.6 1.8 4.3 40.9 1.3 3.1
    Level 4 14.5 1.3 8.8 15.3 1.1 6.9
    Level 5 Note F: too unreliable to be published Note F: too unreliable to be published 37.5 1.4Note E: Use with caution 0.4 28.9
    45 to 54 years Below level 1 5.8 0.8 14.4 5.2 0.6 12.1
    Level 1 15.5 1.4 9.1 14.6 0.9 6.1
    Level 2 33.7 1.9 5.5 33.0 1.3 4.0
    Level 3 33.1 1.8 5.5 34.5 1.4 3.9
    Level 4 11.1 0.9 8.3 11.8 0.9 7.5
    Level 5 Note F: too unreliable to be published Note F: too unreliable to be published 42.6 0.8Note E: Use with caution 0.2 26.2
    55 to 65 years Below level 1 5.3 0.6 12.2 5.7 0.6 9.8
    Level 1 17.8 1.1 6.2 17.1 0.9 5.4
    Level 2 38.0 1.6 4.2 37.0 1.3 3.4
    Level 3 31.4 1.5 4.7 32.1 1.1 3.4
    Level 4 7.2 0.9 12.4 7.8 0.8 10.6
    Level 5 Note F: too unreliable to be published Note F: too unreliable to be published 58.7 Note F: too unreliable to be published Note F: too unreliable to be published 48.9
    Total, 16 to 65 years Below level 1 4.0 0.3 8.1 3.8 0.2 6.3
    Level 1 13.5 0.7 5.0 12.6 0.5 3.6
    Level 2 32.4 0.8 2.6 32.0 0.7 2.1
    Level 3 37.0 0.9 2.4 37.7 0.7 1.8
    Level 4 12.3 0.5 4.1 12.9 0.5 3.8
    Level 5 0.8Note E: Use with caution 0.2 18.5 1.0 0.1 14.8

    Table 8 shows results of a simple regression model of literacy scores with age groups. Literacy score is the dependent variable, and the independent variables are indicator variables for the age groups (with the age group 16-24 being the reference group). The regression formula is: y literacy scrore =a+ b 1  ×Ag e 25 34 + b 2  ×Ag e 3544 + b 3  ×Ag e 4554 + b 4  ×Ag e 55 or older MathType@MTEF@5@5@+= feaagKart1ev2aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr 4rNCHbGeaGqiVu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq=Jc9 vqaqpepm0xbba9pwe9Q8fs0=yqaqpepae9pg0FirpepeKkFr0xfr=x fr=xb9adbaqaaeGaciGaaiaabeqaamaabaabaaGcbaaeaaaaaaaaa8 qacaWG5bWdamaaBaaaleaapeGaamiBaiaadMgacaWG0bGaamyzaiaa dkhacaWGHbGaam4yaiaadMhacaGGGcGaam4CaiaadogacaWGYbGaam 4BaiaadkhacaWGLbaapaqabaGcpeGaeyypa0JaamyyaiabgUcaRiaa dkgapaWaaSbaaSqaa8qacaaIXaaapaqabaGcpeGaaiiOaiabgEna0k aadgeacaWGNbGaamyza8aadaWgaaWcbaWdbiaaikdacaaI1aGaeyOe I0IaaiiOaiaaiodacaaI0aaapaqabaGcpeGaey4kaSIaamOya8aada WgaaWcbaWdbiaaikdaa8aabeaak8qacaGGGcGaey41aqRaamyqaiaa dEgacaWGLbWdamaaBaaaleaapeGaaG4maiaaiwdacqGHsislcaaI0a GaaGinaaWdaeqaaOWdbiabgUcaRiaadkgapaWaaSbaaSqaa8qacaaI ZaaapaqabaGcpeGaaiiOaiabgEna0kaadgeacaWGNbGaamyza8aada WgaaWcbaWdbiaaisdacaaI1aGaeyOeI0IaaGynaiaaisdaa8aabeaa k8qacqGHRaWkcaWGIbWdamaaBaaaleaapeGaaGinaaWdaeqaaOWdbi aacckacqGHxdaTcaWGbbGaam4zaiaadwgapaWaaSbaaSqaa8qacaaI 1aGaaGynaiaacckacaWGVbGaamOCaiaacckacaWGVbGaamiBaiaads gacaWGLbGaamOCaaWdaeqaaaaa@83F4@

    The intercept reflects the mean score for the age group 16-24 and is the same value as observed in Table 6. The coefficient estimates are similar in both models. The standard errors of the estimates are higher with LISA than with PIAAC; nevertheless, both LISA- and PIAAC-based models are able to show statistical differences between the age groups.

    Table 8 Regression results of literacy on age groups, for LISA and PIAAC, for Canada excluding territories
    Table summary
    This table displays the results of Table 8 Regression results of literacy on age groups LISA and PIAAC (appearing as column headers).
      LISA PIAAC
    Estimate Variance Standard error t p-value Estimate Variance Standard error t p-value
    a: Intercept 273.1 3.7 1.9 142.2 <.0001 275.9 1.6 1.3 216.5 <.0001
    b1: Age(25 to 34 years) 11.5 6.7 2.6 4.4 <.0001 9.4 3.3 1.8 5.1 <.0001
    b2: Age(35 to 44 years) 5.4 7.0 2.7 2.0 0.0439 3.8 3.8 1.9 2.0 0.0514
    b3: Age(45 to 54 years) -8.0 7.2 2.7 -3.0 0.0030 -7.9 2.8 1.7 -4.7 <.0001
    b4: Age(55 to 65 years) -13.6 6.1 2.5 -5.5 <.0001 -15.5 2.7 1.7 -9.4 <.0001

    Box 1: Treatment of literacy-related non-response in LISA

    Literacy-related non-response in the PIAAC subsample selected for LISA is treated differently than in the full PIAAC sample. OECD (2013a, p. 69) explains literacy-related non-response as a result of adults being “unable to complete the background questionnaire as they were unable to speak or read the language of the assessment, had difficulty reading or writing, or had a learning or mental disability”.

    Literacy-related non-response records do not have assessment scores and have minimal background information. The group is also likely to have low levels of proficiency. As a result, PIAAC does not adjust for literacy-related non-response through weight adjustments as it does for other types of non-response. Instead, these records are kept on the dataset with non-zero weights and with missing values for the variables. This allows data users to measure the size of the literacy-related non-response and make adjustments as they see fit.

    LISA, on the other hand, treats literacy-related non-response in the same manner as other forms of non-response—that is, through weight adjustments. As a result, literacy-related non-respondents in the PIAAC subsample selected for LISA has zero-value PIAAC weights—as weights of other PIAAC respondents are adjusted to account for the non-response.

    The different treatment of literacy-related non-response in LISA was based on the following considerations:

    1. Only 0.9% of the PIAAC weighted population in the 10 provinces were literacy-related non-respondents; thus, limiting the differences resulting from different treatment of literacy-related non-response.
    2. Most literacy-related non-respondents were also non-respondents to the LISA questions. Assuming data users would always be using LISA variables in conjunction with assessment scores (otherwise, the full PIAAC dataset would be a better data source), missing values in the LISA variables limit a data user’s ability to make literacy-related non-response adjustments.
    3. Most data users would exclude the literacy-related non-response records from their analysis if they were in the dataset. This effectively treats the non-response as completely missing at random. Addressing the literacy-related non-response through weight adjustments results in comparatively less biased estimates as the relation to the weighting and calibration variables are taken into account.

    The impact of different treatments of literacy-related non-response on literacy and numeracy mean scores is demonstrated in OECD (2013a, p. 69). It indicates that complete exclusion of literacy-related non-respondents produces an upper bound of estimated proficiency scores for literacy and numeracy, while an assumed score of 85 points presents a reasonable lower bound. Using these two approaches, OECD (2013a) estimates, for Canada, the two treatments of literacy-related non-responses has a difference of 1.7 for the mean literacy score and 1.6 for the mean numeracy score.

    Conclusion

    This paper demonstrates the use of direct measures of literacy, numeracy, and PSTRE in LISA and provides a flavour of how it compares to estimates produced from the PIAAC dataset. We conclude that:

    1. Data exploration can be based on a single set of plausible values (with variance estimations using bootstrap weights). This will produce reliable point estimates, but will underestimate the standard error as the variability resulting from indirectly observing the proficiency scores is not included.
    2. The best and final estimates should be based on all ten sets of plausible values, following the directions in OECD(2013b).
    3. Analysis of proficiency scores using LISA should also make use of variables available in LISA but not in PIAAC; otherwise, the PIAAC dataset should be used instead, as the larger sample size allows for more precise estimates.
    4. LISA estimates of proficiency scores cannot be directly compared to published estimates from PIAAC due to different treatment of literacy-related non-response and different geographical coverage. However, reliable estimates can be produced from LISA; and, the relative rankings of proficiency estimates for subgroups and the conclusions that are drawn from them should be similar to what would be concluded from the full PIAAC dataset.

    References

    Organisation for Economic Co-operation and Development. (2013a). OECD Skills Outlook 2013: First Results from the Survey of Adult Skills.

    Organisation for Economic Co-operation and Development. (2013b). Technical Report of the Survey of Adult Skills (PIAAC).

    Statistics Canada. (2014). LISA, 2012: Quick Start User Guide. Ottawa, ON: Statistics Canada.

    Statistics Canada. (2013). User Guide: The Programme for International Assessment of Adult Competencies, 2012. Ottawa, ON: Statistics Canada.

    Wu, M.L., and Adams, R. J. (2002). Plausible Values: Why they are important. Paper presented at the International Objective Measurement workshop, New Orleans, LA.

    Annex A: Example of SAS Code used to obtain plausible values

    /***********************************************************************************************
    The following code uses BOOTVAR v3.2 to generate estimates for the mean literacy score and the standard error of the estimate for each of the 10 plausible value sets.
    ***********************************************************************************************/
    libname mylib "{directory}"; /*Sets personal directory*/

    /*Reading in LISA Wave 1 data*/
    %LET datafid = '{directory}\LISA_MASTER_2011.txt';
    data mylib.lisa_w1;
    %include '{directory}\LISA_MASTER_2011_i.SAS';
    %include '{directory}\LISA_MASTER_2011_lbe.SAS';
    if IRPW_FLG = 1; /*Keeping only ISA respondents*/
    keep PersonID IRPW PVLIT1-PVLIT10 dummy; /*Keeping only the necessary variables*/
    dummy=1; /*creating dummy variable for denominator in the ratio calculation of the average*/
    run;

    /*Getting PIAAC respondent person bootstrap weights (IRPW)*/
    %LET datafid = "{directory}\LISA_MASTER_2011_BOOTSTRAP_irpw_w1.txt";
    data mylib.lisa_bsw;
    %include "{directory}\LISA_MASTER_2011_BOOTSTRAP_irpw_w1_i.sas";
    run;
    /*Add final weight to bootstrap weight file because BOOTVAR requires it*/
    data mylib.lisa_bsw2;
    merge mylib.lisa_bsw (IN=a) mylib.lisa_w1(keep=personid irpw);
    by personid;
    if a;
    run;

    /************************************************************************************************
    ******************** START OF TEMPLATE PROVIDED BY BOOTVAR SOFTWARE ***************************** ************************************************************************************************/

    *************************************************************************
    ** SPECIFY THE NAME OF THE FOLLOWING 2 DIRECTORIES (directories only): **
    *************************************************************************;
    libname in1 "{directory}";  
    libname out "{directory}";  

    *************************************************************************************
    ** SPECIFY THE NAME OF THE ANALYSIS FILE (CREATED IN STEP 1) (without extension):  **  
    *************************************************************************************;
    %let Mfile = in1.lisa_w1;      

    *************************************************************************
    ** SPECIFY THE NAME OF THE FILE CONTAINING THE BOOTSTRAP WEIGHTS:      **
    *************************************************************************;
    libname in2 "{directory}";
    %let bsamp=in2.lisa_bsw2;

    *********************************************************************************************
    ** SPECIFY, IF DESIRED, THE BREAKDOWN VARIABLE(S) (EG: PROVINCE, SEX, ETC...):             **
    **     Write the name of the breakdown variable(s) below.                                  **
    **                                                                                         **
    **     -  If the analysis includes all of the data in the file created in step 1, put      **
    **        a dot. (%let classes =. )                                                        **
    **     -  If more than one variable, leave a blank between each variable                   **
    **        (%let classes=var1 var2)                                                         **
    **     -  DO NOT ERASE OR COMMENT OUT THIS COMMAND                                         **
    *********************************************************************************************;
    %let classes = .;

     

    *********************************************************************************************
    ** SPECIFY THE FOLLOWING INFORMATION (SPECIFIC TO THE SURVEY YOU ARE USING):               **
    **    You must specify:                                                                    **
    **           1- The unique identifier variable(s) (separated by a space)                   **
    **           2- The final weight (variable included in the bootstrap weight file)          **
    **           3- The prefix of the bootstraps weight variables                              **
    **           4- Parameter for the mean bootstrap (set to 1 for regular bootstrap)          **
    **           5- The number of bootstrap weights to use (note: For testing, B must be >= 2. **
    **              IT IS NECESSARY TO USE ALL THE BOOTSTRAP WEIGHTS WHEN PERFORMING THE FINAL **
    **              ANALYSIS.)                                                                 **
    **                                                                                         **
    **           - Refer to AppendixC  to obtain this information                              **
    *********************************************************************************************;
    %let ident = personid;     
    %let fwgt  = IRPW;     
    %let bsw   = BSIRPW;     
    %let R     = 1;
    %let B     = 1000; 

     

    *********************************************************************************
    ** SPECIFY THE DIRECTORY AND THE NAME OF THE FILE THAT CONTAINS THE MACROS     **
    ** (THE PROGRAM MACROE_V32.SAS IF NO MODIFICATIONS HAVE BEEN MADE BY THE USER) **
    *********************************************************************************;
    %include "{directory}\SASBootvar_v32_20110704\MACROE_V32.SAS";

    /*****************************************************************************/
    /***                               SECTION 2                               ***/
    /*****************************************************************************/
    /***                                                                       ***/
    /*** This section lets the user specify the different analyses of interest.***/
    /***                                                                       ***/
    /*****************************************************************************/

    * TO OBTAIN VARIANCE ESTIMATES OF A RATIO, RUN:
    ---------------------------------------------;
    %ratio(PVLIT1, dummy);    
    %ratio(PVLIT2, dummy);
    %ratio(PVLIT3, dummy);
    %ratio(PVLIT4, dummy);
    %ratio(PVLIT5, dummy);
    %ratio(PVLIT6, dummy);
    %ratio(PVLIT7, dummy);
    %ratio(PVLIT8, dummy);    
    %ratio(PVLIT9, dummy);
    %ratio(PVLIT10, dummy);

    %output;  /*Displays the results on the screen.  Do not modify. */

    /************************************************************************************************
    ******************** END OF TEMPLATE PROVIDED BY BOOTVAR SOFTWARE *******************************
    ************************************************************************************************/

    Notes

    Report a problem on this page

    Is something not working? Is there information outdated? Can't find what you're looking for?

    Please contact us and let us know how we can help you.

    Privacy notice

    Date modified: