​​​​​​​NAEP Technical DocumentationNAEP Variance Estimation During Transition to Digitally Based Assessment

Suppose common population linking is used to align the digitally based assessment (DBA) scores from future reporting samples to the existing trend scale used in reporting the paper-based assessment (PBA) results (referred to as trend PBA). In NAEP, two randomly equivalent samples from the same population, one administered with DBA, while the other one with PBA (referred to as bridge PBA), are used in estimating the common population linking function. An approach is described below for estimating error variance of the group statistics for the DBA assessments. The statistic of interest, t hat, can be a mean, a percentage, an achievement-level percentage, etc. 

Procedures for estimating error variance for DBA group scale score statistics, when comparing groups between DBA and trend PBA—External Linking 

Suppose that the common population linking functions are estimated using two randomly equivalent samples—the DBA sample and the bridge PBA sample, and then applied to link the results of a third independent DBA sample to the existing trend scale in reporting the trend PBA results. This is referred to as external linking. 

Suppose the DBA results are linked to the trend scale in reporting PBA via external linking. When comparing the DBA result, t hat sub D B A, to the trend PBA result, t hat sub P B A, where PBA and DBA are two independent samples, the error variance of the absolute value of, t hat sub D B A, end sub, minus t hat sub P B A, end absolute value is

the variance hat of, t hat sub P B A, plus the variance hat of, t hat sub D B A . (1) 

 The variance hat of, t hat sub P B A is estimated as

the variance hat of, t hat sub P B A, equals the sampling variance hat of, t hat sub P B A, , plus the measurement variance hat of, t hat sub P B A,, (2) 

where the sampling variance hat of, t hat sub P B A and the measurement variance hat of, t hat sub P B A are the sampling and measurement variances for t hat sub P B A. 

the variance hat of, t hat sub D B A is estimated as

the variance hat of, t hat sub D B A, equals the sampling variance hat of, t hat sub D B A, plus the measurement variance hat of, t hat sub D B A, plus the linking error variance hat of, t hat sub D B A, (3) 

where the sampling variance hat of, t hat sub D B A and the measurement variance hat of, t hat sub D B A are the sampling and measurement variances for t hat sub D B A. The term the linking error variance hat of, t hat sub D B A ] is the linking error variance associated with the uncertainty in estimating the common population linking function. When external linking is used, the linking error variance hat of, t hat sub D B A ] is estimated using a quadratic function which describes the relationship between the linking variance and t hat sub D B A. Specifically, the quadratic function for any subgroup has the following general form:

the linking error variance hat of, t hat sub D B A, equals a, hat times t hat squared sub D B A, end sub, plus b hat times t hat sub D B A, end sub, plus c hat, (4) 

where a, hat, b hat, andc hat are coefficients estimated through a Monte Carlo based method. See Mazzeo, Donoghue, Liu, and Xu (2018) for more discussion of the method of calculating the linking variance for external linking. 

The approach described here is used in estimating the error variance of the statistics from Mathematics and Reading DBA at grades 4 and 8, when comparing the group scale score statistics from the DBA in 2017 and future years to those from PBA in 2015 and earlier years. Tables that provide the quadratic function coefficients for means and percentages, standard deviations, and achievement-level percentages can be accessed via links in the table below. 

 The linking error component derived from external linking does not apply when

Procedures for estimating error variance for group scale score statistics from combined PBA/DBA sample—Internal Linking

Suppose that the common population linking functions are estimated using two randomly equivalent samples—the DBA sample and the bridge PBA sample, and then applied to link the results of the same DBA sample used in deriving the linking function to the existing trend scale in reporting the trend PBA results from previous years. This is referred to as internal linking. In addition, suppose that the DBA and bridge PBA samples used in deriving the linking function are combined in estimating group statistics of interest, t hat sub combine.

  1.  For the mth set of plausible values (PVs), calculate t hat sub combine comma m, where m equals 1, 2, dot dot dot, 20

  2.  The estimate of the statistic of interest is computed as t hat sub combine, equals the fraction 1 over 20, end fraction, times the sum from m equals 1 to 20 of, t hat sub combine comma m.

  3.  Under internal linking, the error variance of the variance hat of t hat sub combine is estimated as

the variance hat of t hat sub combine, equals the variance hat sub sampling given the linking, end sub, of t hat sub combine, end sub, plus the variance hat sub measurement given the linking, end sub, of t hat sub combine,   (5)

 where 

  •  the variance hat sub sampling given the linking, end sub, of t hat sub combine is the sampling variance, considering the uncertainty due to sampling in deriving the linking functions as well as in the estimation of the group statistics;
  •  the variance hat sub measurement given the linking, end sub, of t hat sub combine is the measurement variance, considering the uncertainty due to latency in deriving the linking functions as well as in estimation of the group statistics.

Estimating error variance of group scale score statistics on subscale, combined PBA/DBA sample and internal linking

The following steps can be taken to estimate the error variance of statistics of interest t hat superscript s, sub combine using the combined PBA/DBA sample, for a subscale s.

To estimate the sampling error variance, the variance hat sub sampling given the linking, end sub, of t hat superscript s, sub combine, a total of 62 pairs of transformation coefficients The pair A superscript s, sub i, comma, b superscript s, sub i, where i equals 1, 2, dot dot dot, 62 are used. These transformation coefficients are derived from linking the DBA mean and standard deviation to the PBA mean and standard deviation for each replicate i. Also, the first set of PVs of the combined sample The vector with component Y superscript s, sub 1 and component X superscript s, sub 1 are used. Here, Y superscript s, sub 1 is the DBA PVs, and X superscript s, sub 1 is the bridge PBA PVs.

  1.  For each pair of transformation coefficients The pair A superscript s, sub i, comma, b superscript s, sub i, where i equals 1, 2, dot dot dot, 62, do the following:

    1.  Apply The pair A superscript s, sub i , comma, b superscript s, sub i to transform Y superscript s, sub 1, in the following way: 

                                                                               Y tilde superscript s, sub i, equals A, superscript s, sub i, times Y superscript s, sub 1, plus B superscript s, sub i .      (6)

    2.  Combine the transformed sets of DBA PVs, Y tilde superscript s, sub i, where i equals 1, 2, dot dot dot, 62 with the bridge PBA-part of the PVs, X superscript s, sub 1 to get the combined PV:
                                                                                      Z superscript s, sub i equals the vector with component Y tilde superscript s, sub i and component X superscript s, sub 1.

    3.  Calculate the statistic of interest, t hat superscript s, sub combine comma i based on Z superscript s, sub i and W superscript c, sub i where W superscript c, sub i denote the ith replicate weight for the combined sample.

  2.  the variance hat sub sampling given the linking, end sub, of t hat superscript s, sub combine is then calculated as 

                                               the variance hat sub sampling given the linking, end sub, of t hat superscript s, sub combine, equals, the sum from i equals 1 to 62 of, open parenthesis t hat superscript s, sub combine comma i, end sub, minus, t hat bar superscript s, sub combine close parenthesis, squared      (7)

     where t hat bar superscript s, sub combine, equals the fraction 1 over 62, end fraction, times the sum from i equals 1 to 62 of, t hat superscript s, sub combine comma i . 
 
To calculate the measurement error variance  the variance hat sub measurement given the linking, end sub, of t hat superscript s, sub combine, 100 pairs of one set of PBA PVs and one set of DBA PVs are randomly chosen. This leads to a total of 100 pairs of transformation coefficients, which are grouped into 5 replications with 20 pairs of coefficients within each replication:

The pair A superscript s, sub j given k, end sub, comma, B superscript s, sub j given k, end sub, where j equals 1, 2, dot dot dot, 20 and k equals 1, 2, dot dot dot, 5 .

  1. In each replication, transformation coefficients are calculated for each of the 20 pairs of a set of DBA PVs and a set of PBA PVs. Hence, for the kth replication, k equals 1, 2, dot dot dot, 5

    1. Apply the transformation coefficients to transform the DBA PVs in the following way:
                                                                     Y tilde superscript s, sub j given k, end sub, equals, A superscript s, sub j given k, end sub, times, Y superscript s, sub j, plus, B superscript s, sub j given k, end sub, where j equals 1, 2, dot dot dot, 20

    2. For each set of  Y tilde superscript s, sub j given k,, combine them with the bridge PBA PVs as
                                                                            Z superscript s, sub j given k, end sub, equals the vector with component Y tilde superscript s, sub j given k, end sub and component X superscript s, sub j given k, end sub, where j equals 1, 2, dot dot dot, 20

      For a given k, the 20 tuple, X superscript s, sub 1 given k, end sub, comma, X superscript s, sub 2 given k, end sub, comma, dot dot dot, X superscript s, sub 20 given k is a random permutation of the 20 sets of bridge PBA PVs,
      The 20 tuple, X superscript s, sub 1, comma, X superscript s, sub 2, comma, dot dot dot, X superscript s, sub 20 which is used in deriving the transformation coefficients The pair A superscript s, sub j given k, end sub, comma, B superscript s, sub j given k.


    3. the variance hat superscript k, sub measurement given the linking, end sub of t hat superscript s, sub combine using the kth replication of coefficients is calculated as

                                   the variance hat superscript k, sub measurement given the linking, end sub of t hat superscript s, sub combine, equals, open parenthesis, 1 plus the fraction 1 over 20, close parenthesis, times, the fraction with numerator the sum from j equals 1 to 20 of, open parenthesis, t hat superscript s, sub combine comma j given k, end sub, minus t hat bar superscript s, sub combine given k, end sub, close parenthesis, squared, and denominator 20 minus 1, end fraction              (8)

      where t hat superscript s, sub combine comma j given k, end sub is calculated based on Z superscript s, sub j given k, weighted by the student sampling weight. And t hat bar superscript s, sub combine given k, end sub, equals, the fraction 1 over 20, end fraction, times the sum from j equals 1 to 20 of, t hat superscript s, sub combine comma j given k.


  2. the variance hat sub measurement given the linking, end sub, of t hat superscript s, sub combine is then calculated as:
                                                 the variance hat sub measurement given the linking, end sub, of t hat superscript s, sub combine, equals, one fifth times the sum from k equas 1 to 5 of, the variance hat superscript k, sub measurement given the linking, end sub, of t hat superscript s, sub combine

Note that for NAEP subjects that report on a univariate scale only (e.g., 2018 Civics), the error variance estimation for group statistics on the overall scale based on the combined PBA/DBA sample follows the procedure described for subscale.

Estimating error variance of group scale score statistics on composite scale, combined sample and internal linking

To calculate the sampling variance of statistics on composite scale, assume there are a total of S subscales and the subscale weights areBeta sub 1, beta sub 2, dot dot dot, beta sub uppercase S where 0 is less than beta sub lowercase s, which is less than 1, where lowercase s equals 1, 2, dot dot dot, uppercase S.

  1. First calculate the DBA composite PVs in the following way: 

                                                                              Y tilde sub i equals the sum, from lowercase s equals 1 to uppercase S of, beta sub lowercase s times Y tilde superscript lowercase s, sub i, where i equals 1, 2, dot dot dot, 62.

  2. Let 
                                                                                 Z sub i equals the vector with component Y tilde sub i and component X sub 1, where i equals 1, 2, dot dot dot, 62

    be the ith set of PVs on composite scale where X sub 1 is the first set of composite scale PVs for bridge PBA. 


  3. For statistic t hat sub combine on composite scale, the variance hat of sub sampling given the linking, end sub, of t hat sub combine is then calculated as

                                      the variance hat sub sampling given the linking, end sub, of t hat sub combine, equals, the sum from t equals 1 to 62 of, open parenthesis t hat sub combine comma i, end sub, minus, t hat bar sub combine, close parenthesis squared      (9)

    where t hat sub combine comma i is calculated based on Z sub I and W superscript c, sub i, where i equals 1, 2, dot dot dot, 62 and t hat bar sub combine, equals the fraction 1 over 62, end fraction, times the sum from i equals 1 to 62 of, t hat sub combine comma i.

To calculate the measurement error variance of the group scale score statistics on composite scale,

  1. Calculate the DBA and bridge PBA composite PVs in the following way: 
                                                            Y tilde sub i given k, end sub, equals the sum from lowercase s equals 1 to uppercase S of, beta sub lowercase s, end sub, times Y tilde superscript lowercase s, sub j given k, end sub, where j equals 1, 2, dot dot dot, 20
    and 
                                                            X sub j given k, end sub, equals the sum from lowercase s equals 1 to uppercase S of, beta sub lowercase s, end sub, times X superscript lowercase s, sub j given k, end sub, where j equals 1, 2, dot dot dot, 20.

  2.  Let 
                                                                  Z sub j given k, end sub, equals the vector with component Y tilde sub j given k, end sub, and component X sub j given k, end sub, where j equals 1, 2, dot dot dot, 62

    be the jth set of PVs within replication k on composite scale; where X sub j given k is the jth set of composite scale PV for bridge PBA within replication k.

  3.  the variance hat sub measurement given the linking, end sub, of t hat sub combine is calculated as 

                                the variance hat sub measurement given the linking, end sub, of t hat sub combine, equals, one fifth times the sum from k equals to one to five of, open bracket, open parenthesis, One plus the fraction 1 over 20, close parenthesis Times the fraction with numerator, the sum from j equals one to twenty of, open parenthesis, t hat sub combine comma j given k, end sub, minus, t hat bar sub combine given k, end sub, close parenthesis, squared, and denominator twenty minus one, end fraction, close bracket       (10)

    where t hat sub combine comma j given k is calculated based on  Z sub j given k, weighted by the student sampling weight, and t hat bar sub combine given k, end sub, equals, the fraction 1 over 20, end fraction, times the sum from j equals 1 to 20 of, t hat sub combine comma j given k.
        
The approaches described here are used in estimating the error variance of the statistics for the 2018 Civics, Geography, and U.S. History assessments, when 
1) estimating error variance for the combined PBA/DBA sample group scale scores statistics themselves within 2018; or
2) comparing the group scale score statistics from the combined PBA/DBA sample in 2018 to those from trend PBA in 2014 and earlier                                   years. 

Tables that provide the transformation coefficients which are needed for calculating the sampling variances can be accessed via links in the table below. 

Also included are tables that provide the transformation coefficients as well as the index of the set of PVs (out of the 20 sets of PBA PVs and 20 sets of DBA PVs) that are used for calculating the measurement variances. For example, (k=2, DBA PV=1, Bridge PBA PV=14) means for the 2nd replication, the first set of PVs from DBA and 14th set of PVs from PBA bridge were used in computing the transformation coefficients. 

When comparing the statistics estimated from the combined PBA/DBA sample, t hat sub combine, to the one estimated from trend PBA in previous years, t hat sub P B A, where the trend PBA sample and the combined sample are independent, the error variance of the absolute value of t hat sub combine, end sub, minus t hat sub P B A, end absolute value is 

                                                                             the variance hat of t hat sub P B A, plus, the variance hat of t hat sub combine.      (11)

 the variance hat of t hat sub P B A is estimated as

                                                            the variance hat of t hat sub PBA equals, the variance hat sub sampling of t hat sub P B A, plus, the variance hat sub measurement of t hat sub P B A,       (12)

where the variance hat sub sampling of t hat sub P B A and the variance hat sub measurement of t hat sub P B A are the sampling and measurement variances for t hat sub P B A.


Links to coefficients used for variance calculation, by subject: 2017, 2018, and 2019
YearSubject area
2019 Mathematics
Reading
Science
2018 Civics
Geography
U.S. history
2017 Mathematics
Reading
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2017, 2018, and 2019 Assessments.




Last updated 03 January 2024 (PG)