Estimating Measurement Error in Annual Job Earnings: A Comparison of Survey and Administrative Data

Estimating Measurement Error in Annual Job Earnings: A Comparison of Survey and Administrative Data (DOI: https://doi.org/10.1162/REST_a_00352) was written by John M. Abowd and Martha H. Stinson. It was published in the Review of Economics and Statistics (2013). (My copy was actually published in the 2012 Social, Economic, and Housing Statistics Division (SEHSD) working papers series.)

Job-level annual earnings are collected from 5 SIPP panels and from the SSA Detailed Earnings Record (DER) (which itself is sourced from IRS W-2 forms). (The authors use confidential, unanonymized data.)

Errors are typically measured with reference to some external, true-er data source. This is a prior. The authors estimate a mixed effects model and observe changes in the levels of error as priors are varied. "The advantage of the MLMM framework is that it shows with full generality how to accommodate two or more matched observations of earnings on the same job, how to vary the prior assumptions about which measure is 'true' systematically, and how to use external audit information, if available, to update the posterior distribution over which value is 'true'."

Preface to the model

Building off of Fuller and Bound et al, the authors specify a measurement Y_t in terms of a true value y_t and uncorrelated measurement error u_t. The reliability ratio is characterized by:

When a variable is a function of the true value (like A_t = βy_t + ε_t) but is regressed on the measurement (like A_t = βY_t = βˆ(y_t + u_t)), then the estimated coefficient is biased to zero.

The model

There are I individuals, S sequential job spells, and T time periods. Job earnings are measured per individual i, per sequential job spell s, and per time period t: y_ist. For every level s by t, not every i has a measurement. All stacked up, there N measurements.

There are also J employers. All job earnings measurements (i.e., y_ist) have a corresponding employer j.

Fixed effects are estimated for a K parameters. The fixed effect measurements and fixed effect coefficients are stacked into x_ist and B, both vectors of size K

Random effects are estimated at individual- and employer-levels. Per individual i, there is a random effect θ_i. These are stacked into Θ, a vector of size I. Per employer j, there is a random effect ψ_j. These are stacked into Ψ, a vector of size J.

The model could be specified as:

y_ist = x_istB + d_iΘ + f_istΨ + η_ist

where:

• η_ist is the residual

• d_i (individual-level random effects design) is of size I

• f_ist (employer-level random effects design) is of size J

This model could then be estimated as:

Y = XB + ZU + H

• both Y (stacked y_ist) and H (stacked η_ist) are of size N

• X (stacked x_ist) is of shape N by K

• Z ≡ [D F]

  • D (stacked d_i) was of shape N by I

  • F (stacked f_ist) was of shape N by I

  • the combination of these two, Z, is of shape N by (I + J)

• U ≡ [Θ Ψ] (combined random effects) is of size (I + J)

But the intention of this model is to estimate the outcome from multiple sources of job earnings. The number of sources is Q.

The fixed effect coefficients are set in B of shape K by Q (per source).

Per individual i, there is a random effect vector θ_i of size Q (per source). These are stacked into Θ, a matrix of shape I by Q. Per employer j, there is a random effect vector ψ_j of size Q (per source). These are stacked into Ψ, a matrix of shape J by Q.

The model is now specified as:

y_ist = x_istB + d_iΘ + f_istΨ + η_ist

where:

• η_ist is the residual vector of size Q (per source)

And the model is now estimated as:

Y = XB + ZU + H

where:

• both Y (stacked y_ist) and H (stacked η_ist) are of shape N by Q

• U ≡ [Θ^T Ψ^T]^T (combined random effects) is of shape (I + J) by Q

  • the repeated transpositions just massage the combination into the correct shape

The model is fit using restricted maximum likelihood (REML) (via ASReml).

Varying priors of the model

The outcome estimations are combined with a positive orthogonal weight vector ω; each ω_q is between 0 and 1 and they sum to 1.

The signal of the model is calculated as Sig(y_ist) = ωy_ist and the measurement error is the deviation of outcome estimations from the signal as ME(y_ist) = y_ist - Sig(y_ist).

As an example, ω = [0 1] effectively is the statement that the second source is certain truth. This is how most measurement error evaluation studies are formalized: the secondary data source is the external, true-er one. The authors proceed with estimating the signal and measurement error using ω = [0.5 0.5] and ω = [0.1 0.9].

The authors note that a full Bayesian approach would be to model the priors per measurement, as ω_ist, using some other predictors (e.g., data quality measures g_ist). The model would then update the priors into a posterior.

Authors find that "reliability statistics for SIPP and DER earnings measures were quite comparable except for the sub-sample of SIPP person-jobs where at least one year of SIPP earnings contained a Census Bureau imputation". Also very little meaningful measurement error in fixed effects. The administrative data had greater variance, even when conditioned on the predictors.

CategoryRicottone