Survey Weights

Survey weights account for the survey design, sampling error, and non-sampling error.

Description

Survey data is collected through a mechanism which can be specified statistically. If it is not specified, bias can be introduced and estimates can be over-confident.

Inverse variance weights are related, but not the same.

Survey weights begin with design weights reflecting probability of selection. Generally this is simply the inverse of the sampling probability: n_k/N for all strata k.

All real surveys feature non-sampling error, especially nonresponse. If nonresponse is uncorrelated with key metrics, it is negligible. Otherwise there is potential for nonresponse bias. This bias can be corrected through survey weights in a few ways:

• inverse propensity adjustments

• weighting class adjustments

Modeling on insignificant or uncorrelated attributes does not introduce bias, but it does inflate variance.

Calibration can be used to:

• make estimates be consistent with known true population proportions

• correct sampling error like undercoverage or overcoverage

• further correct for non-sampling error like nonresponse bias

The methods here include:

• raking
• iterative proportional fitting
• RIM weighting

• GREG estimators

Weighted Estimators

Survey weights w are designed such that a population proportion μ can be calculated using the weighted estimator Σ(wx) / Σw.

In the case that all cases have equal weight, it is straightforward to show that the variance of that estimator is w²σ².

In any other case, the variance is given by Σ(w²σ²) / (Σw)². This ratio must then be linearized or simulated to arrive at an approximate variance. Taylor expansion is a common strategy for linearization.

Reading Notes

• The Effect of Weight Trimming on Nonlinear Survey Estimates, Frank J. Potter, 1993

• Sampling Weights and Regression Analysis, Christopher Winship and Larry Radbill, 1994

• Improving on Probability Weighting for Household Size, Andrew Gelman and Thomas C. Little, 1998

• Using Calibration Weighting to Adjust for Nonresponse and Coverage Errors, Phillip S. Kott, 2006

• Struggles with Survey Weighting and Regression Modeling, Andrew Gelman, 2007

• The calibration approach in survey theory and practice, Carl-Erik Särndal, 2007

• A single frame multiplicity estimator for multiple frame surveys, Fulvia Mecatti, 2007

• Practical Considerations in Raking Survey Data; Michael P Battaglia, David C Hoaglin, and Martin R Frankel (and sometimes David Izrael); 2009

• Statistical Paradises and Paradoxes in Big Data, Xiao-Li Meng, 2018

• A New Paradigm for Polling, Michael A. Bailey, 2023

• The “Law of Large Populations” Does Not Herald a Paradigm Shift in Survey Sampling, Roderick J. Little, 2023

• Surveys of Consumers Technical Report: Technical Documentation for the 2024 Methodological Transition to Web Surveys, 2024

• The effect of online interviews on the University of Michigan Survey of Consumer Sentiment, Ryan Cummings and Ernie Tedeschi, 2024

CategoryRicottone