|
Size: 3694
Comment: Relink
|
Size: 6576
Comment: Link
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 3: | Line 3: |
| '''Survey weights''' account for the [[Statistics/SurveySampling|design of a survey sample]] and [[Statistics/SurveyInference#Non-sampling_Error|non-sampling error]]. | '''Survey weights''' account for the [[Statistics/SurveySampling|survey design]], [[Statistics/SurveyInference#Sampling_Error|sampling error]], and [[Statistics/SurveyInference#Non-sampling_Error|non-sampling error]]. |
| Line 13: | Line 13: |
| The design weight, or base weight, reflects unequal [[Statistics/SurveySampling|probabilities of selection]]. Generally this is simply the inverse of the sampling probability: ''n,,k,,/N'' for all strata ''k''. | Survey data is collected through a mechanism which can be specified statistically. If it is not specified, bias can be introduced and [[Analysis/Estimation|estimates]] can be over-confident. |
| Line 15: | Line 15: |
| [[Statistics/InverseVarianceWeights|Inverse variance weights]] are related, but not the same. | |
| Line 16: | Line 17: |
| Survey weights begin with [[Statistics/DesignWeights|design weights]] reflecting [[Statistics/SurveySampling|probability of selection]]. Generally this is simply the inverse of the sampling probability: ''n,,k,,/N'' for all strata ''k''. | |
| Line 17: | Line 19: |
| === Non-Response Adjustments === | All real surveys feature [[Statistics/SurveyInference#Non-sampling_Error|non-sampling error]], especially [[Statistics/SurveyNonresponse|nonresponse]]. If nonresponse is uncorrelated with key metrics, it is negligible. Otherwise there is potential for [[Statistics/NonresponseBias|nonresponse bias]]. This bias can be corrected through survey weights in a few ways: * [[Statistics/InverseProbabilityWeights|inverse propensity adjustments]] * [[Statistics/WeightingClassAdjustment|weighting class adjustments]] |
| Line 19: | Line 23: |
| All real surveys feature [[Statistics/SurveyInference#Non-sampling_Error|non-sampling error]], especially non-response. If non-response is uncorrelated with key metrics, it is negligible. There almost always is some observable [[Statistics/NonresponseBias|nonresponse bias]], i.e. an attribute that is known for the entire population and is correlated with both a key metric and responsivity. This bias can be corrected with a '''non-response adjustment''' to the survey weights. | Modeling on insignificant or uncorrelated attributes does not introduce bias, but it does inflate [[Analysis/Variance|variance]]. |
| Line 21: | Line 25: |
| It is also reasonable to expect that there is ''unobserved'' bias, i.e. an attribute that is not known. | [[Statistics/Calibration|Calibration]] can be used to: * make estimates be consistent with known true population proportions * correct [[Statistics/SurveyInference#Sampling_Error|sampling error]] like undercoverage or overcoverage * further correct for non-sampling error like nonresponse bias |
| Line 23: | Line 30: |
| A non-response adjustment factor generally moves weight from non-respondents to comparable respondents. If there are no significant attributes that can be used to establish comparability, then the adjustment is a flat multiplier: the total of cases over the count of respondents. (Non-respondents have their weight set to 0.) If there are significant attributes, responsivity can be modeled. There are generally two approaches: * '''weighting class adjustment''': The population (or stratum subpopulation) is partitioned into N-tiles according to the predicted responsivity. Each N-tile then receives a separate flat multiplier as described above. * '''propensity score adjustment''': Every respondent's weight is multiplied by the inverse of the predicted responsivity, while non-respondents have their weight set to 0. General practice is then to re-normalize the weights such that they sum to the same total as before applying the adjustment. Modeling on insignificant or uncorrelated attributes does not introduce bias, but it does inflate [[Statistics/Variance|variance]]. === Post-Stratification === Post-stratification is employed in survey weighting for several reasons: * There may be measurable [[Statistics/SurveyInference#Sampling_Error|sampling errors]], such as undercoverage, which can be corrected. * Incorporating auxiliary information, i.e. the known distribution of the population, into survey estimates should increase accuracy. * Post-stratified estimates are consistent. Estimates across surveys will match on e.g. the proportion of women in the population if they are all post-stratified according to the same targets. There are two approaches to this post-stratification: [[TheCalibrationApproachInSurveyTheoryAndPractice|GREG estimation and calibration estimation]]. Calibration is known under a variety of other names: '''raking''', '''iterative proportional fitting''', and '''RIM weighting'''. |
The methods here include: * raking * iterative proportional fitting * RIM weighting * [[Statistics/GeneralizedRegressionEstimator|GREG estimators]] |
| Line 46: | Line 40: |
| == Usage == | == Weighted Estimators == Survey weights ''w'' are designed such that a population proportion ''μ'' can be calculated using the weighted estimator ''Σ(wx) / Σw''. In the case that all cases have equal weight, the [[Analysis/Variance|variance]] of that estimator is ''w^2^σ^2^''. In any other case, the variance is given by ''Σ(w^2^σ^2^) / (Σw)^2^''. This ratio must then be linearized or simulated to arrive at an approximate variance. [[Calculus/TaylorSeries|Taylor expansion]] is a common strategy for linearization. ---- |
| Line 50: | Line 52: |
| === Weighted Estimators === | == Reading Notes == |
| Line 52: | Line 54: |
| Survey weights ''w'' are designed such that a population proportion ''μ'' can be calculated using the weighted estimator ''Σ(wx) / Σw''. In the case that all cases have equal weight, [[Statistics/Moments#Description|it is straightforward to show]] that the variance of that estimator is ''w^2^σ^2^''. In any other case, the variance is given by ''Σ(w^2^σ^2^) / (Σw)^2^''. This ratio must then be linearized or simulated to arrive at an approximate variance. [[Calculus/TaylorSeries|Taylor expansion]] is a common strategy for linearization. |
* [[AGeneralizationOfSamplingWithoutReplacementFromAFiniteUniverse|A Generalization of Sampling Without Replacement from a Finite Universe]], D. G. Horvitz and D. J. Thompson, 1952 * [[CalibrationEstimatorsInSurveySampling|Calibration estimators in survey sampling]], Jean-Claude Deville and Carl-Erik Särndal, 1992 * [[TheEffectOfWeightTrimmingOnNonlinearSurveyEstimates|The Effect of Weight Trimming on Nonlinear Survey Estimates]], Frank J. Potter, 1993 * [[PostStratificationAModelersPerspective|Post-Stratification: A Modeler's Perspective]], Roderick J.A. Little, 1993 * [[SamplingWeightsAndRegressionAnalysis|Sampling Weights and Regression Analysis]], Christopher Winship and Larry Radbill, 1994 * [[ImprovingOnProbabilityWeightingForHouseholdSize|Improving on Probability Weighting for Household Size]], Andrew Gelman and Thomas C. Little, 1998 * [[RandomEffectsModelsForSmoothingPoststratificationWeights|Random-effects Models for Smoothing Poststratitication Weights]], Laura C. Lazzeroni and Roderick J.A. Little, 1998 * [[TheGeneralizedExponentialModelForSamplingWeightCalibrationForExtremeValuesNonresponseAndPostStratification|The generalized exponential model for sampling weight calibration for extreme values, nonresponse, and poststratification]], R.E. Folsom and A.C. Singh, 2000 * [[UsingCalibrationWeightingToAdjustForNonresponseAndCoverageErrors|Using Calibration Weighting to Adjust for Nonresponse and Coverage Errors]], Phillip S. Kott, 2006 * [[EstimationInMultipleFrameSurveys|Estimation in Multiple-Frame Surveys]], Sharon Lohr and J. N. K. Rao, 2006 * [[StrugglesWithSurveyWeightingAndRegressionModeling|Struggles with Survey Weighting and Regression Modeling]], Andrew Gelman, 2007 * [[TheCalibrationApproachInSurveyTheoryAndPractice|The calibration approach in survey theory and practice]], Carl-Erik Särndal, 2007 * [[ASingleFrameMultiplicityEstimatorForMultipleFrameSurveys|A single frame multiplicity estimator for multiple frame surveys]], Fulvia Mecatti, 2007 * [[WeightTrimmingInTheNationalImmunizationSurvey|Weight Trimming in the National Immunization Survey]]; Sadeq Chowdhury, Meena Khare, and Kirk Wolter; 2007 * [[PracticalConsiderationsInRakingSurveyData|Practical Considerations in Raking Survey Data]]; Michael P Battaglia, David C Hoaglin, and Martin R Frankel (and sometimes David Izrael); 2009 * [[MethodsForAdjustingSurveyWeightsWhenEstimatingATotal|Methods for Adjusting Survey Weights When Estimating a Total]], Kimberly Henry and Richard Valliant; 2012 * [[StatisticalParadisesAndParadoxesInBigData|Statistical Paradises and Paradoxes in Big Data]], Xiao-Li Meng, 2018 * [[ANewParadigmForPolling|A New Paradigm for Polling]], Michael A. Bailey, 2023 * [[TheLawOfLargePopulationsDoesNotHeraldAParadigmShiftInSurveySampling|The “Law of Large Populations” Does Not Herald a Paradigm Shift in Survey Sampling]], Roderick J. Little, 2023 * [[SurveysOfConsumersTechnicalReport|Surveys of Consumers Technical Report: Technical Documentation for the 2024 Methodological Transition to Web Surveys]], 2024 * [[TheEffectOfOnlineInterviewsOnTheUniversityOfMichiganSurveyOfConsumerSentiment|The effect of online interviews on the University of Michigan Survey of Consumer Sentiment]], Ryan Cummings and Ernie Tedeschi, 2024 * [[LeveragingPredictiveModellingFromMultipleSourcesOfBigDataToImproveSampleEfficiencyAndReduceSurveyNonresponseError|Leveraging Predictive Modelling from Multiple Sources of Big Data to Improve Sample Efficiency and Reduce Survey Nonresponse Error]]; David Dutwin, Patrick Coyle, Joshua Lerner, Ipek Bilgen, and Ned English; 2024 * [[EnergyBalancingOfCovariateDistributions|Energy balancing of covariate distributions]], Jared D. Huling and Simon Mak, 2024 * [[Unifying Small Area Estimators Based on Area-Level and Unit-Level Models Through Calibration|UnifyingSmallAreaEstimatorsBasedOnAreaLevelAndUnitLevelModelsThroughCalibration]]; William Acero, Isabel Molina, and Juan Miguel Marín; 2026 |
Survey Weights
Survey weights account for the survey design, sampling error, and non-sampling error.
Description
Survey data is collected through a mechanism which can be specified statistically. If it is not specified, bias can be introduced and estimates can be over-confident.
Inverse variance weights are related, but not the same.
Survey weights begin with design weights reflecting probability of selection. Generally this is simply the inverse of the sampling probability: nk/N for all strata k.
All real surveys feature non-sampling error, especially nonresponse. If nonresponse is uncorrelated with key metrics, it is negligible. Otherwise there is potential for nonresponse bias. This bias can be corrected through survey weights in a few ways:
Modeling on insignificant or uncorrelated attributes does not introduce bias, but it does inflate variance.
Calibration can be used to:
- make estimates be consistent with known true population proportions
correct sampling error like undercoverage or overcoverage
- further correct for non-sampling error like nonresponse bias
The methods here include:
- raking
- iterative proportional fitting
- RIM weighting
Weighted Estimators
Survey weights w are designed such that a population proportion μ can be calculated using the weighted estimator Σ(wx) / Σw.
In the case that all cases have equal weight, the variance of that estimator is w2σ2.
In any other case, the variance is given by Σ(w2σ2) / (Σw)2. This ratio must then be linearized or simulated to arrive at an approximate variance. Taylor expansion is a common strategy for linearization.
Reading Notes
A Generalization of Sampling Without Replacement from a Finite Universe, D. G. Horvitz and D. J. Thompson, 1952
Calibration estimators in survey sampling, Jean-Claude Deville and Carl-Erik Särndal, 1992
The Effect of Weight Trimming on Nonlinear Survey Estimates, Frank J. Potter, 1993
Post-Stratification: A Modeler's Perspective, Roderick J.A. Little, 1993
Sampling Weights and Regression Analysis, Christopher Winship and Larry Radbill, 1994
Improving on Probability Weighting for Household Size, Andrew Gelman and Thomas C. Little, 1998
Random-effects Models for Smoothing Poststratitication Weights, Laura C. Lazzeroni and Roderick J.A. Little, 1998
The generalized exponential model for sampling weight calibration for extreme values, nonresponse, and poststratification, R.E. Folsom and A.C. Singh, 2000
Using Calibration Weighting to Adjust for Nonresponse and Coverage Errors, Phillip S. Kott, 2006
Estimation in Multiple-Frame Surveys, Sharon Lohr and J. N. K. Rao, 2006
Struggles with Survey Weighting and Regression Modeling, Andrew Gelman, 2007
The calibration approach in survey theory and practice, Carl-Erik Särndal, 2007
A single frame multiplicity estimator for multiple frame surveys, Fulvia Mecatti, 2007
Weight Trimming in the National Immunization Survey; Sadeq Chowdhury, Meena Khare, and Kirk Wolter; 2007
Practical Considerations in Raking Survey Data; Michael P Battaglia, David C Hoaglin, and Martin R Frankel (and sometimes David Izrael); 2009
Methods for Adjusting Survey Weights When Estimating a Total, Kimberly Henry and Richard Valliant; 2012
Statistical Paradises and Paradoxes in Big Data, Xiao-Li Meng, 2018
A New Paradigm for Polling, Michael A. Bailey, 2023
The “Law of Large Populations” Does Not Herald a Paradigm Shift in Survey Sampling, Roderick J. Little, 2023
The effect of online interviews on the University of Michigan Survey of Consumer Sentiment, Ryan Cummings and Ernie Tedeschi, 2024
Leveraging Predictive Modelling from Multiple Sources of Big Data to Improve Sample Efficiency and Reduce Survey Nonresponse Error; David Dutwin, Patrick Coyle, Joshua Lerner, Ipek Bilgen, and Ned English; 2024
Energy balancing of covariate distributions, Jared D. Huling and Simon Mak, 2024
UnifyingSmallAreaEstimatorsBasedOnAreaLevelAndUnitLevelModelsThroughCalibration; William Acero, Isabel Molina, and Juan Miguel Marín; 2026