Practical Considerations in Raking Survey Data
Practical Considerations in Raking Survey Data (DOI: https://doi.org/10.29115/SP-2009-0019) was written by Michael P Battaglia, David C Hoaglin, and Martin R Frankel in 2009. It was published in Survey Practice (vol. 2, no. 5).
There is also a paper of the same name, with an additional author (David Izrael), that has been distributed online. This alternative paper retains the core content but is much more verbose on the history of statistical computing, implementation details of specific raking algorithms (esp. the IHB SAS macro), and discussion of the existing literature.
The authors recommend selecting raking variables on the basis of strong associations to key metrics, nonresponse, or noncoverage.
Control totals often come from an external source, such as CPS. When collecting controls from multiple sources, the dimensions must be adjusted such that they sum to the same population total. For example, if using Census estimates for most controls but supplementing with CPS estimates for household income as an additional control, the authors recommend adjusting the household income dimension by the ratio of population totals.
The authors offer two approaches for handling missingness in raking variables:
- Impute the missing values using response data.
- Insert a 'missing' level into the control totals. If 4% of the survey responses lack data, then 4% of the population total is allocated to a new control level. The other control levels are then adjusted by the ratio of 'adjusted' and original population totals, such that the original population total is retained.
Complex raking dimensions can be created from the interactions of two or more raking variables. The authors recommend this in cases where there is a strong interaction, e.g. age and race/ethnicity. This generally eliminates any need for the non-interacted raking dimensions.
Convergence is usually defined in terms of the differences between marginal totals and corresponding control totals. Convergence is not guaranteed by the raking algorithm. The authors suggest an upper limit on iterations, e.g. 50. To enable convergence, three characteristics should be avoided:
- too many marginal constraints
marginal categories with small cell sizes (<5% of the sample)
- contradictory constraints
In other words, there is a trade off between the number of raking dimensions and the number of levels in raking dimensions. There is no clear answer to what is more optimal between e.g. 5 dimensions with 10 levels each or 10 dimensions with 5 levels each.
The authors discuss the trade off between precision and bias that comes from weighting trimming. "The MSE will be lower if the reduction in variance is large relative to the increase in bias arising from weight trimming." They mention that commonly used thresholds are: median plus 5x the IQR; 5x the mean; or the 95th percentile; etc. They also discuss how the IHB SAS macro can perform trimming between iterations of the raking procedure, rather than the more conventional post-raking adjustment.