Where to place sensitive questions? Experiments on survey response order and measures of discriminatory attitudes
Where to place sensitive questions? Experiments on survey response order and measures of discriminatory attitudes (DOI: https://doi.org/10.1017/psrm.2025.10017) was written by Amanda Sahar d’Urso, Tabitha Bonilla, and Genni Bogdanowicz in 2025. It was published in Political Science Research and Methods.
A sensitive question is defined as a survey item that is intrusive, increases risk of identification for the respondent, or has a social desirability dimension. This category can include demographics, but most attention is on question that measures e.g., illicit behavior like drug use, or (un)desirable behaviors like racism.
The authors investigate the effect of survey item ordering.
- Treating respondents with the treatment prior to sensitive items may bias the measurements as by priming or non-random attrition. From treatment onward, the control and treatment cohorts are not administered the same survey; any post-treatment measurements are not necessarily comparable.
- Placing the sensitive items ahead of treatments may activate some self-identification that changes responses. The experiment estimates treatment effects conditioned on activation.
The authors attempt to measure the ordering effect by reproducing experiments. Their experimental design is based on both a substantive treatment (i.e., whether or not the respondents read a vignette) and an order assignment (sensitive items pre- or post-treatment).
In particular, prior research has found ordering effects resulting from racial identity questions. So the authors here place priority on selecting experiments that study racial biases.
They do not find consistent evidence of an ordering effect. The exception is study 2, which is actually a conjoint experiment. (This is a two-wave experiment set in the U.S. that asks respondents to 'give a green card' to one of two profiles. The aim of the experiment is to decompose biases about race and religion; Muslim and MENA identites are often conflated but biases against one or the other may actually differ. Finding statistically significant differences in a bias toward one identifier does not necessarily affect tests of the conjoint hypothesis, and indeed did not in this reproduction.)