Measurement error when surveying issue positions: a MultiTrait MultiError approach
Measurement error when surveying issue positions: a MultiTrait MultiError approach (DOI: https://doi.org/10.1017/psrm.2025.3) was written by Kim Backström, Alexandru Cernat, Rasmus Sirén, and Peter Söderlund in 2025. It was published in Political Science Research and Methods.
The authors estimate method effects, acquiescence (a bias towards 'agree' statements on agreement-disagreement scales), and random errors in issue preference surveys.
The authors designed a 2 wave experiment incorporated into waves 5 and 6 of the Citizens’ Opinion panel's 2023 Finnish parliamentary election study. This is a mixed probability and nonprobability sample. The experiment was conducted after the election in question. The entire sample included 4875 respondents, but analysis was restricted to the 3175 that participated in both waves. Also excluded were those who completed the wave 2 survey before or on the same day as the wave 1 survey. "[M]ean time between waves of 9.6 days".
The experiment incorporated randomization between 5- and 11-point scales (method effects), and between scale directions (acquiescence). Within-respondent effects were measured by administering the same questionnaire with complementary assignment in the second wave. Question order was also randomized.
The authors first use a MTME model to decompose measured variance between traits and each type of error. They found method effects and acquiescence to be minimal. Splitting by methods, they found that 5- and 11-point scales are very similar in terms of trait variance (78% to 75%). Splitting by the traits, they found substantial differences in trait variance, suggesting that certain traits are more susceptible to random error.
The authors then use several multigroup MTME models to estimate moderating effects. In particular, they look at political interest, "internal political efficacy" ("Internal political efficacy is seen as an individual’s assessment of their ability to understand what is going on in politics (Niemi et al., 1991)."), and having a degree.
- Overall, political interest does not have a substantial moderating effect (trait variance is 78% for high interest and 76% for low interest). But splitting by traits, they found large differences. Suggests that there is more random error among those with low political interest.
- "Allowing private corporations to be responsible for elder care to a greater degree" sees trait variance of 73% for high interest and 60% for low interest.
- Largely the same conclusions for other two moderating variables.
The authors perform a sensitivity analysis with regard to time between the two interviews. Controlling for the time difference, they found no significant differences.
Lastly, the authors perform a sensitivity analysis with regard to sampling source. Overall, the difference is minimal. Splitting by the traits, they found differences that were not consistent in direction.
- "Allowing private corporations to be responsible for elder care to a greater degree" sees trait variance of 70% for non-probability panel and 64% for probability sample.
- "Increasing the digitalization of society" sees trait variance of 70% for probability sample and 65% for non-probability panel.
Reading notes
Really impressive work that demonstrates how to validate a survey question. I don't fully understand the methods but this would be a great replication to try out.