Unexpected Event during Surveys Design: Promise and Pitfalls for Causal Inference
Unexpected Event during Surveys Design: Promise and Pitfalls for Causal Inference (DOI: https://doi.org/10.1017/pan.2019.27) was written by Jordi Muñoz, Albert Falcó-Gimeno, and Enrique Hernández in 2020. It was published in Political Analysis (vol. 28, no. 2).
The authors describe a framework that takes advantage of coincidences of an active survey and an unexpected event. The pre-event respondents are the control, and the post-event respondents are the treatment.
They describe the following threats to validity:
- The event must be unexpected; anticipation of an event is expected to lead to smoother behavioral adjustments.
- Exclusion
- The event must be singular; "collateral events" and "simultaneous events" confound the causal effects estimated.
- "Endogenous timing of the event" could be a factor for events such as violence or a news story breaking. Some people decided when the event occurs, so it is not an unexpected event for everyone. At best, those segments of the population have to be excluded from analysis.
- Ignorability
- The coincidental experience of the event must be "good as random".
- Imbalances on observables and unobservables (e.g. reachability and attrition) are confounders.
- Noncompliance
- The treatment cohort must be uniformly exposed to the event.
Checking for imbalances between control and treatment is recommended, as by differences of means.
Evaluation of the tradeoff between candidate bandwidth levels and statistical power is recommended. A wider bandwidth captures more responses, but increased temporal distance between responses increases risks to causal analysis such as comparability.
If there are failures of ignorability, methods such as matching techniques can improve covariate balance. This should not be done using any covariates that themselves may have been influenced by the event, e.g. self-identification.
Patterns of nonresponse around the event should be analyzed for differential changes across groups, in anticipation that certain groups will be differently influenced.
The data should be analyzed for pre-existing trends irrespective of the event.
If causal effects are expected to be constrained to a particular region or country, this should be exploited as a counterfactual; estimate the treatment effect where there should be none.
If there is noncompliance among the treatment cohort, the estimated effect should be analyzed as intent to treat effect (ITT) rather than average treatment effect on the treated (ATT).
They use an illustrative original analysis: did the Charlie Hebdo terrorist attack influence support for the French government, as measured by the 7th round of the ESS? They include the power analyses that lead to selecting a bandwidth of ±20 days around the event. They re-balance covariates through entropy balancing to re-weight units.