Is It the Message or the Messenger? Examining Movement in Immigration Beliefs
Is It the Message or the Messenger? Examining Movement in Immigration Beliefs was written by Hassan Afrouzi, Carolina Arteaga, and Emily Weisburst. It was published in the Journal of Political Economy Microeconomics volume 2 (2024).
The authors design an experiment to isolate the effects of political messages and the effects of who gave the political message. Pro- and anti-immigration speeches, as well as non-ideological (turkey pardoning) speeches, are selected from Obama and Trump. The same ideological speeches are reproduced with a voice actor. The non-ideological speeches enable control for effects of political messages. The reproduction enables control for the effects of who gave the political message.
Measured outcomes are considered as relative to the individual's sociopolitical group; an anti-immigration belief is indicated by 1 for Republicans and by 0 for Democrats.
Overall, treatment with a message results in a change of beliefs in the direction of the message. Changes are of greater degree when the message direction is opposite to what is expected from the source.
Source persuasion effects only exist for within-group audiences. That is, a message from Obama can persuade Democrats toward an anti-immigration belief, and Trump can persuade Republicans toward a pro-immigration belief.
There is no significant priming effect; presentation of a politician alone does not polarize.
In summary: When politicians give a message that is 'expected', there is a polarizing yet negligible effect on the audience's beliefs. When politicians give a message that is 'unexpected', there is a de-polarizing effect. This suggests a strong cult of personality effect.
Probability Theory
A sample space Ω represents an individual's opinion about immigration (binary; favorable or unfavorable). Subjective opinions--as in given S sources of political messages and M expected messages from each source--are given in Ω‾ = Ω * M * S.
The unconditional probability of belief ω is given by P(ω) = Σ P(ω,s',m') for all s' ∈ S and m' ∈ M.
For a 'treatment' s' ∈ S and m' ∈ M, the conditional probability of belief ω is given by P(ω|s',m').
It is more realistic to measure relative probabilities of beliefs in an experiment, however. Therefore the treatment belief relative to the unconditional belief is given by:
Decomposition of Anonymous Message Effects and Source Persuasion Effects
We can measure unconditional beliefs P(ω), beliefs given an anonymous message P(ω|m), and beliefs given a message from a source P(ω|m,s).
Decompose the first equation with Bayes' rule and log probabilities into separate anonymous message effects and source persuasion effects:
Again by Bayes' rule, decompose the conditional probabilities:
...or using an odds ratio (Θ) formulation:
P(ω|m) = P(ω)Θ(m|ω)
This is an immediately useful formulation: the anonymous message effect βm increases with the odds ratio Θ(m|ω) and is zero iff the odds ratio is one.
Similarly now:
Note furthermore that:
Lastly, understand that for all s ∈ S there is a complementary ¬s such that:
P(w|m) = P(w,s|m) + P(w,¬s|m) = P(s|m)P(w|s,m) + P(¬s|m)P(w|¬s,m)
This all comes together to calculate the source persuasion effect βms which increases with the odds ratio Θ(s|m,ω) and is zero iff the odds ratio is one. As an intermediary step:
There is a source persuasion effect iff:
Θ(s|m,ω) > 1
...which can be more easily worked with as:
Θ(s|m,ω) - 1 > 0
Substitute the left-hand side's odds ratio Θ(s|m,ω) and combine terms:
Substitute the numerator's term P(ω|m) with the expanded expression above:
Factor out the redundant term:
Decompose with the complementary rule and factor out that term:
This is interpreted as:
P(ω|m) is a prior belief given messages
(1 - P(s|m)) is a surprise term that captures how unexpected a message is from a source
(P(w|s,m) - P(w|¬s,m)) is a reliability of source term that captures the reliability of the source relative to all others
However, it is possible for prior beliefs to be zero. As a result, it's necessary to re-map this to terms of log probabilities plus 1:
Decomposition of Source Priming Effects
We can additionally measure beliefs given a non-political (turkey pardoning) message m0. If this is truly an independent message, then P(ω|m0,s) = P(ω|s).
Returning to the first equation, decompose into separate source priming effects and identified message effects:
Through nested application of Bayes' rule, decompose the conditional probabilities:
And also note that each underlined term is equivalent to the odds ratio Θ(s|ω).
By the same logic as above, the source priming effect is refactored and remapped as:
Methods
The authors regress one the log probability plus 1 of a certain belief (i.e., ln(1 + P(yi)). The coefficient βt is the total effect (which is separately decomposed into component effects). A double lasso procedure is also employed.
The authors model on data collected from surveying with random treatment of 1 audio; there is also an untreated control group.
Statistical significance of changes in belief are tested with Kolmogorov-Smirnov.
Reading Notes
This paper gives me a headache. I'm not sure if that means I'm terrible at probability theory, or I'm terrible at Bayesian models, or this paper is terribly written. But damn am I proud for finally working through the model.
I am a bit off-put by the outcome measurements (i.e., 1 for Republicans means 0 for Democrats). I have some doubt that they succeeded in contacting a representative cohort of Republicans through a panel. I have significant doubt about the reliability of self-reported party alignment. I think there are severe philosophical issues with simplifying the policy space of immigration to 'Democrats pro, Republicans anti'. I am, overall, unsure of how this impacts the interpretation of the model.