Social Scientists Can’t Predict Better Than Simple Models, Study Finds

I previously covered a study which found social scientists can’t predict better than laymen. As you may recall, ‘experts’ and laymen were asked to predict the size and direction of social change in the U.S. between April and October of 2020. Overall, the two groups were equally inaccurate – there was no overall advantage for the ‘experts’.

A new study finds that social scientists can’t predict better than simple models either. (The study is a pre-print, so hasn’t yet been peer reviewed.)

Dillon Bowen collated data from several previous studies in which ‘experts’ had been asked to predict the impact of various interventions on people’s behaviour. His analysis covered five previous studies, but I’ll focus on the first three since the other two are rather complicated.

In the first study, 90 ‘experts’ were asked to predict how effective 53 different nudges would be at encouraging gym users to exercise more often. These nudges included things like administering a free audiobook and a bonus for returning after missed workouts.

In the second study, 24 ‘experts’ were asked to predict how effective 22 text-message interventions would be at encouraging Walmart customers to get a flu vaccine. These interventions included things like a simple reminder and a joke about the flu.

In the third study, 237 ‘experts’ were asked to predict how effective interventions carried out by two government nudge units would be at changing citizens’ behaviour. These interventions encompassed domains like healthcare, state benefits and community engagement.

For all three studies, Bowen compared the ‘expert’ predictions to a null model that said none of the interventions would work (i.e., none of them would change subjects’ behaviour by a statistically significant amount).

Remarkably, he found that in all three cases the null model performed better than the ‘experts’.

In other words, the ‘experts’ overestimated the interventions’ effects by such an extent that they would have been more accurate by simply guessing ‘no effect’ for every single one. Or put in other way, the effects of the interventions were generally closer to zero than they were to the ‘experts’ predictions.

Behavioural interventions are widely used by governments and other interventions to tweak people’s behaviour without coercing them. Yet practitioners consistently overestimate how well these interventions work. As it turns out, ‘they don’t work at all’ is a safer assumption than whatever the ‘experts’ are saying.