When you give cash to homeless people, do they just waste it on alcohol, drugs and cigarettes? No, according to a new study that’s been covered favourably in Vox, the Washington Post and several other outlets. Yet there’s more to the study than its headline finding.
Ryan Dwyer and colleagues carried out an experiment in which 50 homeless people were randomly assigned to receive an unconditional cash transfer of 7,500 Canadian dollars. Another 65 homeless people served as controls. The two groups were then followed for a year to gauge the impact of the cash transfer.
As is increasingly common in social science, the researchers preregistered their analyses. What does this mean? Before collecting any data, they published a document online describing the analyses they intended to carry out. The rationale for preregistration is to demonstrate that any significant results you report were not found through p-hacking (running and then rerunning the analysis until you find a significant result).
Dwyer and colleagues preregistered the following hypothesis: “Participants who receive cash transfers will demonstrate better outcomes than those who do not receive cash transfers”, with “better outcomes” referring to improvements on measures of cognitive functioning and subjective well-being. (Participants’ “assets and spending” were only measured as a “supplemental outcome” for “exploratory analysis”.)
So what did the researchers find? When it came to their preregistered hypothesis, there was no evidence the cash transfer had any impact. This is shown in the chart below. Those who received the cash transfer did not score significantly better than the controls on any of the measures of cognitive functioning or subjective well-being. (The chart shows results at the one year follow-up. But the differences were also non-significant at the one month follow-up.)
Where the researchers did find significant differences were in number of days homeless and spending on food, rent and durable goods. Here, those who received the cash transfer fared better than the controls. And as you can see, there was no significant difference in spending on “temptation goods” (alcohol, drugs and cigarettes).
In summary, the preregistered hypothesis was not supported – though the cash transfer recipients did fare better with respect to spending and number of days homeless.
Lack of support for the preregistered hypothesis was reported as the study’s finding, right? Wrong. It wasn’t even mentioned in the abstract (as noted by Jon Baron, President of the Coalition for Evidence-Based Policy). Instead, the study’s findings were portrayed as “unambiguously positive”.
And even the more encouraging results are somewhat suspect. There was substantial attrition in both groups (meaning that some participants dropped out of the study after randomisation). So we can’t be sure that the differences in spending and number of days homeless were actually caused by the cash transfer – rather than by selective drop-out.
What’s more, the absence of a difference in spending on “temptation goods” may be partly attributable to the fact that homeless people with “severe” substance abuse problems were exempt from participation.
While Dwyer and colleagues’ study does offer preliminary evidence that unconditional cash transfers help homeless people to get back on their feet, the findings were presented in a rather misleading way. Just reading the abstract of a scientific paper can be a useful time-saver, but it’s always good to check the main results yourself.
Profanity and abuse will be removed and may lead to a permanent ban.