How Bad Was Learning Loss During the Pandemic?

We know that school closures hampered student learning during the pandemic. But just how bad was the effect? And how widespread? Those are the questions that Bastian Betthäuser and colleagues sought to answer in a paper published at the beginning of last year.

The authors carried out a systematic review and meta-analysis of learning loss over the first two-and-a-half years of the pandemic. By combing the literature, including repositories of unpublished papers, they were able to identify 61 relevant studies.

Before running the numbers, Betthäuser and colleagues manually checked each of the 61 studies for various methodological biases, such as confounding. Based on widely used criteria, they determined that 19 studies (that is, 30% of the total) had a “critical” risk of bias. Excluding these from the analysis left them with 42. And since most studies reported multiple estimates of learning loss – for different subjects and grade levels – there were a combined total of 291 estimates.

The authors also checked for publication bias, but found no evidence that the studies with smaller samples reported larger estimates.

So, what did they find? Averaging the estimates for each study and then pooling across studies, they obtained an overall effect size of d = –0.14. This means that average learning loss was about 14% of a standard deviation of students’ test scores. What does this mean in practice? Well, students typically progress by about 0.4 standard deviations per school year, which means that school closures reduced learning by the equivalent of one third of a school year (a whole term). The authors characterise this as “substantial”.

Next they looked to see whether learning loss decreased over the course of the pandemic. Were later estimates smaller than earlier ones? Their chart is shown below:

Chart from ‘A systematic review and meta-analysis of the evidence on learning during the COVID-19 pandemic’.

Here the dots correspond to individual estimates, rather than averages for each study. As you can see, there is no evidence that learning loss decreased; if anything, it increased. However, the authors checked to see whether later estimates were significantly larger than earlier ones, and found that they weren’t. The most reasonable interpretation of the data is that learning loss arose during the first few months of the pandemic (when lockdowns were most severe), and then persisted for the next two years.

Now, it’s possible that students who suffered learning loss will eventually catch up to where they would have been in the absence of school closures. Perhaps if you extended the grey line above into 2023 and 2024, it would begin to slope upwards. Yet as the authors note:

Existing research on teacher strikes in Belgium and Argentina, shortened school years in Germany and disruptions to education during World War II suggests that learning deficits are difficult to compensate and tend to persist in the long run.

Which students were most affected? It stands to reason that those from richer families would be less affected, since their home environments were more conducive to learning, and their parents could afford to compensate by hiring private tutors. And that’s exactly what Betthäuser and colleagues found.

They coded estimates for whether they showed an increase in educational inequality by socio-economic background, a decrease in educational inequality or no change. A large majority showed an increase – indicating that poorer students suffered greater learning loss. The authors also found that learning loss was greater in poorer countries.

So school closures not only disadvantaged poor students relative to rich ones within countries, but also disadvantaged poor countries relative to rich ones. This is obviously ironic, given that many on the left championed school closures and initially denounced those who opposed them.

Betthäuser and colleagues’ study is one of the most comprehensive and careful to date, and it basically confirms what we already knew: school closures came with major costs. What’s noteworthy is that, due to data availability, their sample of studies was skewed toward richer countries. Had it included more studies from poorer countries, the overall learning loss would have been even greater.