Matthew Sweet has written in UnHerd about the importance of following footnotes in studies to find out if the references actually say what the studies claim they say and genuinely back up the argument being made. He suggests this indicates whether or not the study should be considered reliable.
One of his examples is the mask study by Dr Baruch Vainshelboim, now retracted, that I wrote about yesterday. He says a number of the footnotes are misrepresented (this criticism was part of the retraction notice).
If Dr Vainshelboim did misrepresent the papers he cites he would not be the first. As noted yesterday, a recent peer-reviewed study in PNAS claimed surgical masks filter out 95-99% of aerosol droplets. Yet the two papers it cites to back up this claim say nothing of the sort. One concludes: “None of these surgical masks exhibited adequate filter performance and facial fit characteristics to be considered respiratory protection devices.” This is not to defend Dr Vainshelboim’s misrepresentation of course, but to highlight the double standards applied to those who challenge political orthodoxies.
Today I thought I would follow Matthew Sweet’s advice for the Government’s own evidence. We learned yesterday that face masks may continue after June 21st, with no indication of when the mandate may be lifted or what conditions may trigger it. What scientific evidence is this seemingly permanent coercive public health measure based on? After all, the real world evidence for masks preventing outbreaks is feeble, to say the least, as Yinon Weiss has dramatically illustrated.
The Government has often been slow to publish evidence for its supposedly scientifically based interventions. But in January its scientific advisory group SAGE published a paper in which it set out its current evidence on masks. This included an important admission that masks give no real protection to the wearer, saying: “They may provide a small amount of protection to an uninfected wearer; however, this is not their primary intended purpose (medium confidence).” They say they are “predominantly a source control”.
Face coverings worn in public, community and workplace settings are predominantly a source control, designed to reduce the emission of virus carrying particles from the mouth and nose of an infected person. This may have measurable benefits in reducing population level transmission when worn widely, through reducing the potential for asymptomatic or pre-symptomatic people spreading the virus without their knowledge. Analysis of regional level data in several countries suggest this impact is typically around 6-15% (Cowling and Leung, 2020, Public Health England 2021) but could be as high as 45% (Mitze et al., 2020).
This is the key paragraph in terms of providing evidence for the effectiveness of face masks, and on closer inspection it is a mess. It says: “Analysis of regional level data in several countries suggest this impact is typically around 6-15%.” Yet the 6-15% figure comes from the Cowling and Leung paper, which is not an analysis of regional level data but an editorial article drawing on a December 2020 review paper by Brainard et al. The Brainard paper reviews 33 studies including 12 randomised controlled trials (RCTs), but none of these is an analysis of regional level data.
The Mitze paper actually is an analysis of regional level data, but only in Germany not in several countries. It was submitted in July 2020 and is based on data from the decline of the spring wave. As infections were falling then anyway it is very hard to distinguish the possible effect of masks from natural decline. In any case, the mask mandate in Germany did not prevent the winter surge, as the graph above depicts.
The Public Health England 2021 paper does not appear in the bibliography or anywhere else in the paper so remains a mystery as to what it might contain.
Brainard thus appears to be the key paper for evidence, and anyway is the source of the 6-15% claim. As noted, it reviews 33 studies on the effectiveness of face masks in preventing transmission of respiratory disease (not COVID-19), of which 12 are randomised controlled trials (RCTs). The RCTs give the lower figure of 6%. The higher estimates all come from other, less robust types of study (cohort, case-control and cross-sectional studies). Not included (because it wasn’t published at the time the review was written) is the one RCT specifically on masks for COVID-19 – Danmask-19 – which found no significant protection for the wearers of surgical masks.
Contrary to the claims of the SAGE paper, however, Brainard’s 6-15% estimate is expressly for protecting the wearer, not source control. Yet this is supposed to be SAGE’s evidence for masks as source control, having admitted they are no real benefit to the wearer. Brainard does look at the evidence for source control, but it grades the quality of the evidence as low or very low. Many of the studies it looks at show small reductions of around 5%, though some show greater reductions, while others show significant increases (e.g. one where masks are associated with a 200% increase in transmission). The best quality evidence (which isn’t saying much here) is five RCTs where all members of households (ill and well) wear masks, which indicate masks may reduce risk of infection by 19%. However, Brainard grades the quality of this evidence as low, remarking it was “downgraded twice overall for risk of bias, imprecision and inconsistency”.
Brainard calls for “COVID-19-specific studies”. Danmask-19 is the only Covid-specific RCT so far. It wasn’t published in time for Brainard but it was in time for SAGE’s January paper, yet was not included. SAGE has yet to publish any advice which cites it.
So let us summarise. SAGE’s most recent advice on masks says they are of little benefit to the wearer and are “predominantly a source control”. In evidence of their effectiveness for this it cites three papers which it claims provide an “analysis of regional level data in several countries” that suggest masks reduce transmission by 6-15% (for two of the papers) and 45% (for the other). However, one of the papers cannot be traced, another (the source of the 45% figure) is a regional analysis of just one country (which despite a mask mandate suffered a large winter surge), and the other, the source of the 6-15% figure, takes the estimate from a review paper that includes no analysis of regional level data. Furthermore, 6-15% is its estimate for protecting the wearer not source control (with the lower figure coming from the most reliable studies) and it does not include any Covid-specific studies. Its evidence for source control it grades as low or very low.
Matthew Sweet was right. Following footnotes is eye-opening for exposing unreliable sources of evidence. Like SAGE.
