Why Are the UKHSA's Estimates of Vaccine Effectiveness Consistently Far Higher Than Other Studies?

The latest UKHSA Vaccine Surveillance report was released yesterday. It has a new section entitled “Vaccine effectiveness publications” which lists the relevant UKHSA and PHE publications, and which may well have been added in response to the emails of Daily Sceptic readers asking for an update on the UKHSA vaccine effectiveness study.

The top item in the list is the latest from their ongoing test-negative case control study – or rather the update-before-last, as for some reason the link goes to a pre-print from September, which then redirects you to a newer version published on October 6th. In any case, this means I was incorrect to state that there had not been an update using data since May, as these papers had somehow flown under my radar (for which, sincere apologies). In fact, the latest update uses data up to September 3rd, so still two months behind and including nothing from the autumn, but it’s certainly better than stopping in spring.

The chart below shows the estimates over time for the two main vaccines, indicating declining effectiveness.

It is immediately apparent, however that these estimates are significantly above those found in other studies. The new study claims, for example, that the AstraZeneca vaccine remains 47% effective after 20 weeks (five months), when a recent Swedish study found it was down to zero (or lower) after four months. It puts Pfizer at 70% effective after 20 weeks (five months), when the Swedish study put it at 47% after four to six months, and zero after seven months. A Qatari study similarly found the Pfizer vaccine effectiveness down to around zero after five to seven months (these figures are all for symptomatic infection). The UKHSA acknowledges the Qatari study and says it is in line with its own findings in terms of showing decline, but doesn’t discuss why its own study still found substantial protection when the other did not.

The suspicion is that there is something wrong with the design of the UKHSA study that means it is coming up with inflated vaccine effectiveness estimates owing to biases that it is failing to adjust for (and may not be able to). That was certainly the inkling I got in May when I analysed the study design when it first came out with data up to February 19th.

The biggest problem at that point was it wasn’t clear how well the study had controlled for calendar week, and thus for the phase of the epidemic and the background incidence. There were also questions of how representative the sample groups were of the general population given the case-control design and thus the fact that everyone in the study, positive or negative, was suffering from Covid-like symptoms (just not necessarily Covid). These issues may explain why, when comparing the unadjusted and adjusted data, huge adjustments were being made. For instance, AstraZeneca vaccine effectiveness was adjusted from 4% to 73%, while Pfizer was adjusted from 27% to 57%. These are for one dose, and the AstraZeneca figure at least is clearly way too high, given that the trial only found 70% efficacy for two doses.

This hardly gives us confidence for the latest round. However, this time we can’t compare unadjusted and adjusted figures as the study doesn’t provide the unadjusted figures. It provides some raw data, but not broken down over time so we can’t see how it’s changing so as to compare it to the vaccine effectiveness estimates. So we’re none the wiser. However, we can spot some strange features in the adjusted vaccine effectiveness estimates, such as that both vaccines are significantly better against Delta than against Alpha when it comes to hospital admissions for 40-64 year-olds, which raise an eyebrow.

In short, the UKHSA is publishing estimates of vaccine effectiveness that, while declining, are considerably higher than other studies without any real explanation as to why. This means it remains unclear why the real-world data the UKHSA publishes (see below) is showing infection rates in many age groups significantly higher in the vaccinated than the unvaccinated. Signals in the data indicate it may be owing to faults in the study design, which adjustments do not successfully address.

When the UKHSA publishes its next update with data from the autumn (which it really ought to do sooner rather than later), it should include the unadjusted estimates and raw data broken down by time so we can get a better idea of what’s going on. Another improvement would be for the agency to undertake another study using a different design (e.g. a retrospective cohort study) and compare the outcomes in order to assess the reliability of the findings.

In the meantime, we continue to use the real-world data in the Vaccine Surveillance report to estimate unadjusted vaccine effectiveness, with the latest table and charts shown below. These are unadjusted rates of course, and there may also be issues with the population estimates, but they give us some indication of what the vaccines are doing, particularly in the trends. Note that most of the vaccine effectiveness values have stabilised now, though the much higher infection rates in the vaccinated persist. Protection against death for the over-80s continues to decline steadily.