A new population study from Israel, published in the Lancet on Wednesday, finds that the Pfizer vaccine is 95.3% effective against SARS-CoV-2 infection once a person is fully vaccinated (defined as being a week past their second dose). It also finds the vaccine is 94.8% effective in those aged 65 or older once fully vaccinated.
This is in line with other studies and is a very encouraging result. However, as with previous studies, it’s not clear how well the researchers have taken into account the fact that infections were declining anyway during the study period and whether this has led to an over-estimation of vaccine effectiveness.
To test this I accessed the data available from the Israeli Government. I looked at how many cases occurred in each age group each week alongside the proportion of that age group that had been fully vaccinated by that week. This allowed me to calculate how many infections we would expect to occur among vaccinated people in each age group each week if you assume the vaccines don’t have any effect. I then added these together to give a baseline number of cases in each age group to compare against the number of actual cases among the vaccinated as reported in the study. The results are shown below.
I calculated we would expect 43,826 infections among the vaccinated out of a total of 237,700 in the study period (January 24th to April 3rd) if the vaccines have no effect, which is 18.4%.
The study reports 6,266 infections among the vaccinated out of a total of 232,268 during the study period, or 2.7%. (I wasn’t able to discover why the study had about 2.3% fewer infections than the Israeli Government data broken down by age, but by using proportions we can avoid this discrepancy affecting the calculation.)
A proportion of 2.7% is 85.4% lower than a proportion of 18.4% that we estimated if the vaccines had no effect. This suggests a vaccine effectiveness of more like 85% than 95%.
Looking now at the crucial older age group, if the vaccines had no effect I have calculated we would expect 11,332 infections among the vaccinated aged 60 and over out of a total of 29,489 infections in that age group during the study period. The study found 2,201 infections among the fully vaccinated aged 65 or more. (It doesn’t state how many infections there were in total in this age group so we can’t calculate a straightforward proportion from the study.)
We need to adjust our expected figure of 11,332 to allow for the fact that it includes those aged 60-64 (the study uses different age brackets from the publicly available Government data). From the table above this will be about half of the infections in the 60-69 age group, or 2,834. We also need to reduce the expected figure by around 2.3% to allow for the different infection totals of the study and Government data. This gives us an expected figure of 8,329 infections among the vaccinated over 65s.
The 2,201 figure from the study is 73.6% smaller than 8,329, suggesting a vaccine effectiveness among the over 65s of more like 74% than 95%.
It’s not clear why the authors of the study did not do an analysis similar to this one. Taking into account the background prevalence of the virus should be basic, to avoid over-estimating the effectiveness of vaccines when they are rolled out during the decline of the epidemic.
The study (which was funded and approved for publication by Pfizer) briefly mentions lower vaccine effectiveness 2-3 weeks after the first dose, but does not give any information about effectiveness or infection incidence in the first 14 days. This means it gives no more information about the post-vaccination infection spike observed in other studies, though the silence here may be telling.
A further question is why the researchers gave no finer-grained detail about those older than 65 when they must have had the data to do so, and more than half of Covid deaths are in those aged over 80.
An effectiveness of 85% overall and 74% among the over 65s is still good, but it is not as good as the 95% figures in the study. As so often with vaccine studies, on closer inspection you’re left wondering whether you’re getting the full and accurate picture.
(This article has been corrected for some earlier mistakes in the numbers. The overall argument is unaffected.)
