How many COVID-19 vaccine-related serious adverse events are acceptable? This is the question that we will concern ourselves with in this pair of articles, and one which we will fail to provide a simple answer to. In fact, our intention is not to answer the question per se, but to show how difficult it is to answer.
Firstly, let’s define what we mean by a serious adverse event for the COVID-19 vaccines:
An SAE is an adverse event that results in any of the following conditions: death; life-threatening at the time of the event; inpatient hospitalisation or prolongation of existing hospitalisation; persistent or significant disability/incapacity; a congenital anomaly/birth defect; medically important event, based on medical judgement.
Definition taken from Fraiman et al., Vaccine. 2022 40(40) pp 5798-5805
As you can see, an SAE is not a trivial thing. SAEs are potentially life-altering or even life-ending events, but most medicines carry some such risk and so the question is therefore not one of requiring absolute drug safety but understanding whether the benefit of the treatment outweighs the risk; the ‘pill cannot be worse than the ill’. Unfortunately, demonstrating that a treatment has an acceptable benefit-risk ratio is not an easy thing and usually involves both clinical trial data and ongoing post marketing surveillance, but it is one of the most important factors in both gaining and maintaining drug approval.
The fact that the COVID-19 vaccines carry the potential to produce SAEs is beyond dispute (for example see the drug information sheet for the Pfizer COVID-19 vaccinations) and so the real question we are asking with respect to the safety of COVID-19 vaccines is:
How many COVID-19 vaccine related serious adverse events are acceptable given the potential benefit the vaccines provide to individuals receiving them?
To try and answer this question, we’ll look at two different ways of thinking about it either at the individual or population level. We’ll discuss individual benefit-risk in part 2, but the focus of part 1 will be on the population level approach, which we will call the collectivist worldview.
The way the question of vaccine benefit-risk has generally been tackled during the COVID-19 pandemic is by reference to the benefit of the vaccines to the population as a whole. This is a spreadsheet view of medicine that ignores the individual in preference for the ‘collective’. It is also the approach taken by epidemiologists and modellers, who by necessity need to deal with such population-level effects. Ultimately, this worldview boils down to saying that it is worth ‘deploying’ the vaccine as long as the number of COVID-19 deaths and serious illnesses avoided within the population is greater than the number of vaccine-induced SAEs.
But there is a problem with this approach, namely you can’t count what doesn’t happen. Using clinical trial’s data we can get a handle on the likely effectiveness of the vaccines, but this isn’t the same as being able to accurately say how many lives are ‘saved’ and so, to translate the vaccine effectiveness into benefits across the population we must use our old friend, modelling. To illustrate the issues with this, let’s concentrate on COVID-19 mortality and do our own simple population benefit modelling as presumed at the time of vaccine rollout. Here are the assumptions from publications at the time:
- World population is about 8,000,000,000 people.
- Percentage of the population who have been fully vaccinated is about 67%, meaning that 5,360,000,000 people are fully vaccinated.
- If the average infection fatality rate for SARS-CoV-2 is 0.5% and there were no vaccinations and everyone vaccinated was infected, then this means that 26,800,000 (vaccinated) people would have died of COVID-19 (5,360,000,000 × 0.5%)
- The vaccinations are 90% effective and so rather than 26,800,000 people dying, only 2,680,000 people will die (26,800,000 – 26,800,000 × 0.9).
- The vaccines have therefore saved 24,120,000 lives globally (26,800,000 deaths with no vaccine – 2,680,000 deaths with the vaccine)
Real epidemiological models are more sophisticated than this, but the figure from our noddy calculation is not so dissimilar to that bandied around based on the professional modellers. As with all models this one is riddled with assumptions such as the whole population gets infected, the average infection fatality rate is 0.5%, there are no geographic or racial/genetic differences in susceptibility to disease or benefit from vaccination, there is no prior immunity, etc., etc., etc. Any one of these things will affect the prediction from the model. For example, if as seems likely the infection fatality rate is closer to 0.05% that 0.5%, then all of our estimates of COVID-19 related deaths are 10-fold higher than they should be and the consequent number of lives predicted to be saved by our model is now 2.4 million and not 24 million, an order of magnitude lower than our original figure – and this is if all our other assumptions are correct. Such modelling figures of the benefits of COVID-19 vaccines, for all their apparent sophistication, are opinions and best guesses rather than actual data meaning that understanding the true benefit for the vaccinations to the population is off to a fairly shaky start.
So much for the benefits column in our spreadsheet, what about the risks? Here rather than the IFR, we could use the vaccine fatality rate (VFR) i.e., how likely is it that someone will die due to COVID-19 vaccination. This is a good way of understanding the benefit and risk because we’re using very similar concepts and measuring the same endpoint (death). But what is the VFR in the population? As we’ll discuss below, getting an accurate assessment of this is quite hard but first let’s look at how the VFR affects our predictions of lives saved. For a VFR of one in a million then vaccinating 5.36 billion people would be predicted to result in 5,360 deaths, far fewer than the number predicted to be saved even using the more realistic IFR of 0.05%. Our spreadsheet is firmly tilted to the benefits side of the equation. But this figure of risk may be too low and analysis by Dr. Eyal Shahar suggests the VFR for the mRNA vaccines could be as high as 12 in a million, meaning the number of predicted deaths from the vaccinations for these vaccines would be about 64,000. This is still less than prediction for lives saved but is now large enough to have a noticeable effect on the ‘net gain’ to vaccination. Estimates by Professor Norman Fenton of the VFR for the AstraZeneca vaccination suggest for this vaccine it could be even higher at 1 in 5,300 per dose (187 deaths per million doses), a level of vaccine induced fatality that would produce about one million vaccine related fatalities and mean that for an IFR of 0.05% the gains in vaccination are now halved and we’re killing one person to save two – and this is for a vaccine that is assumed to be 90% effective. As you can see, depending on the assumptions we use of things like IFR, VFR and vaccine effectiveness our population level analysis can describe the vaccines as being everywhere from ‘safe and effective’ to ‘dangerous and useless’.
On paper, determining the number of vaccine related SAEs in the population is an easier exercise than modelling the benefits, because these are real events and so theoretically countable. The problem is that doing this outside of the context of a structured assessment such as a clinical trial is extremely difficult as it relies on spontaneous reporting into systems like VAERS and the Yellow Card system. Such systems suffer from both significant under-reporting of SAEs and misattribution of cause meaning that although they can act as a route to identifying potential vaccine-induced SAEs they cannot be relied on to give an accurate account of the number of such SAEs in the population. We can use estimates of both under-reporting and misattribution to try and get a sense of the real frequency of SAEs, but as with modelling the benefit, we are now making best estimates and these are dependent on our assumptions, (again see this article by Professor Norman Fenton).
Then there is the problem that we don’t have anything to directly compare our vaccine SAE reports to, such as a control group. In a clinical study all SAEs are collected, regardless of whether they are drug related or not. As a result, at the end of the study there will be a large list of SAEs for both the treatment group and the control group, and it is this second group that gives us a sense of the type and frequency of SAEs not related to treatment. However, there are no equivalent VAERS or Yellow Card reports from unvaccinated individuals for the obvious reason that people do not produce a spontaneous safety report to say they had a suspected SAE after not having had a treatment. There is no Yellow Card report saying, “I had myocarditis after not receiving a COVID-19 vaccination”. We could use spontaneous reports from other vaccinations, but these will also suffer from their own rates of under-reporting and misattribution. Additionally, many individuals having other vaccinations could also have had the COVID-19 vaccine and so they may not be true comparators, especially if the vaccinations happened close to each other. So, we’re left in the situation where understanding the ‘natural occurrence’ of some SAEs within the non-COVID-19 vaccinated population is hard and as a result quantifying an increase in frequency for the vaccine treated is even harder.
This problem of the control group runs even deeper. In a clinical trial this might be a placebo group or, in the case of the COVID-19 vaccinations, a control vaccination group who receive an unrelated vaccination. This group provides the ‘background’ rate of both serious illness and death from COVID-19 so we can measure our vaccine’s effectiveness, as well as the rates of adverse events so we can assess what is treatment related and what is just ‘normal for Norfolk’. Once on the market we lose access to this group and are left with the question of who now are our controls. We could try and look at alternative vaccinations (such as flu) and try and recreate an equivalent group to the trials, but as noted above this group will be compromised because many such individuals will have had the COVID-19 vaccines and so are not true controls. The obvious answer is therefore to compare outcomes to unvaccinated individuals, but there are major problems in doing this outside of the trial context.
Firstly, we need an accurate count of the number of unvaccinated individuals in the population because any assessment of benefit and risk is going to be sensitive to the ‘denominator’. But doing this isn’t as easy as it might seem, not least because it often needs to be inferred by taking the number of vaccinated individuals away from the potentially vaccinated population and this is open to problems of accurately knowing the size of the ‘potentially vaccinated population’ and defining what we actually mean by ‘unvaccinated’. Professor Norman Fenton presents an example of how miscounting the unvaccinated substantially affects the calculation of vaccine effectiveness, and similar problems may dog calculations of the background rates of SAEs in the unvaccinated population.
Secondly, we really need to compare like with like, which means having ‘matched comparators’ (e.g. similar ages, same sex, equivalent levels of baseline health) and, again, away from the controls in clinical trial we are left in a situation where this may be difficult, and we risk comparing apples and oranges. For example, some individuals may not be vaccinated because they are too unwell to be vaccinated, but if we wished to include these individuals in our deliberations, we need to compare their outcomes with other equally unwell but vaccinated individuals – who, by definition, do not exist! Including these people in any assessment of benefit will bias the results towards more positive vaccine effectiveness because they are too unwell to have the vaccination and so will be in the unvaccinated group, but being so unwell also means they are more likely to die from COVID-19 if they are unfortunate enough to develop the disease. Groups like this can contribute to the so-called ‘healthy vaccinee’ or ‘survivor’ problem.
Finally, we have lost the blind. High quality clinical trials are double-blinded meaning that neither the patient receiving treatment nor the physician giving the treatment knows what he or she is getting or giving. This ‘double-blind’ aims to remove bias in the study that might be introduced due to knowledge of the treatment being received but is obviously lost once the treatment is in general use because everyone knows whether they had it or not. How many vaccinated individuals do you know who when they come down with COVID-19 say something like “think how much worse it would have been if I hadn’t been vaccinated”? There is no evidence that this statement is true and no way of proving it either way, but this is precisely the kind of bias that study blinding aims to avoid. Doing a truly objective assessment of the impact of the vaccination on disease severity is now potentially heavily compromised by an assumption of a positive vaccination effect.
When considering the benefit-risk ratio for a vaccine we need to always keep in mind that these are preventative therapies and as a result are given to otherwise healthy people with the aim of avoiding a future event. In fact, someone only receives the potential benefit from a COVID-19 vaccination if they become exposed to SARS-CoV-2, and if that exposure would have gone on to produce significant disease or even death. This raises the bar for vaccine safety as it means that unlike a disease treatment, the vaccine will be given to healthy people who may receive no benefit, but all of whom will be exposed to the risk of the vaccine. But with the best will in the world, even large trials will not necessarily be able to identify rare SAEs and safety issues and so post-market analyses of the treated population, despite all the issues discussed here, are essential to identifying such rare events. The ‘Green Book’ is the U.K. Health Security Agency’s ‘bible’ with respect to immunisation against infectious disease and makes this specific point in chapter 9 on surveillance and monitoring for vaccine safety.
As not all side effects [from a vaccine] may have been identified prior to licensing, particularly if they occur very rarely, careful surveillance is required throughout their use.
So, the expectation is that the true safety picture will only be painted once the vaccines are in real-world use. If fact, we start with the assumption that there may well be additional safety problems and so for new vaccines make a special effort to try and ensure we capture them:
Newly licensed vaccine products are subject to enhanced surveillance and are given ‘black triangle’ status (indicated by an inverted triangle on the product information). For such products, all serious and non-serious suspected ADRs [adverse drug reactions] should be reported, for both adults and children.
Given the abbreviated development route taken by the COVID-19 vaccines, such a need for enhanced surveillance would appear to be even more pressing. Of course, whether this has happened or not is another question. The report by the Perseus group suggests that in fact this level of enhanced pharmacovigilance has not happened for the COVID-19 vaccines in the U.K. and consequently the words in the Green Book are just that, words – and no more likely to ensure patient safety than clicking one’s heels and whispering ‘safe and effective’ three times.
Because the safety picture of vaccines (or any treatment) only really becomes clear once they are in real-world use, the benefit-risk described after clinical trials is extremely unlikely to be accurate and will almost certainly need to evolve over time. It may well be that the initial post-trial assessment that a treatment is ‘safe and effective’ (i.e., has a good benefit-risk ratio) might not last once it is approved and being given to a much wider and more diverse population of people. Also, for an infectious disease like COVID-19, as the virus mutates and evolves so will the disease severity and vaccine effectiveness and with it our assessment of benefit-risk. However, the issue is that like many things Covid, the first answer we thought of must be ‘The Truth’ because anything else is to admit we didn’t really know. So, the COVID-19 vaccines are ‘safe and effective’ (whatever that means) and there is no room for refinement of this position – well not publicly anyway.
Finally, although taking this kind of a collectivist, population level view of things can be a useful way of understanding the benefit and risk of a vaccine, it can drive us down a troubling road because it can lead to the logical conclusion that it is acceptable to include broader benefits to the population in our benefit-risk calculation. We move the benefit of the vaccine from ‘protecting oneself’ to ‘stopping the spread’ or ‘protecting Grandma’ and doing this has the effect of inflating the benefits side of the benefit-risk considerations. It also means we can justify vaccinating individuals who receive very little personal benefit if doing so produces perceived overall benefits to society, such as limiting viral transmission. Inevitably, this road leads to coercive policies to try and enforce vaccination because to actualise the broader vaccine benefits to society demands that every individual ‘plays their part’.
In part 2 of this article, we will move away from thinking about benefit and risk at the population level and focus on where it is most important, to the individual.
George and Mildred are pseudonyms of senior individuals working in the pharmaceutical industry.
To join in with the discussion please make a donation to The Daily Sceptic.
Profanity and abuse will be removed and may lead to a permanent ban.