Since December 2020 the UK public has been unwittingly involved in a huge experiment that will reveal the clinical effects, both positive and negative, of the Pfizer/BioNTech (PF)1 and Oxford University/AstraZeneca (AZ)2 COVID-19 vaccines. Despite the absence of thorough animal trials of these products and lack of any human data beyond Phase 2 trials lasting a matter of months, they have been approved by the MHRA and are now being deployed in a programme of mass vaccination. As part of its statutory functions, as well as its legal and moral duty, the MHRA is responsible for monitoring the effects of these vaccines to ensure that their benefits to patients outweigh any risks.3
There are a number of reasons for anticipating that these particular vaccines pose more risk to the general public than traditional vaccines. Traditional vaccines contain products against which the immune system directly generates antibodies. These may, for instance, be inactivated forms of an otherwise harmful virus. Whatever their nature, they are not intimately associated with living human cells. In contrast the PF and AZ vaccines operate by hijacking the protein-producing machinery of human cells causing them to produce and display the spike protein of the SARS-CoV-2 virus. The immune system then raises antibodies against these proteins, thus providing protection against the real virus. However, the workings of the immune system are extremely complex, and it is not a simple matter to predict its manifold responses to the presentation of viral spike proteins on the surface of human cells located in multiple tissues throughout the body. A further reason for caution in deploying the novel vaccines is that during the development of vaccines against other coronaviruses it has repeatedly been found that vaccinated animals suffer more severe disease responses when challenged with the coronavirus itself, a phenomenon known as Antibody Dependent Enhancement (ADE).4, 5 We do not know whether ADE will be a problem with the COVID-19 vaccines because the relevant animal experiments have not been performed.
Given these very real risks, it is incumbent on the MHRA to put in place a professional, robust and transparent research programme in real time for quantifying the effects of the novel coronavirus on human health, allowing a rapid policy response if deleterious effects are detected. In order to do this, it is essential that the research programme is founded on sound experimental design. The objective must be to ensure that, if they occur, we can demonstrate and quantify causal relationships between the vaccine treatment and any beneficial or adverse effects.
A fundamental principle of experimental design is that if we wish to demonstrate a causal relationship between a treatment and an effect, we must have, as well as the treatment, a control against which to compare it. In the case of the COVID-19 vaccines our control population must comprise individuals who are part of the same community to whom the vaccine is being administered, but who are willing to forgo the vaccination. Within our experiment we will need to take account of factors other than vaccine treatment that could lead to a different outcome for vaccinated and control populations. Therefore, when documenting our vaccinated and control populations at the beginning of the trial, it will be essential to record as many characteristics of the individuals involved that could affect their response to vaccines. These might include sex, age, comorbidities, height, weight, ethnicity, pre-existing conditions, e.g. diabetes, etc.
The next step in the experimental design should be to decide on the nature of the measurements that we wish to make on our experimental and control populations. The primary information we are interested in is the health of the participants in the experiment. If we wish to involve a very large number of individuals in our trial, it will clearly not be feasible to make use of medical symptoms that can only be diagnosed by a physician. Therefore, for this wider trial, a list of symptoms seen in earlier trials, easily recognised and scorable by the general public, should be created, together with a common scoring system for these symptoms. Such symptoms would include headaches, fever, vomiting, chills, fatigue, muscle ache, swelling, rash, etc. The participants would be responsible for scoring their own outcomes for each health criterion, and the language used would be comprehensible to a lay audience.
Given that particular potential risks of the COVID-19 vaccines have already been identified (e.g. ADE) it would also seem imperative to include symptoms commonly associated with these conditions, even though they may not have been detected in earlier trials. To complement these measurements, it would also seem wise to undertake some targeted measurements of health outcomes that require more sophisticated analysis in a random subset of the participants in the vaccinated and control cohorts. An example might be estimates of lymphocyte counts that are already known, from Stage 2 trials, to be significantly reduced by the PF COVID-19 vaccination.6 Finally, it should be recognised that a number of the potentially adverse effects, e.g. ADE, may take several years to manifest. Therefore, from the outset, the reporting of symptoms by a cohort of participants should be planned for a timescale of 10 or more years, with more intensive reporting in the earlier period of the trial.
In summary, an appropriate experimental design for studying and measuring the health effects of COVID-19 vaccines on the UK population would comprise matched vaccinated and control populations documented for relevant characteristics that reported relevant health outcomes systematically over a continuous period of years. This large-scale trial would be supplemented with a smaller trial involving a random subset of individuals measured for critical tests requiring more sophisticated laboratory protocols.
Having outlined the framework needed comprehensively to monitor and measure the effects of the COVID-19 vaccines on the health of the UK population, we can now consider the programme that has been put in place by the MHRA and ask whether it is adequate for the task. The MHRA programme centres around the Yellow Card scheme that has an entirely descriptive brief, to compile a list of suspected adverse events from individuals who have been vaccinated either with the PF or the AZ product.
As a means of providing data to establish a causal relationship between vaccination and an adverse event, the Yellow Card scheme is fundamentally flawed. It does not yield data from a control group against which to compare the vaccinated group. In this situation all that can be established is a temporal association between vaccination and an adverse event and there is no means of establishing causation. Using this approach, it can always be argued that any association that is found is merely a coincidence. The Yellow Card scheme, because it is not founded on fundamental principles of good experimental design, is therefore not fit for the purpose of increasing our understanding of either the beneficial or the adverse effects of COVID-19 vaccines. It is not providing the protection from the possible harmful effects of COVID-19 vaccination that the UK public deserves.
Aside from this fundamental flaw, the Yellow Card scheme does not even generate a reliable summary of suspected adverse events that follow after vaccination. The reason for this is that it relies on a voluntary app-based reporting system which places the onus either on medically unqualified patients, carers and parents, or on qualified but more distantly connected medical staff to make a connection between vaccination and an adverse health event, and thereafter to file a report on this suspected connection. It takes little critical scrutiny to realise that a system based on these principles will be subject to huge underreporting either by individuals who are not medically trained to make such connections, or by medically trained staff who lack a close temporal connection with vaccinated individuals. A simple illustration of this fact is that in the Phase 2 COVID-19 vaccine trials, where individuals were comprehensively monitored for adverse events, the reported rates for symptoms such as headaches and fatigue were of the order of 30% – 50%, whereas under the Yellow Card scheme they were of the order of 0.3% – 0.5%. Only 1% of these events are being reported by the Yellow Card scheme. Not only is there huge underreporting, but we also anticipate considerable bias in a voluntary reporting scheme; those who believe that vaccines may do harm will be motivated to make a report, while those who are predisposed to dismiss a connection between vaccination and harm will not take the trouble.
Further barriers to accurate reporting of suspected adverse events are associated with the functioning of the Yellow Card app itself. In the first place, this does not demand basic information on the individual who has experienced the adverse event. Data like age, sex, height, weight, ethnicity, date of vaccination are all optional. This means that serious interrogation of the data to determine the effects of these factors in later analyses is precluded. Secondly, the app is set up in such a way that considerable medical experience is required to register an adverse reaction. There is no drop-down menu of possible adverse events to choose from. Instead, the reporter must type in a medical term that can be recognised by the system, which then provides a list of subsets of that medical condition to choose from. Thus, if the term headache is entered, some 50 different types of headache are listed, one of which must be chosen. Needless to say, this is likely to deter all but the most confident or medically informed from ever filing a report. Finally, experience has shown that the app itself is completely unreliable. After successfully filing a single report, it proved impossible to file a subsequent report because the app would not accept data on the nature of the adverse reaction. This level of malfunction is completely unacceptable in an official Government app.
Given that the Yellow Card scheme is so comprehensively flawed, is there yet something that can be salvaged from the data that it has provided? I have indicated that in general it is true that mere descriptive data on the frequency of adverse events following vaccination is insufficient to allow any causal relationship to be drawn between vaccination and the adverse event. All that can be established is an association, and the existence of a causal link can be easily dismissed. In this vein the latest report from MHRA6 and a summary of earlier data in the BMJ7 concludes that the COVID-19 vaccines have not played a role in the death of any vaccinated patients.
However, before accepting this conclusion let us remember that a particular demographic in the UK has, over the same time period (December 9th 2020 to February 14th 2021), been vaccinated with two different COVID-19 vaccines, PF and AZ. From the Yellow Card data, we have information on the number of doses of each of the two vaccines given, and the corresponding number of deaths following vaccination. Our null hypothesis is that the vaccination of individuals has no effect on the background rate of a particular suspected adverse event, for example death. If our null hypothesis is true, the rate of death following vaccination should be the same whether the PF or the AZ vaccine is employed. We can test this null hypothesis making use of the information from the Yellow Card reporting scheme. The test is crude, because the information available in the Yellow Card report does not break down the data according to particular age groups or other relevant factors. Nonetheless, as a preliminary investigation this approach can be informative, and depending on the result, may indicate whether more rigorous analyses are warranted.
The data used were those freely available from the Yellow Card report published on February 25th 2021 covering the period December 9th 2020 to February 14th 2021 .7 According to the report, approximately 8.9M vaccinations (first and second doses) had been given using the PF vaccine, and 6.9M with the AZ vaccine. The number of suspected deaths associated with these vaccinations were 197 (PF) and 205 (AZ). From these data we can draw up the following table to test our null hypothesis that the rate of death is independent of the type of vaccine administered.
| Vaccine | Alive | Dead | Total |
| PF | 8899803 | 197 | 8900000 |
| AZ | 6899795 | 205 | 6900000 |
| Total | 1579598 | 402 | 15800000 |
Contingency table for testing null hypothesis that rates of death following vaccination are independent of the vaccine administered
The result of a contingency χ2 test on these data indicates that the frequency of suspected death events differs highly significantly between the PF and AZ vaccines (χ2(1) = 8.7662, p<0.005). Our null hypothesis that there is no effect of the vaccine type on the rate of death is rejected. For every death associated with a PF vaccine injection, there are 1.34 deaths associated with an AZ vaccine injection.
Care is required in concluding uncritically from this result that vaccination has an effect on the rate of death of participants in the vaccination programme, since we have not had access to data that would allow us to compare the two groups who were given the PF and AZ vaccines. We are assuming in our analysis that the two groups do not differ in some systematic way which would affect their background rate of death, and this may not be true. However, the size and significance of the effect we have demonstrated indicates that it is important to pursue this question further to determine why death rate differs so markedly for the groups receiving PF and AZ vaccines in the UK to date.
The same form of analysis can be extended to determine the influence of the type of vaccine administered on suspected adverse reactions that are far more common than death. This has been summarised together with the previous analysis for a number of suspected adverse reactions that have a relatively high prevalence (Table 2).
Suspected Adverse Event | Ratio PF:AZ | χ2(1) | P |
| Death | 1:1.34 | 8.77 | < 0.005 |
| Tremor | 1:7.09 | 1640.7 | < 0.001 |
| Vomiting | 1:3.05 | 946.0 | < 0.001 |
| Fever | 1:3.20 | 4836.7 | < 0.001 |
| Headache | 1:4.02 | 4169.4 | < 0.001 |
Ratio of suspected adverse reactions following vaccination with PF and AZ vaccines, and results of χ2 testing null hypothesis of no effect of vaccine type on frequency of suspected adverse reactions.
Again, this shows highly significant differences between the vaccines in the rate of suspected less serious adverse events occurring shortly after vaccination. Such analyses should be useful for making recommendations on vaccine choice to minimise the incidence of less serious suspected adverse reactions in the vaccination programme.
We can conclude that the Yellow Card reporting scheme can provide some limited information that may be useful for alerting the UK public to possible adverse effects of the COVID-19 vaccines. However, the initial conception of the scheme as a purely descriptive rather than as an experimental undertaking means that it cannot address the real issues that are of crucial importance to the UK public. These issues are whether there are causal relationships between vaccination with the PF and AZ vaccines and serious adverse effects such as death, and if so, what are the size of these effects. To address these issues an experimental protocol such as that outlined earlier in this article is required, and should be implemented with immediate effect. Without data generated by such an approach it will not be possible to establish scientifically sound policy for COVID-19 vaccinations.
The author is a scientist who has been involved in university teaching and research in biology and scientific enquiry for 35 years.
1 Regulatory approval of Pfizer/BioNTech vaccine for COVID-19
2 Regulatory approval of COVID-19 Vaccine AstraZeneca
3 Report of the Commission on Human Medicines Expert Working Group on COVID-19 vaccine safety surveillance
4 Arvin, A.M. et al. (2020). A perspective on potential antibody dependent enhancement of SARS-CoV-2. Nature 584, 353-363.
5 Halstead S.B., Katzelnick, L. (2020). COVID-19 Vaccines: Should We Fear ADE? Journal of Infectious Diseases 222. DOI: 10.1093/infdis/jiaa518
6 Mulligan, M.J. et al. (2020) Phase I/II study of COVID-19 RNA vaccine BNT162b1 in adults. Nature
7 Coronavirus vaccine – weekly summary of Yellow Card reporting Updated March 4th 2021
8 Covid-19: First UK vaccine safety data are “reassuring,” says regulator BMJ 2021;372:n363
Donate
We depend on your donations to keep this site going. Please give what you can.
Donate TodayComment on this Article
You’ll need to set up an account to comment if you don’t already have one. We ask for a minimum donation of £5 if you'd like to make a comment or post in our Forums.
Sign UpFact Check: “Rishi Sunak Was the Main Person Responsible for Covid’s Second Wave”
Next Post‘Back to Normal’ Supports Anti-Lockdown Candidates in May Elections