MHRA Admits That it Missed Covid Vaccine Safety Signals

I’m not one prone to hyperbole but MHRA’s latest response to one of my freedom of information (FOI) requests seems to me to be a straightforward admission that it missed Covid vaccine safety signals up to mid 2022 (not to mention safety signals with other vaccines).

Last year, whilst trawling through minutes of meetings of the Commission on Human Medicines (here and specifically here), I discovered that MHRA had been concerned about the large numbers of Covid vaccine Yellow Card reports swamping the database and distorting the statistical analysis it relies on to detect safety signals. MHRA was concerned enough to conduct an investigation, write a report and consult the Commission on Human Medicines about it in June 2022. So I submitted an FOI last December asking for a copy of MHRA’s report. This was all mentioned in the Perseus Report of which I was a co-author.

Nearly four months after my FOI, and after two complaints to the ICO (here and here) including the threat of legal action, MHRA eventually sent me its report. It’s quite technical but it makes very interesting reading.

First, though, a bit of background. I expect this will come as a shock, but MHRA’s safety monitoring of medicines doesn’t rely on investigating individual Yellow Card reports and assessing causality (more on that at the end). Instead, it relies on statistical analysis called ‘disproportionality analysis’. This involves mining the database of Yellow Card reports looking for statistically significant differences in the frequencies of different types of side-effect between the drug of interest and other drugs. All medicine regulators use disproportionality analysis but there are differences in the algorithms (ROR, PRR, EBGM etc.) and the signal detection thresholds they use.

As its report says, MHRA was concerned about the effect on its own disproportionality analysis due to the very large proportion of COVID-19 vaccine reports in its Yellow Card database (by June 2022 it was over 80% of all vaccine-related Yellow Cards). Specifically, they were concerned that:

“With the majority of the vaccine dataset now comprised of reports for COVID-19 vaccines, these have the potential to unduly influence the disproportionality statistics for other vaccines.”
“If the safety/reporting profile for the COVID-19 vaccines differs significantly from other vaccines then this will impact disproportionality statistics and either mask potential signals or result in more false positive signals.”
“Additionally, there are potential issues with the large volume of COVID-19 vaccine reports impacting the disproportionality analyses for the COVID-19 vaccines themselves.”

The report concluded that the high number of Covid vaccine Yellow Cards was indeed suppressing signal detection for other vaccines. So the MHRA decided to remove Covid vaccine Yellow Cards from disproportionality analysis of other vaccines.

For disproportionality analysis of Covid vaccine Yellow Cards, the investigation led MHRA to decide to assess Covid vaccines against other drugs (rather than other vaccines) and ditch the use of EGBM/EB05 based on RRR (relative risk ratio, used since 2006) and revert to just using the simpler PRR (proportional risk ratio) technique.

So what, I hear you say. It’s somewhat reassuring that MHRA spotted a potential problem with its safety signal detection methods and acted on it. But it’s the opposite of reassuring, for at least three reasons. First, it is evidence that potential safety signals for the Covid vaccine (and other vaccines) were being missed as the number of Covid vaccine Yellow Cards quickly mounted. Secondly, the Commission on Human Medicines seems to have just meekly accepted MHRA’s report and recommendations (here) without asking any probing questions – such as what about the missed signals, can signals still be missed, what are other regulators doing? Thirdly, it should have been (but wasn’t) a reminder to MHRA that it should also actively follow-up individual fatal and serious Yellow Card reports and not rely so heavily on statistical analysis.

On that last point, MHRA said here that “the MHRA carefully evaluates reports of serious suspected side-effects as soon as they are received to consider whether the medicine or vaccine may have caused the event, or whether the event was likely to be purely coincidental”. However, I can’t see how that can possibly be true because it has also said that it:

has no process for investigating individual Yellow Card reports;
doesn’t know how many its has investigated;
doesn’t know for all fatal and serious Yellow Card reports important things like the age of the subject, his or her medical history or the batch number of the Covid vaccine.

So how confident are you feeling about whether it does, indeed, “consider whether the medicine or vaccine may have caused the event, or whether the event was likely to be purely coincidental”?

To check this out, I’ve submitted an FOI to find out how many Covid vaccine Yellow Card reports of suspected serious adverse events MHRA has assessed as i) not related, ii) unlikely related, iii) possibly related, iv) probably related and v) definitely related. Those are the categories used for describing causality of adverse events in clinical trials, so it makes perfect sense it would use the same ones for assessing causality of spontaneous reports to the Yellow Card system – if indeed it actually does it at all. Watch this space.

Until Nick retired a few years ago, he was a Senior Civil Servant in the Ministry of Defence responsible for the safety and effectiveness of ammunition used by the Armed Forces. He is co-author of the Perseus Group report on U.K. drugs regulator the MHRA.