Dr Clare Craig FRCPath
On 7th September Public Health England announced a change to testing criteria. The change is designed to address the problem of false positives, people that test positive but do not in fact have COVID-19. The consequences of these false positives stretch far beyond the inconvenience to an individual who has to self-isolate unnecessarily. Counting false positives when a disease is not very prevalent can result in misleading, fear-mongering news articles and mistaken health policies.
The impact of this change in testing criteria is yet to affect the case numbers being reported. The new criteria means some cases will need further assessment before being reported so those results will be delayed.
Until Public Health England published this guidance, the working assumption was that false positive test results are irrelevant for testing carried out for COVID-19. This assumption is not true. However, in times of an epidemic, a few false positives does not make a material difference to the reported numbers of infected cases. When 30% of results are positive then 1% being false positive would be immaterial. When only 2% of your results are positive then 1% being false positive is important. As other commentators have pointed out, currently the underlying prevalence of COVID-19 is low and false positives may account for a large number of the reported cases. When test results determine how many people can attend weddings or funerals or whether loved ones can be at the birth of their child or death of their parent, there needs to be a higher threshold of proof.
The Testing Process
The test uses a sophisticated machine to detect the Sars-CoV-2 (COVID-19) RNA using PCR. PCR is a way of testing for RNA. RNA is a virus’ equivalent of DNA, it is the RNA that allows the virus to replicate. For testing to work the RNA is first converted to DNA.
Because you need adequate levels of DNA to detect COVID-19, the DNA present in the sample is amplified multiple times. After every cycle of amplification the sample is tested for the presence of COVID 19. The smaller the quantity of original viral RNA, the more cycles it will take for a positive result.
However, there comes a point where if you repeat cycles too many times then samples without viral RNA will also show a positive result. This is one source of error that produces a false positive.
What change has been proposed?
The number of cycles a laboratory chooses to have confidence in will make a difference to the number of positive results. With every cycle the numbers of positives both false and true will increase. Laboratories have to balance running more cycles, to ensure they increase the RNA sufficiently to miss fewer cases, with not running too many cycles and creating false positives.
Rather than publish any positive result regardless of the number of cycles, the latest government publication requires that each laboratory set a threshold. Above this value positive cases will be retested with either the same sample or another sample.
The number of cycles is not actually specified in the publication. Instead each laboratory must determine their own. A beautiful French study demonstrated the relationship between the number of cycles and the chance that a sample will be from an infectious case. Above 30 cycles and the chances of a test being from an infectious case are only 50/50. Above 34 cycles they are all positive. Another laboratory may find a different cut off. Indeed, a Canadian study found no cases requiring more than 24 cycles were infectious.
Will this change help reduce false positives?
Yes, and no.
30 seems a reasonable number, especially when you consider that all patients requiring ventilation or who died, had samples detected with fewer than 30 cycles.
The proposal is to use repeat PCR testing of the same or a repeat sample to confirm positivity when more than 30 cycles were used. Is that enough? There should be an impact but it will not eliminate false positives altogether.
The causes of false positives are myriad. From other viruses, to contaminant human DNA as well as cross contamination between cases and residual RNA fragments in patients who have cleared the virus. The risk of these can never be completely mitigated. Changing the cycle threshold does not fully address the potential for contamination or sub-optimal test performance in general. So more work needs to be done than just setting an albeit sensible number of amplification cycles.
By addressing the cycle threshold, PHE will eliminate some false positives. The cases that needed more than 30 cycles will be examined further to decide which are real. This ought to include input from the doctors caring for those patients and a repeat PCR test is likely to be carried out too. The numbers will rise again once this additional data is available. We will have to wait and see how low the new baseline is.
That is not the end of the problem with false positives. Other false positive test results look like true positive test results. If this were not the case we would not mistake them for true positive results. And for some false positives the cause will still be there when a second confirmatory PCR is attempted. We desperately need a robust definition of a ‘COVID-19 case’ with criteria beyond a single positive PCR result.
As well as muddying the data, the current situation is detrimental in many ways. It leads to test and trace resources being stretched too thin on irrelevant cases. It prevents new and better tests being able to demonstrate their effectiveness. It will mean vaccine trials will not show their true impact.
How many cases are false positives?
The key question then is: what is the false positive rate for our COVID-19 testing?
Unfortunately, and like so much about COVID-19, we don’t know the rate of false positives.
But we can take a hypothesis at one extreme of the spectrum: that all results through July and August were false positives, and see if the data supports it. And both hospital COVID-19 data testing and random screening of the population support the hypothesis that most COVID-19 positive results in July and August are false positives.
The deaths reported as COVID-19 in hospitals in July and August share the same characteristics as ‘normal’ hospital deaths (pre-COVID-19) rather than characteristics of COVID-19 deaths during the height of the epidemic:
- During the peak of the epidemic we suffered a 6% mortality rate for COVID-19 patients (i.e. 6% of the people admitted to hospital died of COVID-19). This is four times higher than the average mortality rate for people admitted to hospital, no matter what the reason, 1.5% of whom die. For COVID-19 patients in July and August, the mortality rate is 1.5%. So either the virus is much less deadly, or the majority of the COVID-19 cases in hospitals in July and August were not COVID-19 and these false positive patients subsequently fared about as well as an average hospital patient.
- During the peak of the epidemic, men accounted for 60% of COVID-19 deaths; but the deaths in July and August that were reported as COVID-19 were 50% male.
- During the peak of the epidemic, COVID-19 deaths were skewed heavily towards very old patients; in July and August this has reverted towards a normal (pre-COVID-19) distribution.
Looking beyond the hospital data, the random screening selected households for COVID-19 shows 95% of positive cases come from lone cases in households. i.e. only 5% of households have more than 1 person testing positive for COVID-19. This figure of ‘household clustering’ used to be much higher during the height of the epidemic. This suggests either that COVID-19 is no longer very contagious (such that in 95% of households where someone has COVID-19, no one else is detected as having the disease) or the number of COVID-19 cases is being over-stated by false positive results.
We can therefore see that the question of false positives is not about small technical corrections. It has the potential for dramatic changes to the data with real world impact.
What would reduce False Positives further?
There is hope. We now have a category of uncertain results and we’ve agreed there should be further work before confirming these results as positives. This is the first step towards more accurate testing. I would argue that this should be done for any lab where the positive rate has fallen below 2%. At those levels whether a result is a false positive or a true positive is almost guess work.
Extraordinary claims require exceptional evidence. A child testing positive for COVID-19 is extraordinary as is a new case in an area that has not had cases for months. Where these positive results occur, we need further testing such as a positive viral culture (where live is demonstrated) or a chest CT scan.
Spending longer on clarifying and confirming uncertain results, including input from the doctors caring for the patient, would mean delays to accurate daily reports and reduced central control over reporting numbers. But a few days delay in reporting would be a price worth paying to get a more accurate picture.
Public Health England and the NHS have been incredibly transparent with their data and I applaud them for that. They have made a promising start by setting a cycle threshold, but it won’t eliminate false positives and more work needs to be done.