Covid-19 antibody tests face a very specific problem

With the prevalence of coronavirus infection running at about 5%, test manufacturers and regulators alike will have to guard against false positives.

Spotlight

Dozens of antibody tests for immunity to Covid-19 have been released onto the market around the world, so far with little regulatory oversight. This is set to change, with the FDA, CDC and NIH saying last week that they will work together to validate the accuracy of the tests on sale in the US. 

But these agencies have not stated what level of accuracy they might consider sufficient, and by no means all manufacturers of these antibody blood tests have released data on their products’ sensitivity and specificity. And many of the tests for which data are available might simply not be good enough – earlier today Roche’s chief executive, Severin Schwan, said that some of the tests on the market “are not worth anything, or have very little use”. 

According to data compiled by EvaluateMedTech, 29 commercial assays designed to detect antibodies to the novel coronavirus are on sale in the US, only three of which have been granted the FDA’s backing in the form of an emergency use authorisation. But none of these tests, even those with EUAs, have had their accuracy evaluated by the FDA or any other regulatory body. 

Some manufacturers make their own claims for accuracy. Abbott says the IgG test it released in the US last week, when performed at least two weeks after a patient has first exhibited symptoms, has sensitivity of 100% and specificity of 99.5% (New Covid-19 test and a decent first quarter buoy Abbott, April 16, 2020). The antibody test sold in the US by Becton Dickinson, developed by BD’s partner Biomedomics, has sensitivity of 88.7% and specificity of 90.6%.

Roche intends to bring its own antibody assay out next month, so it will be interesting to see what claims that company makes for the product’s accuracy. Other companies known to be working on antibody, or serological, tests for Covid-19 include Siemens Heathineers, Danaher and Diasorin.

Major questions

There are two major questions here. Firstly, are these claims to be trusted without independent verification? Abbott and BD are reputable companies, highly unlikely to make claims they cannot justify, but many of the other antibody tests on sale around the world are from little-known groups and laboratories that might not be so scrupulous.

Secondly, how accurate do these tests need to be? Until the advent of a Covid-19 vaccine, countries aiming to lift lockdown restrictions and begin to accelerate their economic activity are going to rely heavily on antibody testing to determine which individuals can safely return to work. 

Accuracy figures for selected Covid-19 antibody tests 
Company Device name US status EU status Sensitivity (%) Specificity (%)
Abbott Laboratories Abbott Sars-CoV-2 IgG Test Marketed Filed 100.0 99.5
Becton Dickinson/
Biomedomics*
BioMedomics Covid-19 IgM/IgG Rapid Test Marketed Marketed 88.7 90.6
Cellex* Cellex qSars-CoV-2 IgG/IgM Cassette Rapid Test Marketed Unclassified 93.8 95.6
Creative Diagnostics Creative Diagnostics Sars-CoV-2 Antibody Elisa Marketed Marketed 94.5 100.0
CTK Biotech OnSite Covid-19 IgG/IgM Rapid Test Marketed Unclassified 96.9 99.4
Epitope Diagnostics EDI Novel Coronavirus Covid-19 IgG Elisa Kit Marketed Unclassified 100.0 100.0
Epitope Diagnostics EDI Novel Coronavirus Covid-19 IgM Elisa Kit Marketed Unclassified 45.0 100.0
Intec Products InTec Rapid Sars-CoV-2 Antibody (IgM/IgG) Marketed Marketed 95.2 98.0
Nirmidas Biotech Nirmidas Biotech Covid-19 (Sars-CoV-2) IgM/IgG Antibody Detection Kit Marketed Unclassified 93.8 99.5
Ortho-Clinical Diagnostics* Vitros Immunodiagnostic Product Anti-Sars-CoV-2 Total Reagent Pack Marketed Unclassified 83.3 100.0
SD Biosensor Standard Q Covid-19 IgM/IgG Duo Test Marketed Unclassified 81.8 96.6
Note: All accuracy claims made by the companies. *Tests with FDA emergency use authorisation. Source: EvaluateMedTech & company websites.

The table above summarises some of the accuracy figures that selected manufacturers claim for their serological Covid-19 tests. But the validation tests these companies have performed varied widely in size; Abbott’s antibody assay was tested on 1,200 specimens, whereas Epitope’s tests were run on 54 samples from healthy people and just 20 and 30 cases of PCR-confirmed Covid-19 for the IgM and IgG tests respectively. The usual concerns about comparing data from different studies are vastly magnified here. 

And accuracy needs to be high. The prevalence of Covid-19 is estimated at around 5% in the US, and at this low a level the risk of false positives becomes a major problem. If a serological test has 90% specificity, its positive predictive value will be 32.1% – meaning nearly 70% of positive results will likely be false. At this same disease prevalence, a test with 95% specificity will lead to a 50% chance that a positive result is wrong. Only at 99% specificity does the false positive rate become anywhere near acceptable, and even here the chances are that 16% of positive results would be wrong. 

Predictive value of a theoretical test at disease prevalence of 5%
Sensitivity and specificity of 90% NPV 99.4% Probability that antibodies are present when the test is negative 0.6%
  PPV 32.1% Probability that antibodies are not present when the test is positive 67.9%
         
Sensitivity and specificity of 95% NPV 99.7% Probability that antibodies are present when the test is negative  0.3%
  PPV 50.0% Probability that antibodies are not present when the test is positive 50.0%
         
Sensitivity and specificity of 99% NPV 99.9% Probability that antibodies are present when the test is negative 0.1%
  PPV 83.9% Probability that antibodies are not present when the test is positive 16.1%
NPV & PPV = negative & positive predictive values. Source: Evaluate Vantage calculations.

The FDA will need to set demanding accuracy standards for these products if it is to avoid large numbers of people with no immunity to the virus being sent back into contact with others. The agency has said it will take appropriate action against companies making or distributing unvalidated tests or those making false claims, such as issuing warning letters requesting that companies stop their unlawful promotion.

If a company can demonstrate decent, reproducible levels of accuracy, there is money to be made. Leerink analysts assume a price of $5-10 for strip-based assays similar to pregnancy tests – the kind being sold by Cellex and Chembio, for example – and $25-50 for high-throughput testing done at a central lab, such as that offered by Abbott and, next month, by Roche. If half the US population is tested, the market in that country will exceed $3bn. 

Companies used to working at scale will obviously be best placed to gain market share. Abbott says it will be able to conduct 20 million tests per month by June, and Roche said that it will manage five times that.

There is one last headache for those who believe that serological testing is the means through which normality will resume. As Bernstein analysts point out, even a perfect test would still only allow the 5% of the population that had been infected back to work – hardly a respectable basis for kickstarting an economy.

Related Topics

Share This Article