Covid-19 antibody tests face a very specific problem
With the prevalence of coronavirus infection running at about 5%, test manufacturers and regulators alike will have to guard against false positives.
Dozens of antibody tests for immunity to Covid-19 have been released onto the market around the world, so far with little regulatory oversight. This is set to change, with the FDA, CDC and NIH saying last week that they will work together to validate the accuracy of the tests on sale in the US.
But these agencies have not stated what level of accuracy they might consider sufficient, and by no means all manufacturers of these antibody blood tests have released data on their products’ sensitivity and specificity. And many of the tests for which data are available might simply not be good enough – earlier today Roche’s chief executive, Severin Schwan, said that some of the tests on the market “are not worth anything, or have very little use”.
According to data compiled by EvaluateMedTech, 29 commercial assays designed to detect antibodies to the novel coronavirus are on sale in the US, only three of which have been granted the FDA’s backing in the form of an emergency use authorisation. But none of these tests, even those with EUAs, have had their accuracy evaluated by the FDA or any other regulatory body.
Some manufacturers make their own claims for accuracy. Abbott says the IgG test it released in the US last week, when performed at least two weeks after a patient has first exhibited symptoms, has sensitivity of 100% and specificity of 99.5% (New Covid-19 test and a decent first quarter buoy Abbott, April 16, 2020). The antibody test sold in the US by Becton Dickinson, developed by BD’s partner Biomedomics, has sensitivity of 88.7% and specificity of 90.6%.
Roche intends to bring its own antibody assay out next month, so it will be interesting to see what claims that company makes for the product’s accuracy. Other companies known to be working on antibody, or serological, tests for Covid-19 include Siemens Heathineers, Danaher and Diasorin.
There are two major questions here. Firstly, are these claims to be trusted without independent verification? Abbott and BD are reputable companies, highly unlikely to make claims they cannot justify, but many of the other antibody tests on sale around the world are from little-known groups and laboratories that might not be so scrupulous.
Secondly, how accurate do these tests need to be? Until the advent of a Covid-19 vaccine, countries aiming to lift lockdown restrictions and begin to accelerate their economic activity are going to rely heavily on antibody testing to determine which individuals can safely return to work.
|Accuracy figures for selected Covid-19 antibody tests|
|Company||Device name||US status||EU status||Sensitivity (%)||Specificity (%)|
|Abbott Laboratories||Abbott Sars-CoV-2 IgG Test||Marketed||Filed||100.0||99.5|
|BioMedomics Covid-19 IgM/IgG Rapid Test||Marketed||Marketed||88.7||90.6|
|Cellex*||Cellex qSars-CoV-2 IgG/IgM Cassette Rapid Test||Marketed||Unclassified||93.8||95.6|
|Creative Diagnostics||Creative Diagnostics Sars-CoV-2 Antibody Elisa||Marketed||Marketed||94.5||100.0|
|CTK Biotech||OnSite Covid-19 IgG/IgM Rapid Test||Marketed||Unclassified||96.9||99.4|
|Epitope Diagnostics||EDI Novel Coronavirus Covid-19 IgG Elisa Kit||Marketed||Unclassified||100.0||100.0|
|Epitope Diagnostics||EDI Novel Coronavirus Covid-19 IgM Elisa Kit||Marketed||Unclassified||45.0||100.0|
|Intec Products||InTec Rapid Sars-CoV-2 Antibody (IgM/IgG)||Marketed||Marketed||95.2||98.0|
|Nirmidas Biotech||Nirmidas Biotech Covid-19 (Sars-CoV-2) IgM/IgG Antibody Detection Kit||Marketed||Unclassified||93.8||99.5|
|Ortho-Clinical Diagnostics*||Vitros Immunodiagnostic Product Anti-Sars-CoV-2 Total Reagent Pack||Marketed||Unclassified||83.3||100.0|
|SD Biosensor||Standard Q Covid-19 IgM/IgG Duo Test||Marketed||Unclassified||81.8||96.6|
|Note: All accuracy claims made by the companies. *Tests with FDA emergency use authorisation. Source: EvaluateMedTech & company websites.|
The table above summarises some of the accuracy figures that selected manufacturers claim for their serological Covid-19 tests. But the validation tests these companies have performed varied widely in size; Abbott’s antibody assay was tested on 1,200 specimens, whereas Epitope’s tests were run on 54 samples from healthy people and just 20 and 30 cases of PCR-confirmed Covid-19 for the IgM and IgG tests respectively. The usual concerns about comparing data from different studies are vastly magnified here.
And accuracy needs to be high. The prevalence of Covid-19 is estimated at around 5% in the US, and at this low a level the risk of false positives becomes a major problem. If a serological test has 90% specificity, its positive predictive value will be 32.1% – meaning nearly 70% of positive results will likely be false. At this same disease prevalence, a test with 95% specificity will lead to a 50% chance that a positive result is wrong. Only at 99% specificity does the false positive rate become anywhere near acceptable, and even here the chances are that 16% of positive results would be wrong.
|Predictive value of a theoretical test at disease prevalence of 5%|
|Sensitivity and specificity of 90%||NPV||99.4%||Probability that antibodies are present when the test is negative||0.6%|
|PPV||32.1%||Probability that antibodies are not present when the test is positive||67.9%|
|Sensitivity and specificity of 95%||NPV||99.7%||Probability that antibodies are present when the test is negative||0.3%|
|PPV||50.0%||Probability that antibodies are not present when the test is positive||50.0%|
|Sensitivity and specificity of 99%||NPV||99.9%||Probability that antibodies are present when the test is negative||0.1%|
|PPV||83.9%||Probability that antibodies are not present when the test is positive||16.1%|
|NPV & PPV = negative & positive predictive values. Source: Evaluate Vantage calculations.|
The FDA will need to set demanding accuracy standards for these products if it is to avoid large numbers of people with no immunity to the virus being sent back into contact with others. The agency has said it will take appropriate action against companies making or distributing unvalidated tests or those making false claims, such as issuing warning letters requesting that companies stop their unlawful promotion.
If a company can demonstrate decent, reproducible levels of accuracy, there is money to be made. Leerink analysts assume a price of $5-10 for strip-based assays similar to pregnancy tests – the kind being sold by Cellex and Chembio, for example – and $25-50 for high-throughput testing done at a central lab, such as that offered by Abbott and, next month, by Roche. If half the US population is tested, the market in that country will exceed $3bn.
Companies used to working at scale will obviously be best placed to gain market share. Abbott says it will be able to conduct 20 million tests per month by June, and Roche said that it will manage five times that.
There is one last headache for those who believe that serological testing is the means through which normality will resume. As Bernstein analysts point out, even a perfect test would still only allow the 5% of the population that had been infected back to work – hardly a respectable basis for kickstarting an economy.