The FDA gets aggressive with Covid-19 antibody tests

Chembio’s antibody test is the first to have its authorisation revoked. It might not be the last.

The US FDA’s rescinding of emergency use authorisation for Chembio’s Covid-19 antibody test on grounds of inaccuracy caused the company’s stock to plunge more than 60% yesterday. It might also have prompted the makers of other antibody tests with EUAs to wonder whether the agency's action set a precedent. 

The regulator took the action after performing its own investigation into the accuracy of the DPP Covid-19 IgM/IgG system, as Chembio’s test is called. And it is doing the same for the other assays it has authorised. Three so far have been given the all-clear, which leaves 13 authorised antibody tests still potentially vulnerable to being double-checked and found wanting.

The FDA began granting emergency market passes to antibody tests at the start of April, based on accuracy data submitted by the manufacturers; the agency checked this over but did not run separate evaluations of its own. 

In mid-April the agency said it would start performing independent validations in concert with other US government agencies, including the National Cancer Institute. This has been a slow process, however, with reports on only four commercially produced antibody tests with EUAs having so far been made available. Chembio’s is the first to have run into trouble. 

Crunching the numbers

The data Chembio submitted for its test claimed sensitivity of 93.5%, based on correctly identifying 29 of 31 positive samples, and specificity of 94.4%, getting 118 of 125 negative samples right. (These values are for the combined IgM and IgG antibodies; separate values were also given.) Data from an evaluation sponsored by the NCI and performed at the Frederick National Laboratory for Cancer Research found rather different numbers, summarised below. 

Accuracy data on Chembio's antibody test
  Initially submitted by Chembio NCI evaluation
IgM sensitivity 77.4% 57.1%
IgM specificity - 86.2%
IgG sensitivity 87.1% 78.6%
IgG specificity - 91.2%
Combined sensitivity 93.5% 82.1%
Combined specificity 94.4% 81.2%
Combined PPV 46.8% 18.7%
Combined NPV 99.6% 98.9%
PPV and NPV at 5% prevalence. Source: FDA.

In its revocation letter to Chembio the FDA wrote that it was "not reasonable to believe that the test may be effective in detecting antibodies against Sars-CoV-2”, or that its benefits outweighed its risks, including the high rate of false results. 

The FDA also stated in the letter that its current thinking on antibody tests was that they should have a minimum combined sensitivity of 90% and a minimum specificity of 95%. For tests that report specifically IgM and IgG, minimum sensitivity values of 90% and 70% for IgG and IgM respectively are called for. There are also other conditions including the number of samples on which these figures should be calculated. 

So how do the other antibody tests stack up? Of the four assays that have been independently checked the most interesting is Helgen, which the FDA decided was good enough to stay on the market despite a relatively low positive predictive value – the probability that subjects with a positive test result truly have Covid-19 antibodies – of 68%.

Independently evaluated Covid-19 antibody tests with EUAs
Company Test Sensitivity Specificity PPV NPV
Chembio Diagnostic DPP Covid-19 IgM/IgG 82.1% 81.2% 18.7% 98.9%
Euroimmun (Perkinelmer) Anti-Sars-CoV-2 Elisa IgG 90.0% 100% 100% 99.5%
Hangzhou Biotest Biotech RightSign Covid-19 IgG/IgM  100% 100% 100% 100%
Healgen Covid-19 IgG/IgM 100% 97.5% 67.8% 100%
Commercial tests only. PPV & NPV = positive & negative predictive values. PPV and NPV calculated at 5% prevalence. Source: FDA.

It is important to note that the tables above and below are just summaries – they do not include confidence intervals, for instance, or the sizes of the samples on which accuracy values were calculated. When a product tests separately for both IgG and IgM, combined accuracy figures are given. The FDA’s own page on antibody test performance has a fuller picture, including accuracy data for the separate antibodies where appropriate.

As for the 13 commercial tests that have received EUA but have not yet been independently validated, all look good enough to meet the FDA’s minimum standards, though the positive predictive value of Cellex’s test is markedly low at 55%. And no matter how decent these figures look, NCI scientists might come up with different results.

Covid-19 antibody tests with EUAs yet to be independently evaluated
Company Test Sensitivity Specificity PPV NPV
Abbott Alinity i Sars-CoV-2 IgG 100% 99.0% 84.0% 100%
Abbott Architect Sars-CoV-2 IgG 100% 99.6% 92.9% 100%
Autobio Anti-Sars-CoV-2 99.0% 99.0% 84.4% 99.9%
Bio-Rad  Platelia Sars-CoV-2 total Ab 92.2% 99.6% 91.7% 99.6%
Cellex qSars-CoV-2 IgG/IgM 93.8% 96.0% 55.2% 99.7%
Diasorin Liaison Sars-CoV-2 S1/S2 IgG 97.6% 99.3% 88.0% 99.9%
Inbios SCoV-2 Detect IgG Elisa 97.8% 99.0% 83.1% 99.9%
Ortho-Clinical Diagnostics Vitros anti-Sars-CoV-2 IgG 90.0% 100% 100% 99.5%
Ortho-Clinical Diagnostics Vitros anti-Sars-CoV-2 Total 100% 100% 100% 100%
Roche Elecsys anti-Sars-CoV-2 100% 99.8% 96.5% 100%
Siemens Healthineers Advia Centaur COV2T 100% 99.8% 96.5% 100%
Siemens Healthineers Atellica COV2T 100% 99.8% 96.7% 100%
Vibrant America Vibrant Covid-19 Ab 98.1% 98.6% 78.7% 99.9%
Commercial tests only. PPV & NPV = positive & negative predictive values. PPV and NPV calculated at 5% prevalence. Source: FDA.

Share This Article