Abstract
We report evaluation of 30 assays’ (17 rapid tests (RDTs) and 13 automated/manual ELISA/CLIA assay (IAs)) clinical performances with 2594 sera collected from symptomatic patients with positive SARS-CoV-2 rRT-PCR on a respiratory sample, and 1996 pre-epidemic serum samples expected to be negative. Only 4 RDT and 3 IAs fitted both specificity (> 98%) and sensitivity (> 90%) criteria according to French recommendations. Serology may offer valuable information during COVID-19 pandemic, but inconsistent performances observed among the 30 commercial assays evaluated, which underlines the importance of independent evaluation before clinical implementation.
Similar content being viewed by others
Introduction
SARS-CoV-2 causing the new coronavirus disease (COVID-19) has now spread worldwide [1, 2]. SARS-CoV-2 infection may be asymptomatic or can cause respiratory symptoms and other serious complications [3,4,5]. Identification of individuals showing a serological response to SARS-CoV-2 provides important complementary information by giving an evaluation of the fraction of individuals who have previously been infected [6, 7].
Many serological assays are currently available and choosing the best assay may be very challenging for laboratories. Our aim in this article is to report independent evaluations in order to provide an overview of 30 serological assays’ clinical performances in symptomatic patients.
Methods
Thirteen laboratories of virology of AP-HP, located all over Paris region, were involved in practice of virological diagnosis of COVID-19 and in evaluation of diagnostic assays (Supplementary Table 1).
Patients and sera
Between March and May 2020, 2594 sera were collected from symptomatic adults (not immunocompromised) previously diagnosed with COVID-19 by rRT-PCR on a respiratory sample [8]. These patients were attending COVID-19-specific consultations, hospitalization, or emergency units of Paris public hospitals. Symptoms were either severe or mild, but none of the samples were collected from asymptomatic patients as these patients were not referred to hospitals.
We stratified our analysis in 3 periods depending on the time interval between onset of symptoms and serum collection:
-
0–9 days after onset of symptoms (N = 581/2594 (22.4%));
-
10–14 days after onset of symptoms (N = 581/2594 (22.4%));
-
14 days after onset of symptoms (median 22 days) (N = 1432/2594 (55.2%)).
A total of 1996 serum samples expected to be negative for SARS-CoV-2, as collected before the COVID-19 outbreak in France, were also tested to assess specificity. This panel included 665/1996 (33.3%) “potentially interfering sera” collected from patients with acute or chronic viral, bacterial, or malaria infections. Others were named “unselected pre-pandemic sera”.
Samples were not shared from one laboratory to another. Samples were stored at −20 °C until testing and within the same freeze/thaw cycle if tested by multiple methods.
Ethics
This work was a retrospective non-interventional study. Reclassification of biological remnants into research material was approved by the Institutional Review Board of all the Assistance-Publique-Hôpitaux-de-Paris University Hospitals participating to the study. According to the French Public Health Code (CSPArtL.1121-1.1), such protocols are exempted from individual informed consent due to the retrospective chart review design and absence of identifying images or personal/clinical details that could compromise anonymity.
Rapid tests for qualitative detection of anti-SARS-CoV-2 antibodies (RDTs)
A total of 17 qualitative membrane-based immunoassay (CE-IVD approved) were performed according to manufacturer’s instructions (Supplementary Table 1). All rely on immunochromatography lateral flow assay technology and interpreted via visual inspection, except Finecare assay which uses fluorescent detection conjugate with dedicated reader. For analysis, a test was considered positive regardless the intensity of the band.
Automated and manual ELISA/CLIA assays
A total of 13 immunoassays (CE-IVD approved) were performed according to manufacturer’s instructions (Supplementary Table 1). For analysis, all equivocal results were considered as positive.
Statistics
Antibody response was assessed in stratified analysis considering the time interval between the onset of symptoms and the date of sample collection. Sensitivity and specificity of each assay were calculated with their respective 95% confidence interval (95% CI). We compared qualitative serology results in different contexts with chi-squared Pearson tests (considered significant if p < 0.05).
Results
Each assay was evaluated with 50 to 1364 different sera (1571 (60.6%) samples were tested with both RDTs and IAs). Concerning specificity, false positive results were more frequent with “potentially interfering samples” (13.4%) compared with unselected pre-endemic sera (4.4%) (p < 0.001) (Table 1). Results for specificity and sensitivity for each assay are shown in Fig. 1 and Supplemental Table 2a for global results (IgG + IgM/IgA or TAb), Fig. 2 and Supplemental Table 3a for IgG results, and Fig. 3 and Supplemental Table 4a for IgM/IgA results.
Results of rapid tests for qualitative detection of anti-SARS-CoV-2 antibodies (RDTs)
RDTs achieved between 77.4 and 100.0% TAb specificity, with only 6 RDTs fitting the > 98% French recommendations [9]. By 15 days after onset of symptoms, most RDT (12/17) reached the expected sensitivity > 90% [9]. Only 4 RDTs fitted both sensitivity and specificity criteria: Finecare, NADAL, AAZ, and Orientgene.
Results of automated and manual ELISA/CLIA assays
Considering TAb or IgG (for assays detecting IgG only), assays achieved between 58.8 and 100.0% specificity, with only 5/13 IAs fitting the French recommendations (> 98%) [9]. By 15 days after onset of symptoms, 9/13 IAs reached the expected sensitivity > 90% [9]. Only 3 IAs fitted both sensitivity and specificity criteria: Biorad (TAb), Elecsys anti-SARS-CoV-2 (TAb), and Abbott Architect (IgG).
Serology results depending on disease severity and age of patients
For 783/2494 (31.4%) patients (samples collected between day 0 and 91st day after onset of symptoms), information on the necessity of hospitalization was available: 318/783 (40.6%) required hospitalization and 465/783 (59.4%) were released from emergency unit or consultation for mild disease. For these patients, results of IgG + IgM or TAb are reported in Table 2 (a serum was considered positive if at least one assay was positive). More than 14 days after onset of symptoms, 84.8% non-hospitalized patients had positive serology, compared with 95.5% of hospitalized patients (p < 0.05).
Concerning the age of patients, difference is significant for serology performed more than 14 days after onset of symptoms: 94.0% elder patients (> 50 years old) had positive serology, compared with 86.5% younger patients (< 50 years old) (p < 0.05) (Supplemental Table 5).
Discussion
Even if several assays have the minimum expected specificity of 98%, confidence intervals should not be overlooked, as several of them are quite large. Our RDT results suggest that IgG detection is more specific for SARS-CoV-2 infection (33/1996; 1.7%), whereas IgM from other infections or patient background may introduce specificity concerns (54/1996; 2.7%). Consequently, we suggest any RDT positive for IgM only to be as soon as possible investigated with nasopharyngeal rRT-PCR and/or subsequent serological sampling looking for IgG seroconversion. Previous works evaluating a panel of many serologic assays reported a range from 95 to 100% for sensitivity and specificity [10,11,12,13,14,15,16,17]. Concerning specificity, we chose an important number of “potentially interfering sera” compared with these studies, which usually use samples collected from healthy blood donors. Our aim was to identify potential cross-reactivities, and indeed, false positive results, both for RDTs and IAs, were more frequent with “potentially interfering samples” compared with unselected pre-endemic sera. This certainly explains why specificity reported in our study is lower than in previous publications, and it would certainly have been improved if only serum collected from healthy individuals had been used.
The main limitation of serology resides in the fact that sensitivity is low before 10 days after onset of symptoms. Indeed, in our evaluation, only 6/17 RDTs and 6/13 IAs reached 90% sensitivity 10–14 days after onset of symptoms. However, close to what was previously reported, in our study, most assays achieved sensitivity higher than 90%, both for RDTs and IAs, for sera collected more than 15 days after onset of symptoms [10,11,12,13,14,15,16,17]. Even if some current reports show that neutralizing antibodies decline in convalescent individuals in 2 to 5 months after infection, identifying patients with an history of SARS-CoV-2 infection might be particularly interesting to spare vaccine doses as it was recently described those patients presented high titers of neutralizing antibodies after receiving only one vaccine dose, with titers similar or higher than uninfected individuals that received two vaccine doses [18,19,20,21] (https://doi.org/10.1101/2021.01.29.21250653).
Our study presented some limitations. Selection of sera was based on samples collected from symptomatic patients having positive rRT-PCR on upper respiratory tract specimens. More studies are needed to address whether asymptomatic patients, or patients with chest imaging compatible with COVID-19 but negative rRT-PCR, have different antibody response that could influence assays’ performances. RDT, which intend to be used as point of care devices and therefore with capillary blood, were only evaluated with serum samples in our study. Again, additional investigations are needed to provide information on which of these assays are reliable enough to be used in clinical practice. Despite the overall large number of samples, several assays were underpowered for true sensitivity and specificity assessment and the large confidence intervals may not reflect assay performance as much as they reflect sample size in this study. As our evaluation was implemented in the first weeks of French outbreak, most samples were collected during the acute phase of the illness while long-term follow-up and late collection of samples will soon allow assessment of the long-term persistence of specific antibodies.
Overall, our findings provide reassurance that several RDTs/IAs are suitable to detect specific antibodies against SARS-CoV-2 with high levels of sensitivity and specificity more than 14 days after onset of symptoms, but also highlight that assays should be assessed before implementation to ensure analytical capabilities are as needed for the clinical purpose.
Availability of data and material
Raw data provided by each laboratory will be made available on reasonable request.
References
World Health Organization (WHO). Outbreak Investigation [Internet]. [cited 2020 Mar 13]. Available from: https://www.who.int/hac/techguidance/training/outbreakinvestigation_en.pdf
(WHO) WHO. Novel coronavirus – China [Internet]. [cited 2020 Jan 19]. Available from: http://www.who.int/csr/don/12-january-2020-novel-coronavirus-china/en/
Huang C, Wang Y, Li X, Ren L, Zhao J, Hu Y et al (2020) Clinical features of patients infected with 2019 novel coronavirus in Wuhan. China. Lancet 395(10223):497–506
Guan W, Ni Z, Hu Y, Liang W, Ou C, He J et al (2020) Clinical characteristics of coronavirus disease 2019 in China. N Engl J Med 382(18):1708–20
Grasselli G, Zangrillo A, Zanella A, Antonelli M, Cabrini L, Castelli A et al (2020) Baseline characteristics and outcomes of 1591 patients infected with SARS-CoV-2 admitted to ICUs of the Lombardy Region, Italy. JAMA - J Am Med Assoc 323(16):1574–81
Tang YW, Schmitz JE, Persing DH, Stratton CW (2020) Laboratory diagnosis of COVID-19: current issues and challenges. J Clin Microbiol 58(6):e00512-20
Zhang W, Du RH, Li B, Zheng XS, Yang XL, Hu B et al (2020) Molecular and serological investigation of 2019-nCoV infected patients: implication of multiple shedding routes. Emerg Microbes Infect 9(1):386–9
Corman VM, Landt O, Kaiser M, Molenkamp R, Meijer A, Chu DKW et al (2020) Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR. Eurosurveillance 25(3):2000045
Haute Autorité de santé (HAS) (2020) Place des tests sérologiques dans la stratégie de prise en charge de la maladie COVID-19, pp 1–36
Huang A, Garcia-Carreras B, Hitchings M, Yang B, Katzelnick L, Rattigan S et al (2020) A systematic review of antibody mediated immunity to coronaviruses: antibody kinetics, correlates of protection, and association of antibody responses with severity of disease. medRxiv Prepr Serv Heal Sci. https://doi.org/10.1101/2020.04.14.20065771
Whitman JD, Hiatt J, Mowery CT, Shy BR, Yu R, Yamamoto TN et al (2020) Evaluation of SARS-CoV-2 serology assays reveals a range of test performance. Nat Biotechnol 38(10):1174–83
Montesinos I, Gruson D, Kabamba B, Dahma H, Van den Wijngaert S, Reza S et al (2020) Evaluation of two automated and three rapid lateral flow immunoassays for the detection of anti-SARS-CoV-2 antibodies. J Clin Virol 128:104413
Meyer B, Torriani G, Yerly S, Mazza L, Calame A, Arm-Vernez I et al (2020) Validation of a commercially available SARS-CoV-2 serological immunoassay. Clin Microbiol Infect 26(10):1386–94
Grzelak L, Temmam S, Planchais C, Demeret C, Tondeur L, Huon C et al (2020) A comparison of four serological assays for detecting anti–SARS-CoV-2 antibodies in human serum samples from different populations. Sci Transl Med 12(559)
GeurtsvanKessel CH, Okba NMA, Igloi Z, Bogers S, Embregts CWE, Laksono BM et al (2020) An evaluation of COVID-19 serological assays informs future diagnostics and exposure assessment. Nat Commun 11(1):3436
Beavis KG, Matushek SM, Abeleda APF, Bethel C, Hunt C, Gillen S et al (2020) Evaluation of the EUROIMMUN Anti-SARS-CoV-2 ELISA Assay for detection of IgA and IgG antibodies. J Clin Virol 129:104468
Ainsworth M, Andersson M, Auckland K, Baillie JK, Barnes E, Beer S et al (2020) Performance characteristics of five immunoassays for SARS-CoV-2: a head-to-head benchmark comparison. Lancet Infect Dis 20(12):1390–400
Prompetchara E, Ketloy C, Palaga T (2020) Immune responses in COVID-19 and potential vaccines: lessons learned from SARS and MERS epidemic. Asian Pac J Allergy Immunol 38(1):1–9
Okba NMA, Müller MA, Li W, Wang C, Geurtsvankessel CH, Corman VM et al (2020) Severe acute respiratory syndrome coronavirus 2-specific antibody responses in coronavirus disease patients. Emerg Infect Dis 26(7):1478–88
Wajnberg A, Amanat F, Firpo A, Altman DR, Bailey MJ, Mansour M et al (2020) Robust neutralizing antibodies to SARS-CoV-2 infection persist for months. Science (80- ) 370(6521):1227–30
Bruni M, Cecatiello V, Diaz-Basabe A, Lattanzi G, Mileti E, Monzani S et al (2020) Persistence of anti-SARS-CoV-2 antibodies in non-hospitalized COVID-19 convalescent health care workers. J Clin Med 9(10):3188
Funding
L’Agence Nationale de Recherche sur le SIDA et les Hépatites Virales (ANRS, AC43) and Assistance Publique-Hôpitaux de Paris (AP-HP) allowed to buy several kits. Both also pay technicians involved in testing.
Wondfo Biotech Co, NG Biotech SA, nal von Minden, Biosynex, AAZ, Surgentech, Biogyne, Vedal Lab, BIOHIT, SD Biosensor, Abbott, DIA-PRO, Wantai, Vircell, Bio-Rad and BGI-GBI provided part of the kits free of charge.
Author information
Authors and Affiliations
Contributions
Christelle Vauloup-Fellous, Sarah Maylin, and Claire Périllaud-Dubois analyzed the global data and drafted the manuscript.
Anne-Geneviève Marcelin and Diane Descamps critically revised the manuscript.
All authors (Christelle Vauloup-Fellous, Sarah Maylin, Claire Périllaud-Dubois, Ségolène Brichler, Chakib Alloui, Emmanuel Gordien, Marie-Anne Rameix-Welti, Elyanne Gault, Frédérique Moreau, Slim Fourati, Dominique Challine, Jean-Michel Pawlotsky, Nadhira Houhou-Fidouh, Florence Damond, Vincent Mackiewicz, Charlotte Charpentier, Jean-François Méritet, Flore Rozenberg, Isabelle Podglajen, Stéphane Marot, Heloïse Petit, Sonia Burrel, Sepideh Akhavan, Marianne Leruez-Ville, Véronique Avettand-Fenoel, Jacques Fourgeaud, Tiffany Guilleminot, Elise Gardiennet, Stéphane Bonacorsi, Agnès Carol, Guislaine Carcelain, Juliette Villemonteix,Narjis Boukli, Joël Gozlan, Laurence Morand-Joubert, Jérome Legoff, Constance Delaugerre, Marie-Laure Chaix, Ana-Maria Roque-Afonso, Laurent Dortet, Thierry Naas, Jean-Baptiste Ronat, Samuel Lepape, Anne-Geneviève Marcelin, Diane Descamps) either performed the assays or analyzed the data of their own laboratory. All authors have approved the final version of the manuscript.
Corresponding author
Ethics declarations
Conflict of interest
JM Pawlotsky has served as an advisor and/or a speaker for Abbvie, Gilead, GlaxoSmithKline, Merck, Regulus and Siemens Healthcare
C Vauloup-Fellous served as an expert (rubella and CMV serology) for Abbott Diagnostics, Roche Diagnostics, Siemens Healthcare, and DiaSorin, outside the submitted work.
Dr. Charpentier reports personal fees from Gilead Sciences, personal fees from ViiV Healthcare, from MSD, outside the submitted work.
Dr. Marcelin reports grants and personal fees from VIIV Healthcare, grants and personal fees from Gilead, grants and personal fees from MSD, outside the submitted work.
Pr Laurence Morand-Joubert has received honoraria for advisories or invited talks or conferences from Gilead Sciences, Merck Laboratories MSD, Janssen Pharmaceuticals, and ViiV Healthcare and grants from ANRS, outside the submitted work
Dr. Avettand-Fenoel reports grants and personal fees from Viiv healthcare, other from Janssen, other from Roche, outside the submitted work
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Vauloup-Fellous, C., Maylin, S., Périllaud-Dubois, C. et al. Performance of 30 commercial SARS-CoV-2 serology assays in testing symptomatic COVID-19 patients. Eur J Clin Microbiol Infect Dis 40, 2235–2241 (2021). https://doi.org/10.1007/s10096-021-04232-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10096-021-04232-3