https://orcid.org/0000-0003-0376-0227
Figures
Abstract
We examined the usefulness of five COVID-19 antibody detection tests using 114 serum samples at various time points from 34 Japanese COVID-19 patients. We examined Elecsys Anti-SARS-CoV-2 from Roche, and four immunochromatography tests from Hangzhou Laihe Biotech, Artron Laboratories, Chil, and Nadal. In the first week after onset, Elecsys had 40% positivity in Group S (severe cases) but was negative in Group M (mild-moderate cases). The immunochromatography kits showed 40–60% and 0–8% positivity in Groups S and M, respectively. In the second week, Elecsys showed 75% and 50% positivity, and the immunochromatography tests showed 5–80% and 50–75% positivity in Groups S and M, respectively. After the third week, Elecsys showed 100% positivity in both groups. The immunochromatography kits showed 100% positivity in Group S. In Group M, positivity decreased to 50% for Chil and 75–89% for Artron and Lyher. Elecsys and immunochromatography kits had 91–100% specificity. Elecsys had comparable chronological change of cut-off index values in the two groups from the second week to the sixth week. The current SARS-CoV-2 antibody detection tests do not provide meaningful interpretation of severity and infection status. Its use might be limited to short-term epidemiological studies.
Citation: Wakita M, Idei M, Saito K, Horiuchi Y, Yamatani K, Ishikawa S, et al. (2021) Comparison of the clinical performance and usefulness of five SARS-CoV-2 antibody tests. PLoS ONE 16(2): e0246536. https://doi.org/10.1371/journal.pone.0246536
Editor: Pierre Roques, CEA, FRANCE
Received: December 31, 2020; Accepted: January 21, 2021; Published: February 8, 2021
Copyright: © 2021 Wakita et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data contains sensitive identifying information (confidential clinical information) and cannot be shared publicly due to ethical restrictions imposed by Dr. Atsushi Okuzawa, the Chair of Juntendo University Hospital Institutional Review Board. Data access requests can be sent to: Dr. Shigeki Aoki, the Chair of Ethical Committee at Juntendo University: E-mail: saoki@juntendo.ac.jp.
Funding: Reagents and assays for Elecsys Anti-SARS-CoV-2 were provided by Roche Diagnostics. LYHER Novel Coronavirus (2019-nCoV) IgM/IgG Antibody Combo Test, Artron COVID-19 IgM/IgG Antibody Test, CHIL COVID-19 IgG/IgM Rapid Test, and NADAL COVID-19 IgG/IgM test were provided by Hangzhou Laihe Biotech, Artron Laboratories, Chil, and nal von minden, respectively. All providers did not play any roles in the design of the study, analysis and interpretation of the data.
Competing interests: The authors have declared that no competing interests exist.
Introduction
The new coronavirus disease (COVID-19), caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), originated from Wuhan, China in late 2019 and spread worldwide. The World Health Organization (WHO) declared the pandemic on March 11, 2020. To control the pandemic, diagnostic tests such as reverse transcription polymerase chain reaction (RT-PCR) methods were developed [1]. The results of these RT-PCR tests were used for taking political decisions such as imposing lockdown in several countries [2, 3]. However, since RT-PCR tests are feasible only within three weeks since symptom onset, it is inconvenient for epidemiological investigations. To estimate past infection numbers, serological tests were developed (https://www.whitehouse.gov/wp-content/uploads/2020/05/Testing-Guidance.pdf). As of January 2021, more than 33 serological tests are commercially available as they were urgently approved by the United States Food and Drug Administration and European Medicines Agency. Importantly, more than 40 serological assays were not approved (https://open.fda.gov/apis/device/covid19serology/), which suggests that the performance of COVID-19 serological assays were not yet thoroughly investigated. In addition, the significance of serological tests remains unclear as the Center for Disease Control published interim guidelines for their use (https://www.cdc.gov/coronavirus/2019-ncov/lab/resources/antibody-tests-guidelines.html).
SARS-CoV-2, a single-stranded RNA virus belonging to the Orthocoronavirinae subfamily, consists of four structural components, namely, spike glycoprotein (S), envelope protein, membrane glycoprotein, and nucleocapsid phosphoprotein (N), and 16 non-structural proteins [4]. Thus, the accuracy and reliability of these tests rely upon the nucleotide fragments used to develop the antibody. In addition, viral types may differ across infections at different times. To date, at least 116 mutations including three common mutations have been identified [5], and the seroprevalence timing might differ by viral type.
This study aimed to investigate the sensitivity, specificity, and time course of seroprevalence in 34 Japanese COVID-19 patients using an electrochemiluminescence immunoassay (ECLIA)-based Elecsys Anti-SARS-CoV-2 (RUO, Roche Diagnostics) test and four different immunochromatographic (IC) point-of-care tests developed by Hangzhou Laihe Biotech, Artron Laboratories, Chil, and Nadal.
Material and methods
Clinical backgrounds
This study complied with all relevant national regulations and institutional policies and was conducted in accordance with the tenets of the Declaration of Helsinki. The study was approved by the Institutional Review Board (IRB) at Juntendo University Hospital (IRB # 20–036). The need for informed consent from individual patients was waived because all samples were de-identified in line with the Declaration of Helsinki.
Between March and June 2020, 114 serum samples were collected from 34 COVID-19 patients. Table 1 shows the clinical characteristics and timing of sample collection. All patients were confirmed to be positive according to PCR-based testing of SARS-CoV-2 using the Light Mix Modular SARS-CoV-2 (COVID-19) N-gene and E-gene assay (Roche Diagnostics, Tokyo, Japan) or the 2019 Novel Coronavirus Detection Kit (Shimadzu, Kyoto, Japan). We classified patients into two groups according to the WHO criteria: Group M that included mild and moderate cases and Group S that included severe and critical cases. For the negative control, 100 serum samples collected from outpatients without infectious diseases between November and December 2018 were used. The samples were stored at -80°C until use. All data were fully anonymized before access, and de-identified clinical information obtained between March and December 2020 were provided.
Antibody assays
We used the US Food and Drug Administration-approved Elecsys Anti-SARS-CoV-2 electrochemiluminescence immunoassay system (Roche Diagnostics, Basel, Switzerland), which is based on the modified double-antigen sandwich immunoassay with recombinant nucleocapsid protein (N) and measures SARS-CoV-2 total antibody (pan immunoglobulin) with a fully automated Cobas e801 analyzer (Roche Diagnostics) (https://www.accessdata.fda.gov/cdrh_docs/presentations/maf/maf3358-a001.pdf). According to the FDA, the Elecsys Anti-SARS-CoV-2 system has 100% sensitivity (≥14 days after a positive polymerase chain reaction [PCR] assay) and 99.8% specificity (https://www.fda.gov/medical-devices/coronavirus-disease-2019-covid-19-emergency-use-authorizations-medical-devices/eua-authorized-serology-test-performance). The results are reported as numeric values in the form of a cutoff index (COI; signal sample/cutoff) with qualitative results reactive (COI ≥ 1.0; positive). The analytical and clinical performance of the assay have been evaluated and are described elsewhere [6].
The following rapid immunochromatographic IgM/IgG antibody assays were utilized: LYHER Novel Coronavirus (2019-nCoV) IgM/IgG Antibody Combo Test (Hangzhou Laihe Biotech); Artron COVID-19 IgM/IgG Antibody Test (Artron Laboratories); CHIL COVID-19 IgG/IgM Rapid Test (Chil), and NADAL COVID-19 IgG/IgM test (nal von minden). The immunochromatographic IgM/IgG antibody assays target the receptor binding domain of S protein or the nucleocapsid protein, N protein (Table 2). The presence of only the control line indicated a negative result, whereas the presence of both the control line and the IgM or IgG antibody line indicated a positive result for IgM or IgG antibody, respectively. Table 2 summarizes the features of these kits.
Results
Table 3 shows the sensitivity or the rate of positivity of Elecsys and the four immunochromatography kits in a total of 114 serum samples from 34 patients. The results of the immunochromatography kits were considered as positive when IgM or IgG were positive (qualitative tests).
In the first week after onset, Elecsys had a 40% positivity in Group S but was negative in Group M. Additionally, the four immunochromatography kits had 40–60% and 0–8% positivity in the Groups S and M, respectively. In the second week, Elecsys showed 75% and 50% positivity in Groups S and M, respectively. The four immunochromatography kits had 63–88% and 25–75% positivity in Groups S and M, respectively. After the third week, Elecsys showed 100% positivity in both groups, except for the fifth week in Group S (90%). Except for Chil, the immunochromatography kits showed 100% positivity in Group S. In Group M, positivity gradually decreased to 50% for Chil (IgM and IgG) and 75–89% for Artron and Lyher. Elecsys and Nadal showed the most consistent positivity.
Specificity was evaluated using the samples collected before the COVID-19 era. Table 4 shows that the specificity of IgM was as low as 91% for Artron and 96% for Nadal. For IgG, all kits showed a specificity of >98%.
Chronological change of COI
Next, we examined the COI values at various time points after onset using Elecsys. Fig 1 shows that COI tended to increase over time. However, there was no significant difference between Groups M and S until the sixth week. In the seventh week, the COI was higher in Group S than in Group M.
Antibody Index for SARS-CoV-2 PCR-positive patient samples for the indicated weekly timeframes post-onset of symptoms. The data were presented as mean with interquartile ranges. Open bars indicate Group M, and gray bars indicate Group S. Note that none of Group M showed significant COI values in the first week. *p<0.05; NS, no significant difference.
To examine the chronological changes of COI in eight inpatients, the COI values were plotted against the timing of the tests (Fig 2). Table 5 summarizes the patients’ clinical background characteristics. Four patients (#1, 6, 7, and 8) required ventilation support, and unfortunately, all patients could not be rescued. Three patients, except patient #1, showed relatively low COIs. The COI of patient #1 reached 100 when the patient died at 52 days. In patient #6, the COI did not increase at 13 days. Importantly, none of the deceased patients showed high COI values on admission. The patients who survived (#2, 3, 4, and 5) received supplemental oxygen and supporting therapies and were eventually discharged. Three of these (#2, 3, and 4) showed relatively high COIs (around 40).
The cut-off index in eight severe patients were tested using Elecsys. The COI values were plotted as a function of days after onset. Closed symbols depict deceased cases, and open symbols depict survived cases.
Discussion
In this study, we evaluated the performance of five different SARS-CoV-2 antibody detection tests using 114 serum samples from 34 Japanese patients with COVID-19 in a Tokyo metropolitan area. Our study demonstrated several important findings. First, the seroprevalence was approximately 40–60% in severe cases and relatively low in mild cases in the first week. The seroprevalence increased to 60–80% in severe cases and 50–60% in mild cases in the second week. After the third week, the seroprevalence reached almost 100% in both groups. In mild cases, the seroprevalence decreased when tested with Artron and Chil kits (Table 3). Second, the specificity was not 100% for all tests using the samples collected before the COVID-19 era (Table 4). Third, the COI values using Elecsys did not differ significantly over time except for the seventh week (Fig 1). However, this might be the effect of one outlier (patient #1 in Table 5). In addition, the COI values obtained by Elecsys might not reflect disease severity (Fig 2).
It was reported that IgM and IgG could be detected in 20–30% of cases approximately 14 days after onset, and the positive rates reach 80–90% after 15 days [6]. Interestingly, it was reported that IgM and IgG increased almost simultaneously [7]. In currently available antibody detection kits, the antibodies were developed based on the S1 domain of the S protein or the N protein. The N proteins are essential for viral survival and expansion, while the S proteins are essential for binding to the host cell surface receptors [8]. Since the S proteins might be produced before the increase in the N proteins, the performance of antibody detection kits can depend upon the target protein of the antibody. This might explain why Chil and Artron kits showed early decline of antibody levels in mild cases. However, how these kits were designed are confidential. Another concern is the false-positive rate of Artron and Nadal kits. Since our negative control samples were collected before 2018, antibodies against SARS-CoV-2 did not exist in these samples. Speculative explanations are antibody purification issues, difference in the target fragments, and crossreaction with other coronaviruses including SARS and Middle East Respiratory Syndrome (MERS).
Currently, SARS-CoV-2 is detected using RT-PCR, and it is believed that SARS-CoV-2 nucleotides can be detected using RT-PCR several days after symptom onset; however, the sensitivity and specificity of this test are unclear [9]. After a certain time period (more than three weeks), the sensitivity of PCR tests declines, and antibody tests may detect antibodies developed against nucleotide fragments of SARS-CoV-2. Currently, except for supporting therapies, there is no available treatment option for COVID-19 despite several cases of experimental drug use in the past several months [10–14]. Moreover, the pattern of seroprevalence remains unclear. Although the sample number was small, the severe cases in our study did not show any meaningful COI changes using Elecsys (Fig 2). In addition, our recent study showed that seroprevalence in 4147 healthcare workers in our hospital was 0.34% [doi: 10.21203/rs.3.rs-96870/v1]. Since the prevalence of COVID-19 is largely dependent on the number of PCR tests in a given population [15], it is likely that the prevalence of COVID-19 has been underestimated. Therefore, this suggests that the antibodies detected by current methods might disappear within a short period of time after infection [17].
Although many companies continue releasing new tests, we could test only limited numbers of assays commercially available in Japan when the study was performed. However, studies published in December 2020 have reported varying results in newer tests. Using 36 samples obtained from RT-PCR confirmed COVID-19 patients, Sacristan et al. reported that the detection percentage of IgG antibodies were similar in StrongStep SARS-CoV-2 IgG/IgM kit and AllTest COV-19 IgG/IgM kit (83.3%and 80.6%, respectively). In contrast, the IgM detection rates were lower than the IgG detection rates, and different between the two tests (11.1% and 30.6%, respectively) [16]. The timing of the antibody tests was approximately 11 days after RT-PCR tests, which is similar to our results between the second and the third week in Group M. Nilsson et al. compared several assays using 98 samples collected at different time points [17]. The assays included: EUROIMMUN anti-SARS-CoV-2 IgG and IgA ELISAs (EUROIMMUN Medizinische Labordiagnostika AG, Lübeck, Germany); WANTAI SARS-CoV-2 IgM ELISA (Beijing Wantai Biological Pharmacy Enterprise, Beijing, China); Acro IgM/IgG Lateral Flow Test (LFT)(2019-nCoV IgG/IgM Rapid Test Cassette, Acro Biotech, Rancho Cucamonga, CA, USA); Livzon IgM/IgG LFT (Diagnostic Kit for IgM/IgG Antibody to Corona Virus, Zhuhai Livzon Diagnostics, Zhuhai, China); and CTK IgM/IgG LFT (OnSiteTM COVID-19 IgG/IgM Rapid Test, CTK Biotech, Poway, CA). According to their results, WANTAI ELISA and Acro LFT were more sensitive than others in detecting IgM antibodies in the first week after onset. However, the sample size was small, consisting of only three patients. WANTAI ELISA and CTK LFT showed higher positivity for IgM between 8 and 28 days, then declined after 28 days. For the IgG antibody detections, all tests showed low sensitivity in the first week. Acro LFT showed a positivity of 91–100% between 8 and 28 days, which was better than the other assays. The other tests showed 57–94% positivity between 8–28 days, then declined after 28 days. They also compared the positivity among the outpatients, hospitalized and ICU admitted patients. All tests tended to show higher positivity in the inpatients compared to the outpatients, which is consistent with our data (Table 3). However, the positivity varied depending upon the assay. In addition, a meta-analysis of 57 studies published in June 2020 reported the low sensitivity and high heterogeneity of the serological tests [18]. All these results indicate unreliability and difficulty in developing serological tests against SARS-CoV-2, a single strand RNA virus even with slower mutation rates than other RNA viruses [19, 20]. Furthermore, there are many confounding factors such as difference in methodology, antibody development, and uncertainty of pathogens.
This study has several limitations: (1) this is a single-center study with a relatively small number of patients; (2) since the target nucleotides to develop antibodies are not disclosed, data interpretation was incomplete; (3) since the follow-up time was limited to 42 days, we do not know the long-term detection rate; (4) finally, we do not know whether these antibodies act as neutral antibodies.
In conclusion, our data showed that the serological tests including one ECLIA test and four immunochromatography tests had poor sensitivity during the early phase of infection and therefore were unsuitable for diagnosis or screening. In addition, these tests cannot provide meaningful interpretation of infection status. Thus, the current use of these tests might be limited to short-term epidemiological studies unless newer and more reliable technologies are developed in the future.
Acknowledgments
The authors are grateful to the participants in this study. The authors also would like to thank Natsumi Itakura, Masayoshi Chonan, Koji Tsuchiya and Takaaki Kawakami for their technical supports, and Dr. Corina Rosales for critical reading manuscript. Finally, we thank to all medical staff who conducted their duties in the treatment of this pandemic.
References
- 1. Benzigar MR, Bhattacharjee R, Baharfar M, Liu G. Current methods for diagnosis of human coronaviruses: pros and cons. Anal Bioanal Chem. 2020. Epub 2020/11/22. pmid:33219449; PubMed Central PMCID: PMC7679240.
- 2. Mattern J, Vauloup-Fellous C, Zakaria H, Benachi A, Carrara J, Letourneau A, et al. Post lockdown COVID-19 seroprevalence and circulation at the time of delivery, France. PLoS One. 2020;15(10):e0240782. Epub 2020/10/16. pmid:33057392; PubMed Central PMCID: PMC7561105.
- 3. Pachetti M, Marini B, Giudici F, Benedetti F, Angeletti S, Ciccozzi M, et al. Impact of lockdown on Covid-19 case fatality rate and viral mutations spread in 7 countries in Europe and North America. J Transl Med. 2020;18(1):338. Epub 2020/09/04. pmid:32878627; PubMed Central PMCID: PMC7463225.
- 4. Satarker S, Nampoothiri M. Structural Proteins in Severe Acute Respiratory Syndrome Coronavirus-2. Arch Med Res. 2020;51(6):482–91. Epub 2020/06/05. pmid:32493627; PubMed Central PMCID: PMC7247499.
- 5. Khailany RA, Safdar M, Ozaslan M. Genomic characterization of a novel SARS-CoV-2. Gene Rep. 2020;19:100682. Epub 2020/04/18. pmid:32300673; PubMed Central PMCID: PMC7161481.
- 6. Zhao J, Yuan Q, Wang H, Liu W, Liao X, Su Y, et al. Antibody responses to SARS-CoV-2 in patients of novel coronavirus disease 2019. Clin Infect Dis. 2020. Epub 2020/03/30. pmid:32221519; PubMed Central PMCID: PMC7184337.
- 7. Mazzini L, Martinuzzi D, Hyseni I, Benincasa L, Molesti E, Casa E, et al. Comparative analyses of SARS-CoV-2 binding (IgG, IgM, IgA) and neutralizing antibodies from human serum samples. J Immunol Methods. 2020:112937. Epub 2020/12/01. pmid:33253698; PubMed Central PMCID: PMC7695554.
- 8. Kang S, Yang M, Hong Z, Zhang L, Huang Z, Chen X, et al. Crystal structure of SARS-CoV-2 nucleocapsid protein RNA binding domain reveals potential unique drug targeting sites. Acta Pharm Sin B. 2020;10(7):1228–38. Epub 2020/05/05. pmid:32363136; PubMed Central PMCID: PMC7194921.
- 9. Wang Y, Zhang L, Sang L, Ye F, Ruan S, Zhong B, et al. Kinetics of viral load and antibody response in relation to COVID-19 severity. J Clin Invest. 2020;130(10):5235–44. Epub 2020/07/08. pmid:32634129; PubMed Central PMCID: PMC7524490.
- 10. Cavalcanti AB, Zampieri FG, Rosa RG, Azevedo LCP, Veiga VC, Avezum A, et al. Hydroxychloroquine with or without Azithromycin in Mild-to-Moderate Covid-19. N Engl J Med. 2020;383(21):2041–52. Epub 2020/07/25. pmid:32706953; PubMed Central PMCID: PMC7397242.
- 11. McCartney M. COVID-19: has EBM been replaced by hype-based medicine? Drug Ther Bull. 2020;58(7):99–100. Epub 2020/05/27. pmid:32451323.
- 12. Farnsworth CW, Anderson NW. SARS-CoV-2 Serology: Much Hype, Little Data. Clin Chem. 2020;66(7):875–7. Epub 2020/04/29. pmid:32343775; PubMed Central PMCID: PMC7197624.
- 13. Alhumaid S, Mutair AA, Alawi ZA, Alhmeed N, Zaidi ARZ, Tobaiqy M. Efficacy and Safety of Lopinavir/Ritonavir for Treatment of COVID-19: A Systematic Review and Meta-Analysis. Trop Med Infect Dis. 2020;5(4). Epub 2020/12/03. pmid:33260553.
- 14. Emani VR, Goswami S, Nandanoor D, Emani SR, Reddy NK, Reddy R. Randomised controlled trials for COVID-19: evaluation of optimal randomisation methodologies-need for data validation of the completed trials and to improve ongoing and future randomised trial designs. Int J Antimicrob Agents. 2020:106222. Epub 2020/11/16. pmid:33189891; PubMed Central PMCID: PMC7659806.
- 15. Wu SL, Mertens AN, Crider YS, Nguyen A, Pokpongkiat NN, Djajadi S, et al. Substantial underestimation of SARS-CoV-2 infection in the United States. Nat Commun. 2020;11(1):4507. Epub 2020/09/11. pmid:32908126; PubMed Central PMCID: PMC7481226.
- 16. Sacristan MS, Collazos-Blanco A, Cintas MIZ, Garcia AS, de Villavicencio CY, Maestre MM. Comparison of various serological assays for novel SARS-COV-2. Eur J Clin Microbiol Infect Dis. 2020. Epub 2020/11/26. pmid:33236270; PubMed Central PMCID: PMC7685776.
- 17. Nilsson AC, Holm DK, Justesen US, Gorm-Jensen T, Andersen NS, Ovrehus A, et al. Comparison of six commercially available SARS-CoV-2 antibody assays-Choice of assay depends on intended use. Int J Infect Dis. 2020;103:381–8. Epub 2020/12/15. pmid:33310021; PubMed Central PMCID: PMC7726521.
- 18. Deeks JJ, Dinnes J, Takwoingi Y, Davenport C, Spijker R, Taylor-Phillips S, et al. Antibody tests for identification of current and past infection with SARS-CoV-2. Cochrane Database Syst Rev. 2020;6:CD013652. Epub 2020/06/26. pmid:32584464; PubMed Central PMCID: PMC7387103.
- 19. Duffy S. Why are RNA virus mutation rates so damn high? PLoS Biol. 2018;16(8):e3000003. Epub 2018/08/14. pmid:30102691; PubMed Central PMCID: PMC6107253.
- 20. Callaway E. The coronavirus is mutating—does it matter? Nature. 2020;585(7824):174–7. Epub 2020/09/10. pmid:32901123.