https://orcid.org/0000-0002-5688-2116
Figures
Abstract
SARS-CoV-2 antibody tests have been marketed to diagnose previous SARS-CoV-2 infection and as a test of immune status. There is a lack of evidence on the performance and clinical utility of these tests. We aimed to carry out an evaluation of 14 point of care (POC) SARS-CoV-2 antibody tests. Serum from participants with previous RT-PCR (real-time polymerase chain reaction) confirmed SARS-CoV-2 infection and pre-pandemic serum controls were used to determine specificity and sensitivity of each POC device. Changes in sensitivity with increasing time from infection were determined on a cohort of study participants. Corresponding neutralising antibody status was measured to establish whether the detection of antibodies by the POC device correlated with immune status. Paired capillary and serum samples were collected to ascertain whether POC devices performed comparably on capillary samples. Sensitivity and specificity varied between the POC devices and in general did not meet the manufacturers’ reported performance characteristics, which signifies the importance of independent evaluation of these tests. The sensitivity peaked at ≥20 days following onset of symptoms, however sensitivity of 3 of the POC devices evaluated at extended time points showed that sensitivity declined with time. This was particularly marked at >140 days post infection. This is relevant if the tests are to be used for sero-prevalence studies. Neutralising antibody data showed that positive antibody results on POC devices did not necessarily confer high neutralising antibody titres, and that these POC devices cannot be used to determine immune status to the SARS-CoV-2 virus. Comparison of paired serum and capillary results showed that there was a decline in sensitivity using capillary blood. This has implications in the utility of the tests as they are designed to be used on capillary blood by the general population.
Citation: McCance K, Wise H, Simpson J, Batchelor B, Hale H, McDonald L, et al. (2022) Evaluation of SARS-CoV-2 antibody point of care devices in the laboratory and clinical setting. PLoS ONE 17(3): e0266086. https://doi.org/10.1371/journal.pone.0266086
Editor: Gheyath K. Nasrallah, Qatar University, QATAR
Received: December 8, 2021; Accepted: March 13, 2022; Published: March 31, 2022
Copyright: © 2022 McCance et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting information files.
Funding: Funding and support for this study was provided by Lothian NRS BioResource obtained by SJ. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Abbreviations: COVID-19, coronavirus disease 2019; hCoV, Human Coronavirus; HCW, healthcare worker; LFA, lateral flow assay; NPV, negative predictive value; NT50, half maximal neutralising titre; POC, point of care; PPV, positive predictive value; RT-PCR, real time polymerase chain reaction; SARS-CoV-2, severe acute respiratory syndrome coronavirus 2
Introduction
The emergence of SARS-CoV-2 has resulted in rapid research and development of commercial diagnostic tests by both laboratories and commercial manufacturers. The unusually fast pace of this development risks potential compromises on quality in the absence of rigorous independent evaluation and validation. It is therefore essential to ensure adequate performance of tests for population wide or individual use to prevent roll out of devices which add no, or at best, minimal clinical value to individual patients or the wider population. At worst, inadequately performing diagnostic tests can produce misleading clinical information with potential for harmful consequences.
The current most widely used diagnostic test for SARS-CoV-2 is based on real-time polymerase chain reaction (RT-PCR) amplification of viral RNA from an upper respiratory tract sample [1,2]. However, due to a limited time window of active infection, capacity constraints, and access to these tests mostly restricted to symptomatic patients, cases determined using this method underestimate the true burden of infection. In contrast, serological assays test for previous infection and are therefore a key additional tool for monitoring prevalence of infection within the population. Antibody tests can also be used as an aid in diagnosis where COVID-19 is suspected clinically but the PCR time window has passed [3]. Significant interest exists in the potential for use of these tests at an individual level to provide an indication of immune status and act as an ‘immunity passport’. For countries where vaccine availability is limited, pre-screening the population with antibody tests in order to identify individuals who may either not require vaccination or be suitable for a reduced vaccine dosing regimen may help optimise the use of limited vaccine resources.
Reliable antibody tests also have the potential to identify hospitalised COVID-19 patients who may benefit from the use of monoclonal antibody treatment. Ronapreve, a combination of Casirivimab and Imdevimab, is a monoclonal antibody treatment directed at the spike protein receptor binding domain on SARS-CoV-2 [4] which has been shown to significantly reduce 28 day mortality in seronegative hospitalised patients [5].
A large number of commercially available immunoassays have been developed to detect SARS-CoV-2 IgG, IgM, IgA and total antibodies [6]. Although the majority of antibody production is directed towards the more abundant N (nucleocapsid) protein, the S (spike) protein contains the receptor-binding domain responsible for host cell attachment, and antibodies to the S protein are therefore predicted to be neutralising [7].
In contrast to laboratory-based immunoassays, which require venous sampling and transport to centralised testing sites, lateral flow immunoassays (LFA) offer the potential to allow rapid, cheap, mass population antibody testing on capillary samples in the home environment. In order to offer clinical utility at a home population level, and relieve pressure on clinical services, the LFA must be able to reliably operate using capillary whole blood samples; the test must also offer sufficient ease of use and interpretation of results to be acceptable to the general ‘lay-person’ population. Additionally, if LFAs are to be used for population sero-surveillance they will have to be sensitive enough to detect the presence of antibodies in those who only suffered from mild disease or were asymptomatic. Of even greater importance is the test specificity when testing at a population level where the pre-test probability is low. Without adequate specificity the chance of a positive result being a false positive is considerable and this is of particular concern if these tests were to be used as evidence of immune status or for ‘immunity passports’. It is currently unclear if the detection of antibodies to the COVID 19 virus by LFA confers immunity to the virus, and some manufactures fail to disclose whether the assay is directed towards the N or S protein.
Worldwide, over 200 LFAs have been produced to date [6]; for the majority of these devices only manufacturer reported performance data is available. CE marking of these devices does not require external validation, and the manufacturer’s in-house validation process is often not publically available. The UK Medicines and Healthcare Products Regulatory Agency (MHRA) advise that any SARS-CoV-2 LFA should have sensitivity and specificity > 98% (95% CI 96–100%) in samples collected ≥ 20 days post symptom onset irrespective of whether they are performed as a home self test kit or by a trained health care professional [8]. Despite claims by many manufactures to have achieved this target (see S1 Table), UK external validations of LFAs have so far failed to reproduce this [9–11].
As part of a nationwide evaluation of SARS-CoV-2 diagnostic tests, we carried out an independent validation of 14 POC SARS-CoV-2 antibody tests, including 13 LFAs. We assessed their specificity and sensitivity against the manufacturer’s claims and determined compatibility with MHRA criteria. For tests that performed well against the MHRA criteria, we studied changes in sensitivity with increasing time from infection. To investigate suitability for home use, we evaluated ease of use and interpretation of these tests when performed directly by patients. Neutralising antibody status was determined for samples to establish whether the detection of antibodies by the LFA correlated with immune status.
Methods
Point of care (POC) device selection and the evaluation process
National Service Scotland, a Non-Departmental Public Body which provides advisory services to NHS Scotland, was approached by a series of commercial manufacturers for external evaluation of their diagnostic devices. The manufacturer claims and costing were reviewed and manufacturers who passed these initial checks were invited to send up to 100 POC test kits for initial evaluation. 14 POC devices designed to detect antibodies to the SARS-CoV-2 virus were evaluated. The tests included in the evaluation were: AbC-19, Alpha Pharma, Biomerica, Biozek, Fortress, Jiangsu, Lepu, Menarini Healgen, Mologic, Pharmact, Roche, Wuhan Easy Diagnosis, Wuhan Life Origin Biotech (Syzbio), and LumiraDX. 13 of the POC devices were LFAs, consisting of immunochromatography based cassettes, while the LumiraDX assay used a POC microfluidic immunofluorescence assay. Each device measured IgG and IgM except AbC-19, which measured IgG only, and LumiraDX, which gave an antibody result of unspecified subclass. Target antigens (either S or N) were not always disclosed by the manufacturers. The tests were performed as per the manufacturer’s instructions. Typically, 2.5μL to 10μL of serum or capillary sample was pipetted into the sample well followed by a pre-specified volume of buffer supplied by the manufacturer (see S1 Table). Serum samples were analysed at room temperature in the laboratory at the Royal Infirmary of Edinburgh. The read time varied from 10 to 20 minutes depending on manufacturer’s instructions. Photographic records of serum results were taken. Capillary samples were run as part of a research clinic. Antibody results were visually read and recorded as positive, weak positive, or negative for IgG and IgM. Positive and weak positive results were deemed an overall positive result.
In the first stage of the process up to 50 serum samples from RT-PCR confirmed SARS-CoV-2 positive individuals (mainly hospitalised patients) and up to 50 pre-pandemic SARS-CoV-2 negative samples were run. The number of days the sample was collected post symptom onset was recorded for the positive cases. Initial sensitivity and specificity were calculated and if the test kit performed within agreed parameters (typically defined as an IgG specificity of >98% and sensitivity of >95% at ≥day 20 post symptom onset) the company was taken forward to the second stage of evaluation and asked to provide further test kits. Although the MHRA definition suggests that SARS-CoV-2 LFA devices should achieve >98% sensitivity, the cut-off was reduced to >95% sensitivity for the purposes of this study as only one manufacturer’s device was able to achieve the >98% cut-off. Further stages of the evaluation process which were carried out for a subset of kits included more stringent specificity testing, determination of batch to batch variation, sensitivity as time from infection increased, comparison of capillary to serum results, and evaluation of ease of use by study participants in a research clinic.
Overview of participants and samples
Serum samples from hospitalised patients with RT-PCR confirmed SARS-CoV-2 were collected between March and November 2020. Excess serum was stored at the point of discard from hospitalised COVID-19 patients using ethical permissions obtained through NRS BioResource.
Out-patients with previous RT-PCR confirmed SARS-CoV-2 infection were also recruited to two COVID convalescent research studies where participants were invited to donate serial blood samples (SR1407 BioResource study, n = 112 participants) and to provide capillary and serum blood samples (COVID-19 Antibody Test Evaluation (CATE) Study, n = 82 participants). For both studies the date of symptom onset and positive RT-PCR test result were recorded. Inclusion criteria were those aged >16 years old with previous RT-PCR confirmed diagnosis of COVID-19. Only 5 participants required hospitalisation following infection, and none of these required intensive care. Due to the limited access to COVID-19 testing at the start of the pandemic when most study recruitment occurred many study participants were health care workers. Frail or shielding individuals were excluded due to the study requirements for travel. Each participant provided between 1 and 5 serum samples. For participants providing serial samples these were collected between 29 and 224 days post PCR confirmed infection. Capillary and paired serum samples were collected as part of the CATE study research clinic between September and November 2020. Capillary samples were tested in the clinic setting with paired serum samples subsequently run on the same POC devices in the laboratory to assess concordance on capillary and serum samples. A subset of CATE study participants were asked to carry out one of the POC tests themselves including finger prick testing with the aid of the manufacturer’s printed instructions. Health care workers intervened if the participant needed support in completing the testing and any intervention was recorded. Afterwards participants were asked to complete a questionnaire on their experience of performing the test. For a subset of other test kits capillary results were interpreted by both the participant (with the aid of a labelled diagram) and the researcher.
Negative controls consisted of venous samples collected prior to December 2019. For more stringent specificity testing, samples positive for rheumatoid factor, seasonal coronaviruses, other respiratory pathogens, CMV (cytomegalovirus) or EBV (Epstein-Barr virus) were used as negative controls along with samples from the national antenatal screening programme and SNBTS (Scottish National Blood Transfusion Service).
Wherever possible the same panel of samples were run on each POC device but due to limitations in sample volume not all devices were assessed with the same panel of samples. However, a similar variety of samples types were used to test each kit.
Neutralisation assay
Neutralising antibody assays were performed on serum samples using a pseudotyped chimeric virus, expressing the SARS-CoV-2 spike protein, as described elsewhere [12].
Statistical analysis
The primary outcome was the sensitivity and specificity of each POC test. For sensitivity, tests were compared against PCR-confirmed SARS-CoV-2 infection. Specificity of each POC test was evaluated against pre-pandemic negative samples, with all positives counting as false positives. The analysis included all available data for the relevant outcome and is presented with the corresponding 95% confidence intervals (Clopper Pearson).
For samples where neutralising antibody levels are available positive predictive value (PPV) and negative predictive value (NPV) are calculated at two NT50 thresholds; ≥ 50 and ≥ 160.
For the comparison of the performance of each POC test between clinic capillary and serum samples we calculated the sensitivity and 95% confidence interval. The McNemar test was used to assess for significant difference between dependent groups. Agreement between serum and capillary samples was measured using the Cohen’s Kappa. Results of the Kappa were interpreted as previously described (<0, poor agreement; 0.00–0.20, slight agreement; 0.21–0.40, fair agreement; 0.41–0.60, moderate agreement; 0.61–0.80, substantial agreement; and >0.8, almost perfect agreement) [13].
All data were analysed using PRISM (Version 9) and SPSS, and a p value<0.05 was considered significant.
Ethics
Ethical approval for access to pre-pandemic stored samples and hospitalised COVID-19 patient samples was obtained through the NRS BioResource (Ref: SR1410). Outpatient COVID-19 convalescent patient samples were obtained through the NRS BioResource (Ref: SR1407) and the COVID-19 Antibody Test Evaluation (CATE) study, with ethical approval from London-Brent Research Ethics Committee (REC ref: 20/HRA/3764 IRAS: 286538).
Results
Comparison of POC device performance using serum samples
14 POC devices were evaluated for sensitivity and specificity including 13 LFAs and one microfluidic immunofluorescence assay (Lumira DX). In the first stage the sensitivity of the devices was assessed using a panel of 50 serum samples from RT-PCR positive individuals. The specificity of the device was assessed using a panel of 50 serum samples collected prior to December 2019 for routine virological investigations. Adequately performing devices (IgG specificity > 98% and IgG sensitivity >95% ≥ 20 days post symptom onset) then progressed to stage 2, where they were evaluated against an expanded panel of serum samples. In particular, a more stringent specificity panel was used, including serum samples taken from individuals with a positive PCR result for the endemic coronaviruses, other respiratory pathogens, and samples with positive rheumatoid factor, CMV or EBV serology.
The overall results for each device are summarised in Table 1 and Fig 1. For each sample, and each device, the IgG and IgM result was scored as negative, weak positive or positive (the AbC-19 LFA only had an IgG window, and the LumiraDX platform only returned an antibody result). For the purposes of the sensitivity and specificity calculations, weak positive and positive results were counted as positive.
Sensitivity against specificity for the kits tested (IgG only, and positive or negative result for the LumiraDx assay). For each kit the ≥20 days post symptom onset sensitivity is shown. The MHRA targets of 98% sensitivity and specificity and additional 95% sensitivity cut off are shown as dotted lines. Pharmact was not included on the graph due to low specificity. The devices that did not progress beyond this stage are shown in red.
Only the LumiraDX POC assay achieved the MHRA criteria for sensitivity and specificity when all testing was considered (> 98% for both), although a number of the LFA devices (Biomerica, Biozek, Fortress, Menarini and Roche) surpassed the specificity criteria, even when tested against the high stringency specificity serum samples. One empirical finding was that there was extensive variation in the strength of the staining between the kits, which could affect ease of interpretation. Another more concerning observation was that batch-to-batch variation in performance was observed for the Menarini assay, with batch 2005156 giving a sensitivity of 76.3% on samples >20 days post infection versus batch 2003288 giving a sensitivity of 94.9% on the same samples (n = 99).
The AbC-19, Biomerica, Biozek, Fortress, Menarini (batch 2003288) and Roche LFA devices and the LumiraDX assay underwent further analysis using an expanded panel of RT-PCR positive samples. Initially, sensitivity was examined at proximal time points to infection for the Biomerica, Biozek, Menarini, Roche and LumiraDX devices and the results are shown in Table 2 and Fig 2. AbC-19 and Fortress were not included in this part of the analysis as sensitivity data at proximal time points to infection were not available for these kits. This reflected the times the different kits became available, the amount of serum sample that was available when the kits became available, and our wish not to duplicate the work that was being done by other groups at the time. All the devices tested showed an initial increase in sensitivity, peaking at ≥20 days post symptom onset, with sensitivities of ≥ 90% obtained for all devices from this time point onwards. Between 8–20 days post symptom onset, performance was more variable, with Roche showing the lowest sensitivity, and Menarini the highest.
Sensitivity against time post symptom onset for selected kits (IgG for all devices, with the exception of LumiraDX, where overall antibody result was used).
It was important to determine the sensitivity of the POC devices at later time points post onset of symptoms, as it has been demonstrated that antibody levels wane over time following the primary antibody response to SARS-CoV-2 [14–16]. In order to examine this, we used a set of samples taken from the non-hospitalised convalescent cohort, since the majority of patients in the general population were not hospitalised with SARS-CoV-2 infection, and serial samples were available. Sensitivity was analysed across three time windows ≥20 days post symptom onset for the AbC-19, Biomerica, Roche and LumiraDX predominantly using samples from the convalescent cohort, and the results are shown in Table 3 and Fig 3.
Sensitivity against time post symptom onset for selected kits (IgG for all devices, with the exception of LumiraDX, where overall antibody result was used).
The LumiraDX assay was the only assay evaluated that maintained its sensitivity at 98% across the time interval studied. The three LFA devices (AbC-19, Biomerica, Roche) showed variable drops in sensitivity with increasing time from symptom onset, with Biomerica dropping from 91% to 54% over the time points studied.
For the convalescent cohort samples, neutralising titres were also available, and the POC result (for AbC-19, Biomerica, Roche and LumiraDX) could be compared with the half maximal neutralising titre (NT50). Initially, the NT50 result for each serum sample was plotted, dividing each device by positive and negative IgG (or for LumiraDX, antibody) result (Fig 4).
NT50 against binary POC test result. The NT50 results for the positive and negative device result are plotted by kit (the mean and standard error of the mean (SEM) are denoted by black lines). Dotted lines represent the NT50 ≥ 50 and ≥ 160 thresholds, and the p values from a Kruskal-Wallis test are shown.
There was a statistically significant difference in NT50 level between the positive and negative result for all assays. However, for the positive result, a large range of NT50 values was observed. To examine this further, the sensitivity and specificity of the device was evaluated at two NT50 thresholds; ≥ 50 and ≥ 160 (Table 4). These two thresholds were selected based on a detectable NT50 (≥ 50) and a threshold which may confer protection from reinfection (≥ 160) [17].
The sensitivity of all the POC devices for detecting an NT50 ≥ 50 (of the 280 serum samples, 222 had an NT50 ≥ 50 and 58 had an NT50 < 50) was at least 90%, and for Roche and LumiraDX it was > 98%. However, the NPV varied between 40 and 80%, as a result of the small number of samples that were negative by the POC devices relative to the number of positive samples (Fig 4). Specificity performance was also poor with a large number of false positive results (POC positive, but NT50 < 50). In particular, the LumiraDX assay had a specificity of < 5%, whereas the AbC-19 assay had the highest specificity of 62%. These results impacted the PPV in this high prevalence setting, ranging from 90% for the AbC-19 assay and 84% for the LumiraDX assay.
When the NT50 threshold was raised to ≥ 160 (where 142 of the serum samples had an NT50 ≥ 160 and 138 had an NT50 < 160), the sensitivity for all the POC devices rose to at least 96%, with a consequential increase in NPV to at least 90%. However, the specificity of all of the devices fell, with the AbC-19 kit having the highest specificity at 42%. This affected the PPV, which also fell from the lower NT50 threshold, to between 55 and 64%.
Comparison of POC device performance in serum and capillary samples
A major advantage of the POCT devices over immunoassays performed within accredited laboratories is the lack of requirement for phlebotomy. However, many evaluations of POC performance have not compared serum and capillary samples. Furthermore, it has been suggested that LFA could be suitable for home use, outside of a healthcare setting, and this would require the lay participant to interpret their test result.
In this study, paired serum and capillary samples collected on the same study visit were available for a number of participants in the convalescent cohort. The performance of seven POC assays (AbC-19, Biomerica, Biozek, Fortress, Menarini, Roche and LumiraDX) was compared. The serum sample was processed in the laboratory, and the result read by a health care worker (HCW), while the capillary sample result was read by the study participant (with the aid of a labelled diagram or instruction leaflet) and a second HCW in the research clinic. Sensitivity findings for the 3 matched interpretations are shown in Table 5 and Fig 5.
Sensitivity for POC devices comparing serum, and HCW and participant interpreted capillary data. Sensitivity and 95% CI (Clopper-Pearson) were plotted.
A common feature of the POC results in capillary samples was a reduction in sensitivity compared to the serum samples. This was seen in 5 out of the 7 devices analysed, however the magnitude of the effect varied. Both Biozek and Roche showed significant decreases between serum and capillary sensitivity when interpreted by either HCW or participants. This decrease in sensitivity ranged from 19.1 to 34.2%. AbC-19 showed a significant decrease in sensitivity between serum and capillary results read by participants, decreasing from 73.3% to 33.3%. The Fortress LFA showed a small increase in sensitivity in capillary samples compared to serum but this was not statistically significant. The LumiraDX assay showed 100% sensitivity for both samples (n = 11) but a large number of test fails (n = 18) occurred on capillary samples at the research clinic.
The concordance between serum, HCW and participant read capillary results are displayed in Table 5. Concordance between the devices varied. For example, of the 45 AbC-19 results, 24 were interpreted as positive by a HCW but only 15 by the study participants. In contrast, for the Fortress LFA, there was a single discrepancy between the HCW and participant interpretations, where the participant reported a positive result, while the HCW reported a negative result.
Participants who performed self-testing using the AbC-19 LFA in the clinic also completed a questionnaire that addressed aspects of self testing using the kits (n = 35 participants). Their responses are summarised in Fig 6. Overall, 89% of participants reported it was “very easy” or “easy’ to understand the leaflet explaining the results. 75% reported it was “very easy” or “easy” to understand the instructions. Any issues appeared to be predominantly associated with taking the capillary samples and using the test kit, with 20% of participants reporting it was “difficult” or very difficult’ to take the sample.
Summary of ease of use questionnaire. Participant ease of use responses to the indicated categories were expressed as a percentage of responses.
Discussion
The SARS-CoV-2 pandemic has generated unprecedented demand for testing both to confirm acute infection and past virus exposure. In particular, serological assays measure prior exposure to the virus and are being used in high volume settings to measure seroprevalence at a population level. The role for these tests at an individual level is still poorly understood, and there are huge risks associated with the use of poorly performing tests. For example, using a test with a inadequate specificity in a low prevalence population would result in a higher proportion of people with a false positive result compared to a true positive result, leading people to believe they were antibody positive, when they are not.
This study was designed as part of a national evaluation of SARS-CoV-2 antibody POC tests within NHS Scotland, and aimed to address several questions regarding the performance of the POC tests. To be effective, POC devices should have similar performance in samples from patients who experience relatively mild illness compared to patients requiring hospitalisation. This was addressed using a cohort of convalescent patients, the majority of whom did not require hospital treatment during their illness. This cohort was also used to assess the performance of the POC device with increasing time from infection, and to examine how well they were able to predict neutralising antibody titres.
Another major requirement is for POC devices to perform well in capillary samples compared to serum. This point is critical since a major strength of the POC devices is that they could be used on capillary samples, thus reducing the requirement for phlebotomy.
In this work, 14 POC devices were evaluated on serum samples, and a number of these tests underwent further evaluation on capillary samples. From the serum evaluation, it became clear that specificity performance of many of the POC devices was good–with 11 out of 14 reaching the MHRA specificity target of > 98% for IgG, or in the case of LumiraDX, total antibody. However, only a single device (LumiraDX) reached the MHRA sensitivity criteria of >98% ≥ 20 days post symptom onset. The MHRA criteria state that both sensitivity and specificity should be ≥98% therefore only LumiraDX met these standards. The sensitivity panel that was used to assess these POC devices in the first instance consisted predominantly of hospitalised patients, who tend to have higher antibody titres than patients with milder disease [18–23], making disease severity a less likely explanation for these observations. The reduced sensitivity of the devices compared to manufacturer’s claims is not unique to the methodology; the sensitivity of a laboratory analyser immunoassay on this cohort was 93% at > 20 days post symptom onset. Therefore at relatively early time points post symptom onset, a negative result cannot rule out SARS-CoV-2 exposure. Indeed, for the subgroup of POC devices where sensitivity with time was examined it was clear that, at early time points, sensitivity increased with time reaching maximal sensitivity after 20 days post symptom onset. This was in keeping with other reports that have studied the time to reach maximal antibody titres [18,24–26].
An equally important question regarding timing of testing was how long post infection the device could detect an antibody response. The issue of sero-reversion is important for seroprevalence studies, and the ideal test for this purpose would have a high sensitivity for a long period of time. Otherwise, sero-reversion may result in underestimation of seroprevalence, and models to account for this trend of waning antibodies themselves risk over- or under-correcting for this effect [27,28]. To examine this, a cohort of SARS-CoV-2 convalescent individuals for whom longitudinal serum samples were available were used. Crucially, these patients were most representative of the majority of SARS-CoV-2 infections in so far as only 5 participants providing serial samples required hospitalisation during their initial COVID-19 infection, and none required treatment in intensive care. In this study, the LumiraDX assay was the only assay to maintain a high sensitivity at time points from 21 to >140 days post symptom onset; the other assays showed variable levels of declining sensitivity. This was most marked for the Biomerica LFA, which had a sensitivity of 91% at 21–79 days post symptom onset, dropping to 54% at ≥ 140 days post symptom onset.
The relative importance of specificity compared to sensitivity at least in part depends on the intended use. For example, for serosurveillance, particularly in low prevalence populations, specificity should be maximised to reduce the false positive rate. However, in clinical settings, specificity may be compromised in favour of sensitivity if there were adequate follow up tests to identify true positives.
A major question remains over the significance of a positive antibody test result in terms of protection from reinfection, or protection from infection following vaccination. This is pertinent given the continued discussions regarding “immunity passports” and identifying non-immune individuals, through antibody testing, for priority vaccination where vaccine supply is limited. In this study, we compared the POC device result with the NT50 level (Fig 4 and Table 4). It was apparent that a positive result was associated with a wide range of neutralising antibody levels, and that a number of positive samples had low NT50 levels (< 50). This adds to the concern that a positive POC result may not provide relevant information about the likelihood of protection from infection. The risk of using these tests for the purpose of ‘immunity passports’ is further exacerbated by concerns over specificity and the circulation of new variants that may be able to evade existing humoral responses [29,30]. Whilst some of the better performing devices may be suitable for sero-prevalence studies, they should not be used to draw conclusions on an individual’s protection from reinfection.
A major advantage of POC devices stems from their potential to relieve requirement for phlebotomy and to be performed outside of a healthcare setting. However, for this potential to be realised, additional factors including POC device performance in capillary samples and ease of use should be considered. In this study, a head to head comparison of POC device performance on serum and capillary samples was performed, and for the LFAs HCW and participant interpretation of the test result was also compared. There were differences in performance on serum and capillary samples, with the majority of the devices showing reduced sensitivity on capillary samples compared to serum. Concordance between serum and capillary results was sometimes poor, indicating that the performance of a test on serum under laboratory conditions cannot be assumed to equate to performance on a capillary sample. Only the LumiraDX device was equally sensitive on both capillary and serum samples.
There was good agreement between capillary results interpreted by HCW and participants for the Roche, Menarini, Biozek and Fortress kits, indicating that these LFA tests may be best suited for use as home test kits. For other LFAs, such as AbC-19, many study participants had difficulty noticing the faint lines produced by the test kit and interpreted the result as being negative when it was interpreted as positive by a HCW.
A disadvantage of the POC devices compared to laboratory immunoassays is that they are strictly qualitative (with the exception of LumiraDX, where a numeric value is converted to a qualitative result). This means the possibility of introducing equivocal zones is not available. Further issues that came to light during the course of this evaluation work included the possibility of batch-to-batch variation in kits from the same manufacturer. For example two different batches of Menarini kits showed marked differences in sensitivity (76.7% on one batch verus 94.9% on another batch). Whilst we did not observe any failed tests during our evaluation of the LFA using serum or capillary samples, a high failure rate was seen using the LumiraDX device with capillary samples with 18 out of 29 participants having a failed test result. Possible explanations for this include sample clotting in the device prior to analysis or unrefined design leading to inadequate capillary action to draw up sample from the application point. Unpublished data available from another POC device evaluation site in Scotland (NHS Tayside), where a newer version of the LumiraDX device was being trialled, indicated that there were no issues with failed SARS-CoV-2 antibody capillary tests. This suggests that this issue has now been resolved by the company. However, strategies to identify such issues should continue to be employed by the manufacturer and/or the end user to ensure accurate performance on capillary samples.
The strength of this study lies in its contribution to our understanding of POC device performance; in particular performance at different time points post infection, confirmed evidence of batch to batch variation with some POC kits, and demonstration of the general lack of correlation between a positive device result and the presence of neutralising antibodies. The data also demonstrates that the results obtained in the lab using serum cannot automatically be assumed to be representative of finger-prick capillary test results. Furthermore, a significant proportion of this research used a convalescent cohort of patients with relatively mild disease, making the findings applicable to community-based studies.
However, the study does have limitations. Due to sample availability constraints and the large number of devices evaluated, not all of the kits could be evaluated on the same panel of positive and negative samples. In particular, the numbers of samples where paired serum and capillary sample results were available to assess concordance was limited for the majority of the devices examined. For one of the test kits ease of use was assessed but as many of the study participants providing capillary samples in this study were healthcare workers, the data obtained through this may not be representative of the general population.
Conclusion
Our results highlight a wide variation in performance of SARS-CoV-2 antibody test kits and illustrate the importance of verifying multiple different aspects of test performance. Checking for batch to batch variation, changes in sensitivity as time from infection increases, correlation with neutralising antibodies, and performance on capillary samples should all be evaluated. This is essential prior to considering the utilisation of these tests for ‘immunity passports’ and identification of hospitalised patients who would benefit from monoclonal antibody treatment, as well as utilisation of these devices to enable targeted vaccine distribution in areas of the world of vaccine inequity.
Supporting information
S1 Table. Manufacturers’ published product specifications and performance data.
https://doi.org/10.1371/journal.pone.0266086.s001
(DOCX)
Acknowledgments
Support for this study was provided by Lothian NRS BioResource. We would also like to thank members of the National Service Scotland team for their support with co-ordinating this work.
References
- 1.
Guidance and Standard Operating Procedure. COVID-19 Virus Testing in NHS Laboratories [Internet]. NHS England and NHS Improvement; https://www.rcpath.org/uploads/assets/90111431-8aca-4614-b06633d07e2a3dd9/Guidance-and-SOP-COVID-19-Testing-NHS-Laboratories.pdf.
- 2.
Interim Guidelines for Collecting, Handling, and Testing Clinical Specimens for COVID-19 [Internet]. Centers for Disease Control and Prevention; https://www.cdc.gov/coronavirus/2019-nCoV/lab/guidelines-clinicalspecimens.html.
- 3. Guo L, Ren L, Yang S, Xiao M, Chang D, Yang F, et al. Profiling Early Humoral Response to Diagnose Novel Coronavirus Disease (COVID-19). Clin Infect Dis. 2020 Mar 21;ciaa310.
- 4. van de Veerdonk FL, Giamarellos-Bourboulis E, Pickkers P, Derde L, Leavis H, van Crevel R, et al. A guide to immunotherapy for COVID-19. Nat Med. 2022 Jan;28(1):39–50. pmid:35064248
- 5. RECOVERY Collaborative Group, Horby PW, Mafham M, Peto L, Campbell M, Pessoa-Amorim G, et al. Casirivimab and imdevimab in patients admitted to hospital with COVID-19 (RECOVERY): a randomised, controlled, open-label, platform trial [Internet]. Infectious Diseases (except HIV/AIDS); 2021 Jun [cited 2021 Oct 20]. http://medrxiv.org/lookup/doi/10.1101/2021.06.15.21258542.
- 6.
SARS-CoV-2 diagnostic pipeline [Internet]. https://www.finddx.org/covid-19/pipeline/.
- 7. Sethuraman N, Jeremiah SS, Ryo A. Interpreting Diagnostic Tests for SARS-CoV-2. JAMA. 2020 Jun 9;323(22):2249. pmid:32374370
- 8.
Target product profile antibody tests to help determine if people have immunity to SARS-CoV-2, 2020. [Internet]. MHRA; https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/883897/Target_Product_Profile_antibody_tests_to_help_determine_if_people_have_immunity_to_SARS-CoV-2_Version_2.pdf.
- 9. Flower B, Brown JC, Simmons B, Moshe M, Frise R, Penn R, et al. Clinical and laboratory evaluation of SARS-CoV-2 lateral flow assays for use in a national COVID-19 seroprevalence survey. Thorax. 2020 Aug 12;thoraxjnl-2020-215732.
- 10. Adams ER, Anand R, Andersson MI, Auckland K, Baillie JK, Barnes E, et al. Evaluation of antibody testing for SARS-Cov-2 using ELISA and lateral flow immunoassays [Internet]. Infectious Diseases (except HIV/AIDS); 2020 Apr [cited 2020 Apr 20]. http://medrxiv.org/lookup/doi/10.1101/2020.04.15.20066407.
- 11. Pickering S, Betancor G, Galão RP, Merrick B, Signell AW, Wilson HD, et al. Comparative assessment of multiple COVID-19 serological technologies supports continued evaluation of point-of-care lateral flow assays in hospital and community healthcare settings. Fouchier RAM, editor. PLOS Pathog. 2020 Sep 24;16(9):e1008817. pmid:32970782
- 12. Muecksch F, Wise H, Batchelor B, Squires M, Semple E, Richardson C, et al. Longitudinal analysis of serology and neutralizing antibody levels in COVID19 convalescents. J Infect Dis. 2020 Nov 3;jiaa659.
- 13. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977 Mar;33(1):159–74. pmid:843571
- 14. Bal Antonin, Trabaud M-A, Fassier J-B, Rabilloud M, Saker K, Langlois-Jacques C, et al. Six-month antibody response to SARS-CoV-2 in healthcare workers assessed by virus neutralisation and commercial assays. Clin Microbiol Infect. 2021 Jan;S1198743X21000057.
- 15. L’Huillier AG, Meyer B, Andrey DO, Arm-Vernez I, Baggio S, Didierlaurent A, et al. Antibody persistence in the first six months following SARS-CoV-2 infection among hospital workers: a prospective longitudinal study. Clin Microbiol Infect. 2021 Jan;S1198743X21000318.
- 16. Dan JM, Mateus J, Kato Y, Hastie KM, Yu ED, Faliti CE, et al. Immunological memory to SARS-CoV-2 assessed for up to 8 months after infection. Science. 2021 Feb 5;371(6529):eabf4063. pmid:33408181
- 17. Addetia A, Crawford KHD, Dingens A, Zhu H, Roychoudhury P, Huang M-L, et al. Neutralizing Antibodies Correlate with Protection from SARS-CoV-2 in Humans during a Fishery Vessel Outbreak with a High Attack Rate. McAdam AJ, editor. J Clin Microbiol. 2020 Aug 21;58(11):e02107–20, /jcm/58/11/JCM.02107-20.atom. pmid:32826322
- 18. Long Q-X, Liu B-Z, Deng H-J, Wu G-C, Deng K, Chen Y-K, et al. Antibody responses to SARS-CoV-2 in patients with COVID-19. Nat Med [Internet]. 2020 Apr 29 [cited 2020 Apr 29]; http://www.nature.com/articles/s41591-020-0897-1. pmid:32350462
- 19. The UCSF COMET Consortium, Combes AJ, Courau T, Kuhn NF, Hu KH, Ray A, et al. Global absence and targeting of protective immune states in severe COVID-19. Nature. 2021 Mar 4;591(7848):124–30. pmid:33494096
- 20. Tan AT, Linster M, Tan CW, Le Bert N, Chia WN, Kunasegaran K, et al. Early induction of functional SARS-CoV-2 specific T cells associates with rapid viral clearance and mild disease in COVID-19 patients. Cell Rep. 2021 Jan;108728.
- 21. Guthmiller JJ, Stovicek O, Wang J, Changrob S, Li L, Halfmann P, et al. SARS-CoV-2 Infection Severity Is Linked to Superior Humoral Immunity against the Spike. Schultz-Cherry S, editor. mBio. 2021 Jan 19;12(1):e02940–20, /mbio/12/1/mBio.02940-20.atom. pmid:33468695
- 22. Wang P, Liu L, Nair MS, Yin MT, Luo Y, Wang Q, et al. SARS-CoV-2 neutralizing antibody responses are more robust in patients with severe disease. Emerg Microbes Infect. 2020 Jan 1;9(1):2091–3. pmid:32930052
- 23. Wajnberg A, Amanat F, Firpo A, Altman DR, Bailey MJ, Mansour M, et al. Robust neutralizing antibodies to SARS-CoV-2 infection persist for months. Science. 2020 Oct 28;eabd7728. pmid:33115920
- 24. Borremans B, Gamble A, Prager K, Helman SK, McClain AM, Cox C, et al. Quantifying antibody kinetics and RNA shedding during early-phase SARS-CoV-2 infection [Internet]. Open Science Framework; 2020 May [cited 2020 May 19]. https://osf.io/evy4q. pmid:32894217
- 25. Lou B, Li T-D, Zheng S-F, Su Y-Y, Li Z-Y, Liu W, et al. Serology characteristics of SARS-CoV-2 infection after exposure and post-symptom onset. Eur Respir J. 2020 Aug;56(2):2000763. pmid:32430429
- 26. Zhao J, Yuan Q, Wang H, Liu W, Liao X, Su Y, et al. Antibody Responses to SARS-CoV-2 in Patients With Novel Coronavirus Disease 2019. Clin Infect Dis. 2020 Nov 19;71(16):2027–34. pmid:32221519
- 27. Sabino EC, Buss LF, Carvalho MPS, Prete CA, Crispim MAE, Fraiji NA, et al. Resurgence of COVID-19 in Manaus, Brazil, despite high seroprevalence. The Lancet. 2021 Jan;S0140673621001835. pmid:33515491
- 28.
Kadelka S, Bouman JA, Ashcroft P, Regoes RR. Comment on Buss et al., Science 2021: An alternative, empirically-supported adjustment for sero-reversion yields a 10 percentage point lower estimate of the cumulative incidence of SARS-CoV-2 in Manaus by October 2020. ArXiv210313951 Q-Bio [Internet]. 2021 Mar 25 [cited 2021 May 25]; http://arxiv.org/abs/2103.13951.
- 29. Hoffmann M, Arora P, Groß R, Seidel A, Hörnich BF, Hahn AS, et al. SARS-CoV-2 variants B.1.351 and P.1 escape from neutralizing antibodies. Cell. 2021 Mar;S0092867421003676. pmid:33794143
- 30. Vidal SJ, Collier AY, Yu J, McMahan K, Tostanoski LH, Ventura JD, et al. Correlates of Neutralization Against SARS-CoV-2 Variants of Concern by Early Pandemic Sera. J Virol. 2021 Apr 23;JVI.00404-21, jvi;JVI.00404-21v1. pmid:33893169