Main

Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has spread globally over the past year, infecting an immunologically naive population and causing significant morbidity and mortality. Immunity to SARS-CoV-2 induced either through natural infection or vaccination has been shown to afford a degree of protection against reinfection and/or reduce the risk of clinically significant outcomes. Seropositive recovered subjects have been estimated to have 89% protection from reinfection1, and vaccine efficacies from 50 to 95% have been reported2. However, the duration of protective immunity is presently unclear, primary immune responses are inevitably waning3,4,5, and there is ongoing transmission of increasingly concerning viral variants that may escape control by both vaccine-induced and convalescent immune responses6.

A critical challenge at present is to identify the immune correlate(s) of protection from SARS-CoV-2 infection and thereby predict how changes in immunity will be reflected in clinical outcomes. A defined correlate of protection will permit both confidence in opening up economies and facilitate rapid improvements in vaccines and immunotherapies. In influenza infection, for example, a hemagglutination inhibition (HAI) titer of 1:40 is thought to provide 50% protection from influenza infection7 (although estimates range from 1:17 to 1:110, refs. 8,9). This level was established over many years using data from a standardized HAI assay10 applied to serological samples from human challenge and cohort studies. This assay is used to predict vaccine efficacy and to assist in the annual reformulation of seasonal influenza vaccines. At present, however, there are few standardized assays for assessing SARS-CoV-2 immunity, little data comparing immune levels in susceptible versus resistant individuals, and no human challenge model11.

The data currently available for SARS-CoV-2 infection include immunogenicity data from phase 1 and 2 studies of vaccines, and data on protection from preliminary reports from phase 3 studies and from seropositive convalescent individuals (Supplementary Tables 1 and 2). Although antiviral T and B cell memory certainly contribute some degree of protection, strong evidence of a protective role for neutralizing serum antibodies exists. For example, passive transfer of neutralizing antibodies can prevent severe SARS-CoV-2 infection in multiple animal models,12,13 and Regeneron has recently reported similar data in humans14. We therefore focus our studies on in vitro virus neutralization titers reported in studies of vaccinated and convalescent cohorts. Unfortunately, the phase 1 and 2 studies all use different assays for measuring neutralization. Normalization of responses against a convalescent serum standard has been suggested to provide greater comparability between the results from different assays15. Although all studies compare immune responses after vaccination against the responses in convalescent individuals, the definition of convalescence is not standardized across studies. Similarly, among phase 3 studies, the timeframes of study and the case definitions of infection also vary (Supplementary Table 2). Recognizing these limitations, our aim was to investigate the relationship between vaccine immunogenicity and protection.

Results

Identification of neutralization titer as a correlate of immune protection

To compare neutralization titers across studies, we determined the mean and standard deviation (on a log scale) of the neutralization titer in published data from seven vaccine studies (mRNA-1273, NVX-CoV2373, BNT162b2, rAd26-S+rAd5-S, ChAdOx1 nCoV-19, Ad26.COV2.S and CoronaVac) and one convalescent study3,16,17,18,19,20,21,22 (Supplementary Table 1). Because of the different assays used in each study, neutralization titers were normalized to the mean convalescent titer using the same assay in the same study (noting that the definition of convalescence was also not standardized across studies and a variable number of convalescent samples are analyzed in each study). We then compared this normalized neutralization level in each study against the corresponding protective efficacy reported from the seven phase 3 clinical trials19,23,24,25,26,27,28,29 (detailed in Supplementary Table 2). Despite the known inconsistencies between studies, comparison of normalized neutralization levels and vaccine efficacy demonstrates a remarkably strong non-linear relationship between mean neutralization level and the reported protection across different vaccines (Spearman r = 0.905; P = 0.0046) (Fig. 1a).

Fig. 1: Understanding the relationship between neutralization and protection.
figure 1

a, Relationship between neutralization level and protection from SARS-CoV-2 infection. The reported mean neutralization level from phase 1 and 2 trials and the protective efficacy from phase 3 trials for seven vaccines, as well as the protection observed in a seropositive convalescent cohort, are shown (details of data sources are given in Supplementary Tables 1 and 2). The 95% CIs are indicated as vertical and as horizontal whiskers. The red solid line indicates the best fit of the logistic model and the red shading indicates the 95% predictive interval of the model. The mean neutralization level and protective efficacy of the Covaxin vaccine are indicated as a green circle (data from this study were available only after modeling was complete and did not contribute to fitting). b, Schematic illustration of the logistic approach to identifying the protective neutralization level. The data for each study include the distribution of the measured in vitro neutralization titer against SARS-CoV-2 in vaccinated or convalescent subjects (as a proportion of the mean titer in convalescent subjects (dashed line)) (blue/red bell curve), accompanied by a level of protective efficacy for the same regimen. The efficacy is illustrated by the proportions of the bell curve ‘protected’ (blue) and ‘susceptible’ (red) for individual studies. The modeling fits the optimal 50% protective neutralization level (blue solid line, the shaded area indicates the 95% CI) that best estimates the correct levels of protection observed across the different studies. c, Predictions of the leave-one-out analysis. Modeling was repeated multiple times using all potential sets of the seven vaccination studies and the convalescent study to predict the efficacy of the eighth study. The diagonal dashed line indicates the position of a 1:1 correlation (i.e., the relationship if the model were completely accurate). The horizontal whiskers indicate 95% CIs and the vertical whiskers indicate 95% predictive intervals.

Estimation of the protective neutralization level against COVID-19

To further dissect the relationship between immunogenicity and protection in SARS-CoV-2 we considered the parallels with previous approaches to estimating a ‘50% protective titer’ in influenza infection. These historic studies in influenza involved comparison of HAI titers in infected versus uninfected subjects (in either natural infection or human challenge studies) and used logistic or receiver operating characteristic approaches to identify an HAI titer that provided protection7,8,9,30,31. We adapted these approaches to analyze the existing data on reported ‘mean neutralization level’ in different studies and the observed level of protection from infection (details of statistical methods are provided in the Methods).

We first fitted a logistic model to estimate the ‘50% protective neutralization level’ (across all studies) that best predicted the protective effect observed in each study (consistent with the use of a logistic function to model protection in influenza serological studies30,31). We found that this model provided a good explanation of the relationship between mean neutralization level and protection across the studies, and determined that the estimated 50% protective neutralization level was 20.2% (95% confidence interval (CI) = 14.4–28.4%) of the mean convalescent level (Fig. 1a,b). Given that different neutralization assays were used for each study11 (see above and Supplementary Table 1), a 50% protective neutralization level equivalent to 20% of the mean titer in the convalescent subjects equates to a measured in vitro neutralization titer of between 1:10 and 1:30 in most of the assays reported (although up to 1:200 in one assay), or we estimate approximately 54 international units (IU)/ml (95% CI 30–96 IU/ml) (Supplementary Table 4). Given that the model is dependent on the mean and distribution of neutralization levels, we also estimated these using different approaches, which led to similar estimates (see Methods and Extended Data Fig. 1).

To relax the assumption that neutralization levels are normally distributed in the above model, we also estimated the protective level using a distribution-free approach and applied this to the raw data for individual neutralization levels reported in the studies. We refer to this as the ‘protective neutralization classification model’. Although this approach may be slightly unrealistic in applying a protected or unprotected cut-off in a binary fashion (unlike the logistic approach), it has the advantage of being independent of any assumptions of the distribution of neutralization titers. Using this classification approach the estimated protective threshold was 28.6% (95% CI = 19.2–29.2%) of the mean convalescent level. As expected, the estimated protective level using the classification method was slightly higher than the 50% protective level estimated using the logistic method, given that the classification method essentially estimates a level of 100% protection instead of 50% protection.

This analysis suggests that the mean in vitro neutralization level of a vaccine measured early after vaccination is predictive of the subsequent protective efficacy measured in phase 3 trials, and estimates that the 50% neutralization level for SARS-CoV-2 is approximately 20% of the mean convalescent titer. To test the potential utility of this in predicting the protective efficacy of an unknown vaccine, we repeated our analysis using a leave-one-out approach. That is, we repeated our analysis by removing one of the datasets and fitting the model to the remaining seven vaccine or convalescent studies. We then used the parameters obtained from this fitting to predict the efficacy of the eighth (removed) dataset. We repeated this by removing each dataset one at a time. Figure 1c shows the results of using the logistic model of protection to predict the efficacy of each vaccine from the results of the other seven. In addition, after fitting the model to the data for eight vaccine or convalescent studies, the phase 3 efficacy results of another vaccine (BBV152) were released in a press release on 3 March 2021 (ref. 32). Using the observed neutralization level (a mean of 79.2% of the convalescent titer in that study33 (Supplementary Table 1), the predicted efficacy of the new vaccine was 79.6% (95% predictive interval: 76.2–83.0%), which is in very close agreement with the reported efficacy of 80.6% (ref. 32) and suggests good predictive value of the model (Fig. 1a).

Modeling of the duration of immune protection after vaccination

Recent studies have identified a decline in neutralization titer with time for up to 8 months after SARS-CoV-2 infection3,4,5. A major question is whether vaccine-induced responses may be more durable than those measured following infection. Limited studies have analyzed the trajectory of neutralization titer after vaccination34. To compare decay in neutralization titer we fitted a model of exponential decay to equivalent time periods in data from either convalescent3 or messenger RNA vaccination34 cohorts. Comparing neutralization titers measured between 26 and 115 d (the longest time period available for vaccination) after either mRNA-based vaccination34 or symptom onset for post-infection sera3, we found similar half-lives (65 d versus 58 d, respectively; P = 0.88, likelihood ratio test; Extended Data Fig. 2a). Although this comparison relies on limited data, it suggests that decay of vaccine-induced neutralization is similar to that observed after natural SARS-CoV-2 infection.

If the relationship between neutralization level and protection that we observe cross-sectionally between different vaccines is maintained over time, we can use the observed relationship between neutralization and protection to predict how the observed waning of neutralization titers might affect vaccine effectiveness. Important caveats to this modeling are that (1) it assumes that neutralization is a major mechanism of protection (or that the mechanism of protection remains correlated with neutralization over time), although B cell memory and T cell responses may be more durable3,4,5,35 and may play a larger role later after infection or vaccination; (2) it applies the decay of neutralization observed in convalescence to the vaccine data; and (3) it assumes that the decay in titer is the same regardless of the initial starting titer (whereas others have suggested either faster36 or slower37 decay for higher initial levels). These limitations notwithstanding, we analyzed the half-life of neutralization titer using published data from a study of convalescent subjects up to 8 months after infection (using a mixed-effects model with censoring) and estimated that neutralization titer decayed with a half-life of 108 d over this period (Extended Data Fig. 2b)5. We also tried alternative models of decay such as bi-exponential decay (consistent with rapid early decay slowing over time), but found that these did not provide a better fit to the available data. We then used this half-life of 108 d to model the decay of neutralization and protection over the first 250 d after vaccination (Fig. 2a). Our model predicts that even if the waning of neutralization titer over time is the same for different vaccines, this decay will have non-linear effects on the level of protection from SARS-CoV-2 infection, depending on initial vaccine efficacy. For example, a vaccine starting with an initial efficacy of 95% would be expected to maintain 77% efficacy by 250 d. However, a response starting with an initial efficacy of 70% would be predicted to drop to 33% efficacy after 250 d. This analysis can also be used to estimate how long it would take a response of a given initial efficacy to drop to 50% (or 70%) efficacy, which may be useful in predicting the time until boosting is required to maintain a minimum level of efficacy (Fig. 2b). Clearly, data generated from standardized assays are needed to track the long-term decay of post-vaccination immune responses and their relationship to clinical protection. However, this model provides a framework that can be adapted to predict outcomes as further immune and protection data become available. Indeed, if a disconnect between the decay of neutralization titer and protection is observed, this may be a direct pointer to the role of non-neutralizing responses in protection.

Fig. 2: The effects of waning neutralization titer on protection.
figure 2

a, Prediction of the effects of declining neutralization titer. Assuming that the observed relationship between neutralization level and protection is consistent over time, we estimate the decline in efficacy for vaccines with different levels of initial efficacy. The model assumes a half-life of the neutralization titer of 108 d over the first 250 d (as observed in a convalescent cohort5). b, Modeling of the time for efficacy to drop to 70% (red line) or 50% (blue line) for scenarios with different initial efficacy. For example, for a group starting with an initial protective efficacy of 90%, the model predicts that 70% efficacy will be reached after 201 d and 50% efficacy will not be reached before 250 d. c, Estimation of the impact of viral antigenic variation on vaccine efficacy. In vitro studies have shown that neutralization titers against some SARS-CoV-2 variants are reduced compared with titers against wild-type virus. If the relationship between neutralization and protection remains constant, we can predict the difference in protective efficacy against wild-type and variant viruses from the difference in neutralization level. The dashed line indicates equal protection against wild-type and variant strains. Details of the data and modeling are provided in the Methods.

Modeling of the effect of viral variation on immune protection

In addition to the effect of declining neutralization titer over time, reduced neutralization titers and reduced vaccine efficacy to different viral variants have also been observed6,38,39,40,41. For example, it has been reported that the neutralization titer against the B.1.351 variant in vaccinated individuals is between 7.6-fold and 9-fold lower compared with the early Wuhan-related Victoria variant42. Our model predicts that a lower neutralization titer against a variant of concern will have a larger effect on vaccines for which protective efficacy against the wild-type virus was lower (Fig. 2c). For example, a fivefold lower neutralization titer is predicted to reduce efficacy from 95% to 77% in a high efficacy vaccine, but from 70% to 32% for a vaccine with lower initial efficacy.

Estimation of the 50% protective level against severe infection

The analysis above investigates vaccine (and convalescent) protection against symptomatic SARS-CoV-2 infection (using the definitions provided in the different phase 3 and convalescent studies, Supplementary Table 2). However, it is thought that the immune response may provide greater protection from severe infection than from mild infection. To investigate this, we also analyzed data on the observed level of protection from severe infection when these were available. It is important to note that as with symptomatic infection, the definition of severe infection was not consistent across studies (the definitions for each study are detailed in Supplementary Table 3). Given that there have been under 100 severe infections reported across all the phase 3 trials combined, the 95% confidence intervals on the level of protection from severe infection are broad. The neutralization level for 50% protection from severe infection was 3.0% of the mean convalescent level (95% CI = 0.71–13%), which was significantly lower than the 20% level required for protection from any symptomatic infection (P = 0.00039, likelihood ratio test, Supplementary Table 5) (Fig. 3a). An important caveat to this analysis is the implicit assumption that neutralization titer itself confers protection from severe infection. However, it is possible that T cell responses or recall of memory B cell responses may also be important in protection from severe disease43,44,45,46.

Fig. 3: Protection from severe infection.
figure 3

a, The predicted relationship between efficacy against any symptomatic SARS-CoV-2 infection and the efficacy against severe infection. The black line indicates the best fit model for the relationship between protection against any versus severe SARS-CoV-2 infection. The shaded areas indicate the 95% CIs. Efficacy against severe infection was calculated using a threshold that was 0.15 times lower than that for mild infection (95% CI = 0.036–0.65) (see Methods and Supplementary Table 5). b, Extrapolation of the decay of neutralization titers over time. This model uses the estimated half-life of SARS-CoV-2 neutralization titer in convalescent subjects of 108 d over the first 250 d5, after which the decay decreases linearly until it achieves a 10-year half-life (consistent with the long-term stability of antibody responses seen after other vaccines47,48). We simulate three scenarios, with decay of neutralization taking 1 year (blue dashed line), 1.5 years (purple dashed line) or 2 years (red dashed line) to slow to a 10-year half-life. For different initial starting levels the model projects the decay in neutralization titer over the subsequent 1,000 d (the gray shaded area indicates projections beyond the currently available data). The purple shaded region indicates being below the 50% protective titer for any symptomatic SARS-CoV-2 infection, and the orange shaded region indicates being below the 50% protective titer for severe SARS-CoV-2 infection. The model illustrates that, depending on the initial neutralization level, individuals may maintain protection from severe infection while becoming susceptible to mild infection (that is, with neutralization levels remaining in the purple shaded region). c, Extrapolation of the trajectory of protection for groups with different starting levels of protection. The model uses the same assumptions for the rate of immune decay as discussed in b. The projections beyond 250 d (gray shaded region) rely on an assumption of how the decay in SARS-CoV-2 neutralization titer will slow over time. In addition, the modeling projects only how decay in neutralization is predicted to affect protection. Other mechanisms of immune protection may play important roles in providing long-term protection that are not captured in this simulation.

Prediction of the potential for long-term protection

The estimated neutralization level for protection from severe infection is approximately sixfold lower than the level required to protect from any symptomatic infection. Thus, a higher level of protection against severe infection is expected for any given level of vaccine efficacy against mild SARS-CoV-2 infection. Assuming that this relationship remains constant over time, it appears probable that immunity to severe infection may be much more durable than overall immunity to any infection. Long-term studies of antibody responses to vaccinia, measles, mumps or rubella suggest that these responses generally stabilize with half-lives of >10 years47,48. We therefore projected beyond the reported decay of SARS-CoV-2 responses (out to 8 months after infection5), assuming that after 8 months following the infection the decay rate will slow down. We modeled the decay rate of the neutralization titer, assuming that it slowed linearly to a 10-year half-life over 1, 1.5 or 2 years (details in Methods). This analysis predicts that even without immune boosting, a significant proportion of individuals may maintain long-term protection from severe infection by an antigenically similar strain, even though they may become susceptible to mild infection (Fig. 3b,c).

Discussion

Understanding the relationship between measured immunity and clinical protection from SARS-CoV-2 infection is urgently needed to plan the next steps in the COVID-19 vaccine program. Placebo-controlled vaccine studies are unlikely to be possible in the development of next-generation vaccines, and therefore correlates of immunity will become increasingly important in planning booster doses of vaccine, prioritizing next-generation vaccine development, and powering efficacy studies49. Our work uses available data on immune responses and protection to model both the protective titer and the long-term behavior of SARS-CoV-2 immunity. It suggests that neutralization titer will be an important predictor of vaccine efficacy in the future as new vaccines emerge. The model also predicts that immune protection from infection may wane with time as neutralization levels decline, and that booster immunization may be required within a year. However, protection from severe infection may be considerably more durable given that lower levels of response may be required or alternative responses (such as cellular immune responses) may play a more prominent role.

As discussed above, a major caveat of our estimate of the relative protective level of antibodies in SARS-CoV-2 infection is that it includes aggregation of data collected from diverse neutralization assays and clinical trial designs (Supplementary Tables 1 and 2). It is hoped that in the future a standardized neutralization assay may be developed and utilized, which will allow direct comparison of neutralization titers and further refinement of these analyses11. In addition, the development of standardized trials and case definitions is necessary, given that differences in classification (particularly of severe disease) may prove to be important (Supplementary Table 3). The association of neutralization with protection across these studies does not prove that neutralizing antibodies are necessarily mechanistic in mediating protection. It is possible that neutralization is correlated with other immune responses, leading to an apparent association (as has been suggested for the use of HAI titer in influenza50,51,52). Thus, it will be important to study other responses such as T cell responses or B cell memory responses as additional potential correlates of protection. Another important refinement of this approach would be to have standardized measures of other serological and cellular responses to infection, to identify if any of these provide a better predictive value than neutralization. However, despite these limitations, our work identifies neutralizing antibodies as an immune correlate of protection and provides a quantitative prediction of the link between neutralizing antibody levels and clinical protection based on evidence from clinical trials and convalescent cohort studies. An important factor that is not explored in this analysis is the role of age in immunogenicity, in the durability of neutralizing responses, and in clinical protection, given that vaccine studies tend to exclude older individuals.

Our method for estimating protective neutralization titer for COVID-19 uses a very similar modeling approach to that previously used to estimate the protective titer for influenza infection7,8,9,30,31. However, a major difference is the data used in the models. Our approach utilizes the wide range of immunogenicity and protective efficacy across different vaccines to estimate the 50% protective titer. By contrast, studies in influenza infection use data from the HAI titer of individual subjects and their subsequent risk of infection in either cohort or human challenge studies to estimate protection7,8,9,30,31. It will be important to prospectively validate our results using a similar analysis of individual risk of COVID-19 infection in future studies.

Our results are consistent with studies of both influenza and seasonal coronavirus infection, for which reinfection is possible 1 year after the initial infection, although it usually results in mild infection53,54. Similarly, after influenza virus vaccination, protective efficacy is thought to decline by around 7% per month55. Our modeling and predictions are based on a number of assumptions regarding the mechanisms and rate of loss of immunity. Important priorities for the field are the development of standardized assays to measure neutralization and other immune responses, as well as standardized clinical trial protocols. These data will allow further testing and validation of other potential immune correlates of protection. However, our study develops a modeling framework for integrating available, if imperfect, data from vaccination and convalescent studies to provide a tool for predicting the uncertain future of SARS-CoV-2 immunity.

Methods

Data extraction

When possible, data values were used as directly stated in publications. In addition, when necessary, raw data were directly extracted from the original publications using an online digitizer tool (https://automeris.io/WebPlotDigitizer/, version 4.4). All sources of data are described in Supplementary Tables 13.

Statistical methods

Estimation of the standard deviation of neutralization titers

Neutralization titers were extracted for each study (as above) and used to determine the standard deviation of the log10-transformed neutralization titers for each study. The standard deviation for each study had to consider that some measurements of neutralization titer fell below the limit of detection (LOD) of that study assay (LOD varied for each study, Supplementary Table 4). To remove the effect of LOD censoring on estimates of the standard deviation of neutralization titers we used a censored regression model to fit the distribution of the neutralization titers for each study. The likelihood function is given by

$${\cal{L}}({\boldsymbol{D}}_{\boldsymbol{s}}|\mu _{cens},\sigma _{cens},L_s) = \mathop {\prod }\limits_{n_i \in {\boldsymbol{D}}_{\boldsymbol{s}}} f\left( {n_i|\mu _{cens},\sigma _{cens}} \right)^{1 - I_i}F\left( {L_s|\mu _{cens},\sigma _{cens}} \right)^{I_i}$$
(1)

where Ds is a vector of all of the log10-neutralization titers, ni, for the study s. The function f is the probability function of a normal distribution with mean μcens and standard deviation σcens, and F is the cumulative density function of the same distribution. The LOD of the assay for study s is given by Ls. The index variable Ii is 1 when ni ≤ Ls and 0 otherwise. The negative log of this likelihood function was minimized in R using the built-in optimizer nlm to estimate the mean and standard deviation of the log10-transformed neutralization titers after factoring in the LOD. When no LOD was reported or when all of the values were above the LOD, Ls was set to negative infinity (−Inf).

Pooled standard deviation

Given the differing sample size of the neutralization data for each study, the accuracy of estimations of the standard deviation for each study varied considerably. Therefore, despite finding some limited evidence of a difference in the standard deviation between each study (P = 0.049, Fligner–Killeen test), we combined all extracted data and calculated the standard deviation of the pooled data. A test of normality showed that the pooled neutralization data were consistent with a normal distribution (P = 0.26, Shapiro–Wilk test). To estimate a pooled standard deviation, we first centered the neutralization data for each study at the reported mean of the neutralization titers for that study. The LOD given for each study was also adjusted in the same way. This provided a combined dataset of neutralization titers from all studies, which was fitted, using the likelihood model in equation (1), to the pooled data to produce an estimate of the standard deviation of the pooled data. All estimated standard deviations are reported in Supplementary Table 4.

Modeling of the relationship between neutralization and protection

In the above sections we used neutralization titer information from each vaccine and from convalescent individuals to estimate the distribution of the neutralization titers for each study. However, as discussed in the main text, owing to the diversity of assays used to assess neutralization in each study (Supplementary Table 1), from this point forward we normalized the neutralization titers in each study by the mean of the neutralization titer in the corresponding convalescent individuals from the same study. Also, in all of the analyses below we use the log10 transform of the normalized neutralization titers. For simplicity, in the remainder of this paper we refer to these log10-transformed normalized neutralization titers as the ‘neutralization levels’.

Logistic method

To model the relationship between the neutralization level of antibodies collected from individuals after vaccination (or during convalescence) and the protection from COVID-19 we assumed a logistic relationship between neutralization level and protective efficacy, such that,

$$E_I\left( {n|n_{50},k} \right) = \frac{1}{{1 + e^{ - k(n - n_{50})}}},$$
(2)

where EI is the protective efficacy of an individual given the neutralization level n (note the definition of neutralization level above). The parameter n50 is the neutralization level at which an individual will have a 50% protective efficacy (that is, half the chance of being infected compared with an unvaccinated person). The steepness of this relationship between protective efficacy and neutralization level is determined by the parameter k.

We assume that a vaccine (or prior exposure) will induce a (normal) distribution of neutralization levels (n) in a population with some mean μs and standard deviation σs. The mean neutralization level for each study, μs, is the difference between the log10-transformed mean neutralization titers for vaccinated and for convalescent individuals in that study. The standard deviation, σs, is the standard deviation for vaccinated individuals in that study only. (Note that the distribution of neutralization level for convalescent individuals has a mean of zero by definition (that is, the log of the mean of neutralization titers for convalescent individuals normalized by itself).) Therefore the proportion of the vaccinated population for a study, s, that will be protected will be given by

$$P\left( {n_{50},k,\mu _s,\sigma _s} \right) = \mathop {\smallint }\limits_{ - \infty }^\infty E_I\left( {n|n_{50},k} \right)f(n|\mu _s,\sigma _s)dn,$$
(3)

where f is the probability density function of a normal distribution and EI is the logistic function in equation (2). The above integral is the so-called logistic-normal integral and the mean of the logit-normal distribution, which has no analytical solution56. Therefore, we use a simple numerical approximation (left Riemann sum).

Fitting of the logistic model and confidence intervals

The above model of protection was fitted to data on the protective efficacy of vaccines from phase 3 (and another large cohort study of convalescent individuals). For each vaccine and convalescence study the number of control (unvaccinated, or placebo or naive) individuals enrolled (\(N_s^c\)), the number of control individuals infected (\(I_s^c\)), the number of vaccinated (previously exposed) individuals enrolled (\(N_s^v\)), and the number of vaccinated individuals infected (\(I_s^v\)) were used in the fitting of the model. The likelihood of observing the number of infected individuals in the control and vaccinated groups for each study, given some parameters, is

$$\begin{array}{l}{\cal{L}}_s\left( {N_s^c,I_s^c,N_s^v,I_s^v,\mu _s,\sigma _s|n_{50},k,b_s} \right) = Bi\left( {N_s^c,I_s^c,b_s} \right)\\Bi\left( {N_s^v,I_s^v,b_s\left( {1 - P(n_{50},k,\mu _s,\sigma _s)} \right)} \right),\end{array}$$
(4)

where bs is the probability of an unvaccinated control individual becoming infected in study s (baseline risk), \(b_s\left( {1 - P(n_{50},k,\mu _s,\sigma _s)} \right)\) is the probability of infection in the vaccination group (see equation 3) and Bi(N, K, p) is the binomial probability mass function of the probability of K events from a sample size of N, for which each event has a probability p of occurring. However, we wish to fit all studies simultaneously, and so the total likelihood of observing the data in all studies, given some parameters, is

$$\begin{array}{l}{\cal{L}}\left( {{\boldsymbol{N}}^{\boldsymbol{c}},{\boldsymbol{I}}^{\boldsymbol{c}},{\boldsymbol{N}}^{\boldsymbol{v}},{\boldsymbol{I}}^{\boldsymbol{v}},{\boldsymbol{\mu }},{\boldsymbol{\sigma }}|n_{50},k,{\boldsymbol{b}}} \right) = \mathop {\prod }\limits_{\forall s} Bi\left( {N_s^c,I_s^c,b_s} \right)\\ Bi\left( {N_s^v,I_s^v,b_s\left( {1 - P(n_{50},k,\mu _s,\sigma _s)} \right)} \right),\end{array}$$
(5)

where Nc, Ic, Nv, Iv, μ, σ are vectors containing the data \(N_s^c,I_s^c,N_s^v,I_s^v,\mu _s,\sigma _s\) for each study s, and b is a vector of the baseline risk parameters bs for each study. The best-fitting parameters n50, k and b were found using the nlm optimizer in R by minimizing \(- \log \left( {\cal{L}} \right)\). The standard error (s.e.) of these estimates was estimated using the hessian H output from this function and the formula \({\mathrm{s.e.}} = \sqrt {{\mathrm{diag}}(H^{ - 1})}\). The 95% CIs were taken as ±1.95 × s.e. of the estimated parameters.

The variable μs is the mean neutralization level, which can be calculated in two ways; first, by dividing the geometric mean of the neutralization titers in vaccinated individuals by the geometric mean of the neutralization titer in convalescent individuals in the same study. This is, in most cases, the ratio of two values directly reported in the immunogenicity studies. However, this approach does not account for situations in which the neutralization assay had neutralization titers below the LOD, therefore we also used a second method, in which we estimated this value by extracting the neutralization titers from the figures in each immunogenicity study (Supplementary Table 1) and computing the mean neutralization titer for vaccinated and convalescent individuals in each study using censoring regression (equation 1). Additionally, although it was in principle possible to compute the standard deviation of neutralization levels for each study (as above), these appeared somewhat confounded by the varying numbers of individuals between studies, hence we fitted the above model using (1) the standard deviation estimates for each study, (2) one standard deviation from one larger study to which we had direct access to raw data3 (that is, no manual data extraction required), and (3) an estimate of the standard deviation for all studies pooled together. The two different methods of estimating the mean neutralization level for each study and the three methods of estimating the standard deviation of the neutralization levels give rise to six versions of the above model. All of these versions of the model were fitted, and the estimated protective levels were very similar (Extended Data Fig. 1).

Protective neutralization classification model

The above modeling approach assumed that neutralization levels were normally distributed. Here, we present a method for determining a protective threshold that is free of assumptions regarding the distribution of neutralization levels. This model assumes that there is a protective neutralization level, T, above which individuals will be protected from infection and below which individuals will be susceptible. The protective efficacies observed in phase 3 clinical trials of vaccinated individuals (and another large cohort study of convalescent individuals1; Supplementary Table 2) are denoted by Es. These represent the proportion of individuals in each study, s, who should possess a neutralization level above the protective threshold. It follows then that the number of individuals above the protective threshold in study s is a function of T, which we denote Ks(T). Therefore, the probability of observing Ks(T) individuals above the protective threshold, given that there were Ns individuals in the immunogenicity study (which are much smaller than the phase 3 studies), is given by

$$P(K_s(T)|N_s,T) = Bi(N_s,K_s(T),E_s),$$
(6)

where Bi is a binomial distribution. Note that

$$K_s\left( T \right) = \mathop {\sum }\limits_{n_i \in {\boldsymbol{D}}_s} H(n_i - T),$$
(7)

and Ns = |Ds|, such that H is the heavy-side step function taking the value 1 when ni − T > 0, or 0 otherwise, and |Ds| denotes the size of set Ds (that is, the number of neutralization levels measured). To determine one protective threshold using the results of all efficacy studies in this paper, we construct a likelihood function

$${\cal{L}}\left( {{\boldsymbol{D}}|T} \right) = \mathop {\prod }\limits_{{\boldsymbol{D}}_{\boldsymbol{s}} \in {\boldsymbol{D}}} Bi(N_s,K_s(T),E_s),$$
(8)

where D is the set of vectors of the neutralization levels from each study. Note that this likelihood function is discontinuous as the threshold T is varied. Therefore, we evaluate this likelihood function with the threshold T set equal to all observed neutralization levels ni across all studies, and find the value of T that maximizes this likelihood (Extended Data Fig. 3). This method determines a protective level at which the proportion of individuals with neutralization levels above the threshold is in greatest agreement with the observed protective efficacy of that vaccine.

Equation (8) is the likelihood function that should be adopted when neutralization measurements are not affected by an LOD. In the case that some neutralization levels are below the LOD, the likelihood function is adjusted as follows:

$${\cal{L}}\left( {{\boldsymbol{D}}|T} \right) = \mathop {\prod }\limits_{{\boldsymbol{D}}_{\boldsymbol{s}} \in {\boldsymbol{D}}} Bi\left( {N_s,K_s\left( T \right),E_s} \right)^{1 - J_s} \times Q\left( {N_s,C_s,1 - E_s} \right)^{J_s},$$
(9)

where Js is an index that takes the value 1 when the LOD of study s is above the threshold T and at least one value is censored, or 0 otherwise. Cs is the number of censored values in study s and Q is the cumulative binomial distribution function. This later term considers the probability that as many as all of the censored values were below the threshold T given the protective efficacy of the study Es.

To determine the 95% CIs for the estimated protective neutralization level, a bootstrapping approach was used in which the neutralization levels were resampled 1,000 times at random with replacement. Resampling was performed so as to preserve the total number of neutralization levels in each study. These randomly generated samples of the original data were then fitted in the same way as described above, which generated 1,000 corresponding estimates of the protective neutralization level. The 95% CI was calculated as the 2.5 and 97.5 percentiles of these 1,000 estimates of the protective neutralization level.

Assessment of the predictive ability of the model

To determine the ability of the model to predict the efficacy of a vaccine, we performed a leave-one-out analysis in which we systematically excluded one of the vaccine studies (or the convalescent study) and performed the same model-fitting procedure described above. Using the model fitted on the subset of the studies we then estimated the efficacy of the vaccine that was left out from the fitted model. This leave-one-out analysis was performed for all versions of the logistic model (that is, using the six methods of estimating the mean neutralization level and standard deviation outlined above). The predicted efficacy for each vaccine and convalescence obtained while leaving the study out are plotted against the reported efficacy in Fig. 1c. Also, the 50% protective level that was estimated each time a study was left out of the analysis provides a metric of the sensitivity of the model to the inclusion of each study. Note that the exclusion of any of the studies did not greatly influence the estimate of the 50% protective level (Extended Data Fig. 1).

Error bars and regions in efficacy and neutralization

In Fig. 1a there are both horizontal and vertical error bars as well a 95% predictive interval for the fitted model. The vertical error bars indicate the 95% CIs for the efficacy estimates for each study, these were calculated using the Katz-log method specified in supplementary table 1 of ref. 57. The horizontal error bars indicate 95% CIs in the difference of the mean of the (log10) neutralization titers for vaccinated and convalescent individuals in each study. That is, these represent

$$\pm 1.96 \times \sqrt {\frac{{\sigma _v^2}}{{n_v}} + \frac{{\sigma _c^2}}{{n_c}}},$$
(10)

where σv and σc are the standard deviation in the log10 of the neutralization titers for vaccinated and convalescent individuals, respectively, and were estimated as described in the section on estimating the standard deviation of neutralization titers above. nv and nc are the number of vaccinated and convalescent individuals who were included for each study, respectively. The 95% predictive interval in Fig. 1a was calculated using the delta method58.

Comparison of the protective level in mild and severe infection

We also tested if the protective neutralization level was different between mild and severe infection by fitting the combined dataset with two different mathematical models. The simplest model assumes that we could use the same protective level in both severe and mild infection (that is, the shared steepness parameter (k) and 50% protective level (n50) in the logistic model (above)); and the alternative model uses different protective level parameters while we constrained the model to have the same k in severe and mild infection (equation 3). We used both the Akaike information criterion and a likelihood ratio test to determine which model provided the best fit of the dataset for both severe COVID-19 cases and all COVID-19 cases reported (Supplementary Table 5).

Modeling of the decay of neutralization and the effects of antigenic variation

Comparison of decay in convalescence and vaccination

A number of studies have analyzed the decay in neutralization titer in convalescent subjects. These studies have generally shown a rapid early decay that slows with time3,4,5,59,60. We identified one study by Widge et al.34 in which the time course of neutralization titer after mRNA vaccination was able to be analyzed. That study measured the decay of neutralization titer out to 115 d. To compare the half-life of decay of the neutralization titer in vaccinated versus convalescent cohorts we analyzed the decay in that vaccination study compared with a previously published study of convalescent neutralization titer3, restricting the convalescent data to data collected within 115 d (Extended Data Fig. 2a).

We compared the decay rates using linear mixed-effects modeling and censoring values below the LOD61, by treating ‘vaccination group’ (vaccinated or convalescent) as a binary variable. The statistical significance of ‘vaccination group’ as a covariate (whether it was significantly different from zero) was calculated using the likelihood ratio test. Note that this restricted convalescent time course was used only so that we could compare decay in vaccination and convalescence over a similar time course (time-limited by the vaccination data).

The half-life of neutralization titers in convalescent subjects was estimated over a longer time course of 240 d using the same mixed-effects modeling approach described above, but applied to the neutralization data reported by Dan et al.5 in supplementary data file 1 of their paper. This is the estimate that was used in the predictive model presented in Figs. 2 and 3.

Predicting the loss of efficacy as neutralization wanes and that due to variants

The decay in efficacy with time (Fig. 2a,b) was modeled by reducing the neutralization levels at a rate corresponding to a half-life of 108 d and recalculating the efficacy given this new distribution of neutralization levels, using equation (3). The efficacy against a SARS-CoV-2 variant, given an associated decrease in neutralization (Fig. 2c), was calculated by reducing the mean neutralization level by a factor of 2, 5 or 10 and using equation (3). Efficacy against severe infection was similarly calculated using the same approach as above, but using the 50% protective level associated with the severe threshold (Supplementary Table 5), which was a factor of 0.15 lower than that for mild infection (95% CI = 0.036–0.65) (Fig. 3a,b). We also extrapolated the decay of the neutralization level beyond the current data, assuming that neutralization levels decay with a half-life of 108 d up until d 250 and that this decay rate slows linearly to a 10-year half-life over 1, 1.5 or 2 years (Fig. 3c).

Ethics statement

This work was approved by the University of NSW Sydney Human Research Ethics Committee (approval HC200242).

Reporting Summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.