An Individualized Risk Calculator for Research in Prodromal Psychosis

Given limitations of available treatments for schizophrenia, with most patients showing substantial deficits in social and occupational functioning throughout life, there is considerable interest in developing preventive approaches to psychotic disorders (1). Ascertainment of individuals at greatest risk is crucial to these efforts. For the majority of patients, onset of fully psychotic symptoms is preceded by the emergence of subtler changes in belief, thought, and perception that appear to represent attenuated forms of delusions, formal thought disorder, and hallucinations, respectively. Among individuals 12–35 years old with a recent onset of such symptoms (termed clinical high-risk cases), approximately 20%−35% develop fully psychotic symptoms over a 2-year period, an incidence rate that is more than 100 times larger than in the same age band in the general population (2). Furthermore, it appears that the clinical high-risk criteria are sensitive to an imminent risk for onset, as most of the conversions occur during the first year after ascertainment, with a decelerating conversion rate thereafter (3).

Although clinical high-risk criteria have been validated in epidemiological studies as sensitive to conversion risk, their utility in individual decision making is currently limited, given that 65%–80% of cases ascertained by these methods do not convert to psychosis within a 2-year time frame. About a dozen studies have examined combinations of clinical and demographic variables to determine whether prediction of psychosis can be enhanced beyond the 20%−35% risk associated with clinical high-risk status (4). Multivariate algorithms requiring particular combinations of symptoms and demographic factors achieve relatively high positive predictive values and specificity (e.g., in the 50%−70% range) but low sensitivity (e.g., in the 10%−30% range) (3). There is consistency among studies in showing (unsurprisingly) that greater severity of the psychosis-risk symptoms at baseline is the best predictor of conversion; nevertheless, the most predictive multivariate profiles vary across studies (as summarized in reference 4). Although it should be noted that few studies have attempted direct replication of one another’s risk algorithms, this pattern suggests heterogeneity among profiles of clinical and demographic risk indicators among patients who convert to psychosis.

To maximize clinical utility, we require an approach that can be applied to scale the risk in an individual patient at the initial clinical contact. Such individualized risk calculation is possible when a large data set on a reference population is available from which risks can be calculated based on one or more predictor variables. Well-performing risk calculators have been developed in numerous somatic disease contexts, including cardiovascular disease and cancer (5–9), where they provide a rationale for clinicians to pursue more or less invasive intervention strategies, based on the level of risk implied by an individual’s profile across a set of risk factors. They also inform patients and their family members, thus helping them make complex treatment decisions.

Here we present such an individualized risk calculator for psychosis, using data from the second phase of the North American Prodrome Longitudinal Study (NAPLS-2). Predictors were chosen a priori based on a review of the literature on psychosis risk prediction in clinical high-risk samples, blindly with respect to the empirical relationships between any of the nominated variables and psychosis outcome within the NAPLS-2 data set. We limited our scope to clinical, cognitive, and demographic measures that are readily obtainable in standard clinical settings. Using time-to-event proportional hazards regression, a risk calculator was generated that calculates risk according to an individual’s values on the included variables. We evaluated the performance of the risk calculator using the concordance index (the Harrell C-index, a measure of overall accuracy, analogous to area under the receiver operating characteristic curve) and assessed the relative importance of each of the included predictor variables.

Method

Study Subjects and Clinical Characterization

The study protocol and consent form were reviewed and approved by the institutional review boards of the eight data collection sites (UCLA, Emory, Beth Israel Deaconess Medical Center, Zucker Hillside Hospital, University of North Carolina, UCSD, Calgary, and Yale). We have previously reported on the methods for evaluation of subjects and data collection (10). Participants were evaluated using the Structured Interview for Prodromal Syndromes (SIPS) (11) and the Structured Clinical Interview for DSM-IV Axis I Disorders (12) by trained interviewers who met high reliability standards (intraclass correlation coefficients, 0.92–0.96) (10). Patients who had ever met DSM-IV criteria for a psychotic disorder, had a history of substance dependence, had a neurological disorder, or had an estimated IQ <70 were excluded. Participants met SIPS criteria (13) for the presence of one or more clinical high-risk syndromes: attenuated psychotic symptoms syndrome, brief intermittent psychotic symptom syndrome, and familial risk and deterioration syndrome.

Follow-up clinical evaluations were scheduled every 6 months after study entry through 2 years. Conversion to psychosis was determined by SIPS criteria, which are designed to operationalize the threshold of delusional ideation or hallucination severity required for a DSM-IV diagnosis of a psychotic disorder. Participants were followed up to the time of conversion to psychosis or the last contact (up to 2 years), the dates for which were recorded, permitting calculation of length of time in the study until conversion or censoring (loss to follow-up).

A total of 743 clinical high-risk patients who met SIPS criteria were enrolled in NAPLS-2 from 2008 to 2013. In the present analysis, we excluded subjects who dropped out of the study before any clinical follow-up was conducted (N=147). The final study cohort consisted of 596 clinical high-risk participants who had at least one follow-up evaluation.

Selection of Predictor Variables

To meet the objective of developing a practical tool for risk prediction, our focus was on demographic, clinical, neurocognitive, and functioning measures that are easily administered in general clinical settings. The number of predictors was limited a priori to eight to ensure that there were a minimum of 10 converters per predictor in the model, which helps mitigate model instability due to overfitting. For the same reason, we avoided including terms for the interactions among the predictors. The NAPLS-2 data set itself was not used to select predictors; doing so would have invalidated the logic of using the underlying data to inform prediction for new cases (i.e., the predictive logic would then be circular). Rather, we evaluated the published literature on psychosis prediction in clinical high-risk samples. Our selection of indicators was based on empirical links to psychosis prediction in two or more prior studies of clinical high-risk cases; there was no attempt to select predictors based on a theoretical model of causes of psychosis or clinical knowledge or intuition. Based on this process, eight variables were chosen for inclusion.

Age at ascertainment was included to help account for variation in age at onset of psychosis (14) and in processes that undergo developmental modification during the age range of our sample (15–17). Greater severity of SIPS items P1 and P2 (unusual thought content and suspiciousness) are strongly predictive of psychosis in clinical high-risk samples (3, 4). Given that the meanings of gradations below the prodromal threshold are likely to be different from those at or above this threshold, these items were modified such that all levels in the nonprodromal range (0–2 on the original scale) were redefined as 0, levels in the prodromal range (3–5 on the original scale) were redefined as 1–3, and psychotic intensity (6 on the original scale) was redefined as 4, and these scores were summed. Several studies have found that slower processing speed and lower verbal learning and memory functioning are predictive of psychosis (18, 19) and in meta-analyses have among the largest effect sizes among converters to psychosis (20). These constructs were represented by scores on the Brief Assessment of Cognition in Schizophrenia symbol coding test (21) and the Hopkins Verbal Learning Test–Revised (sum of trials 1–3) (22), respectively. Many clinical high-risk cases who convert to psychosis show a pronounced decline in social functioning in the year prior to ascertainment (23), measured here using the Global Functioning: Social scale (24). Stressful life events, along with childhood traumas, have been shown to be predictive of psychosis in studies of clinical high-risk samples (25). To represent the former, we aggregated 31 life events designated as negative and potentially relevant to individuals 12–35 years old from the Research Interview Life Events Scale (26), and for the latter, we used the Childhood Trauma and Abuse Scale (27). Family history of psychotic disorder in a first-degree relative is by itself not a robust predictor of psychosis in studies of clinical high-risk samples (3, 4), but it was nevertheless included because it elevates risk by almost 10-fold compared with the general population (28).

Statistical Methods

Risk calculators have been developed to assist health care professionals for a variety of illnesses (5–9) (see http://www.lerner.ccf.org/qhs/risk_calculator/ for examples). Calculators can derive a risk prediction for a particular person from a given set of indicators by querying a multivariate model based on a large sample of similar cases. Through imputation, calculators can accommodate incomplete information on the panel of risk indicators. However, the more complete the information available on a given case, the more powerful they become, with a tighter range of certainty.

We built a multivariate proportional hazards model to predict the likelihood of conversion to psychosis based on each participant’s demographic, cognitive, and clinical characteristics, as defined above. We tested restricted cubic splines in relation to continuous variables; as none were significant, no adjustments were made. As shown in Table 1, there were little or no missing data for age, symptom severity, family history, and social functioning. Cognitive test data were missing in less than 4% of cases, and data regarding stressful life events or traumas were missing in 12%−14% of cases. In order to reduce selection bias and maximize the sample size, missing predictors were multiply imputed with the multivariate imputation by chained equations method before the multivariate regression.

The statistical model was internally validated using 1,000 bootstrap resamples, where the discrimination and calibration performance were evaluated. Harrell’s C-index was used to quantify the discrimination ability for separating psychosis converters and nonconverters, which is analogous to the area under the receiver operating characteristic curve, with a range of 0.5 (no discrimination) to 1 (perfect discrimination), but tailored for censored data (29). A plot of the model-predicted probabilities versus the observed outcomes was used to assess calibration performance.

All statistical analyses were conducted using R, version 3.0.1 (R Core Team, 2013) including the rms and Hmisc packages.

Results

Baseline patient characteristics are summarized in Table 1. There were no significant differences between those who were and were not followed up clinically on any of the predictor variables. Of the 596 participants for whom follow-up data were available, 84 converted to psychosis within the 2-year study period. The mean age of the sample was 18.5 years. Among converters, the mean time from baseline to conversion was 7.3 months, and among nonconverters, the mean follow-up time from baseline to the last contact was 19.1 months. A total of 280 cases were followed up at 24 months without converting, and the remaining “nonconverters” were lost to follow-up at various points between 6 months and 24 months.

The 2-year probability of conversion to psychosis was 0.16 (95% CI=0.13, 0.19). Figure 1 provides frequency distributions of predicted risks for converters and nonconverters. Converters occur at a proportionally higher rate than nonconverters in each successive risk class, beginning at a predicted risk of 0.20. The output of the multivariate proportional hazards model is presented in Table 2. Prodromal symptom severity (SIPS items P1 and P2, modified and summed), decline in social functioning, and verbal learning and memory (Hopkins Verbal Learning Test–Revised scores) were significant predictors, with nonsignificant effects for age at baseline and speed of processing (symbol coding score) (p values, <0.10), although all of these variables were significant in univariate analyses (p values, <0.01). Stressful life events, traumas, and family history of schizophrenia were not significant predictors in univariate or multivariate analyses.

Table 2 provides additional diagnostics of the performance of individual predictor variables. Predictors associated with the largest decreases in the C-index when removed from the model were symptom severity, decline in global social functioning, Hopkins Verbal Learning Test–Revised score, and symbol coding score. Predictors associated with the largest increases in the C-index (i.e., above that of the base model, which included only symptom severity) were symbol coding score, Hopkins Verbal Learning Test–Revised score, decline in social functioning, and age. Family history of psychosis, stressful life events, and traumas did not alter the C-index by more than 0.5% when added to or deleted from the model.

Based on the bootstrap internal validation, the multivariate model achieved a C-index of 0.71. As shown in Figure 2, the calibration plot revealed a high degree of consistency between observed probabilities and model-predicted probabilities of conversion to psychosis within the range of 0.0–0.4, within which 95% of the cases fell (mean=0.18, SD=0.11, median=0.16). Table 3 provides statistics for prediction of actual conversion to psychosis across several thresholds of model-predicted risk. There is a trade-off between the positive predictive value (proportion of cases at the threshold of predicted risk who actually converted) and sensitivity (proportion of actual converters who had predicted risks at that threshold). The positive predictive value is maximal (48.4%) at a threshold of 0.4 of model-predicted risk, but only 17.9% of converters had model-predicted risks at this threshold. Conversely, at a model-predicted risk of 0.2 or higher, the positive predictive value is 28.1%, but with a sensitivity of 66.7%.

An online version of the risk calculator was built to facilitate numeric calculation of the predicted probability of conversion to psychosis (http://riskcalc.org:3838/napls/).

Discussion

The goal of this study was to develop a practical tool for the individualized prediction of psychosis in clinical high-risk patients. A well-performing risk calculator was generated from the NAPLS-2 cohort data using a small number of demographic (age, family history of psychosis), clinical (unusual thought content and suspiciousness), neurocognitive (verbal learning and memory, speed of processing), and psychosocial (traumas, stressful life events, decline in social functioning) predictor variables. The overall model achieved a C-index of 0.71, which is in the range of values for established calculators currently in use for cardiovascular disease and cancer recurrence risk, which range from 0.58 to 0.81 (5–9).

The risk calculator generates a number representing the probability of transition to psychosis given a particular profile of input variables. Technically, this is an observed likelihood of conversion within the NAPLS-2 cohort itself, but this framework uses the logic of predictive inference to extend that observed likelihood based on past cases to the predicted probability for a newly ascertained case with the same profile. This logic rests on the assumption that the new case is ascertained from the same population and in a manner similar to those in NAPLS-2.

In particular, given that this risk calculator assumes a SIPS-based diagnosis of a prodromal risk syndrome as a starting point, the risk prediction tool would not be usable if such a risk syndrome has not been diagnosed. The risk calculator also assumes particular pathways to ascertainment, in that clinical high-risk cases in NAPLS are distressed and treatment seeking. This tool would thus be most useful to clinicians with training in psychosis risk detection using the SIPS (which, in addition to risk status, ascertains severity of unusual thought content and suspiciousness and family history of psychosis), who could then use the calculator for patients who have screened positive for a prodromal risk syndrome. Critically, risk determinations should be communicated to clients by trained clinicians who can help clients understand the meaning of the risk estimates (i.e., calibrated to the sample from which they were generated) and provide commensurate treatment recommendations. Note that within the context of NAPLS-2, with a mean predicted risk of 0.18 (SD=0.11), predicted risks of 0.3 or higher are relatively rare (12.4% prevalence among those meeting clinical high-risk criteria) and potent (39.2% positive predictive value). Proper training in the administration and scoring of the other measures included in the risk calculator (symbol coding, Hopkins Verbal Learning Test–Revised, Global Functioning: Social scale, Research Interview Life Events Scale, Childhood Trauma and Abuse Scale) is also required.

A key advantage of the risk calculator is that it inherently accommodates heterogeneity in profiles of risk factors among clinical high-risk cases. Examining configurations that vary across the significant predictors—greater prodromal symptom severity, lower verbal learning and memory, slower speed of processing, greater decline in social functioning, and younger age—reveals that dozens of separate permutations yield predicted conversion risks of 0.3 or higher. Stressful life events, traumas, and family history of schizophrenia have a negligible impact on their own or in combinations with other variables in the prediction of psychosis, but they were present more frequently among clinical high-risk individuals compared with healthy comparison subjects. Perhaps these variables are more significant for determining presence of a clinical high-risk syndrome and thus are not as sensitive to outcomes within a group of subjects with a clinical high-risk syndrome.

A crucial test of robustness of a statistical model is validation on an independent external data set. In a companion article (30), Carrión et al. report on a replication test of the NAPLS-2 risk calculator in an independent sample from the Early Detection, Intervention, and Prevention of Psychosis Program (EDIPPP) that included 176 clinical high-risk cases diagnosed using the SIPS and followed clinically to monitor conversion. Only the stress and trauma variables—found to be negligible in predicting conversion here—were not collected and were therefore omitted from the replication testing. The remaining six NAPLS-2 risk factors yielded a significant time-to-event proportional hazards regression model predicting conversion in the EDIPPP sample (p<0.003), with a C-index of 0.79, which is even better than in the NAPLS-2 sample (0.71). The predictive model was well calibrated, and the NAPLS-2 calculator provided a reasonable estimation of psychosis risk when considering the risk prediction generated by the validation model and the actual observed outcomes. In addition, when applied to the external EDIPPP sample, the NAPLS-2 calculator showed sensitivity and specificity values comparable to those observed in the NAPLS-2 sample across different levels of model-predicted risk (30).

Our findings also show some degree of convergence with previous studies that reported multivariate models but that used their own samples for variable selection (i.e., model optimization) and did not present a web-based tool for extending individualized risk estimation to future patients (3, 23, 31). For example, a recent study (23) using a smaller (N=92) and nonoverlapping sample of clinical high-risk cases from one of the NAPLS sites developed a classifier that included three of the predictors included in the NAPLS-2 risk calculator (suspiciousness, verbal memory deficits, and decline in social functioning). Note that the sample in that study was less than one-fifth the size of the present sample, used a more restricted age range (ages 12–20), and was not ascertained using SIPS criteria, factors that make risk classifications based on it much less generalizable than that of the NAPLS-2 risk calculator.

The most immediate uses of the risk calculator are likely to be in the selection of subjects for participation in clinical (prevention) trials, given the desire to avoid exposing patients with lower conversion risks to the potential adverse consequences of any interventions and given the potential to evaluate whether interventions differ in effectiveness based on initial risk levels or profiles across predictors. In terms of clinical practice outside the context of a prevention trial, at this point the most likely use of the risk calculator is for the clinician to be able to communicate to the patient and family a scaling of risk that could help in recruiting their cooperation with a monitoring and/or intervention plan.

A current limitation of the psychosis risk calculator is that risk estimates are not bounded by a confidence interval, making it unclear how well the single value output as a conversion risk represents the individual’s actual likelihood of conversion. This issue is particularly problematic for computed risks of 0.5 or higher, for which there is sparse representation in the NAPLS-2 data set and for which calibration of the risk calculator could consequently not be adequately tested. Nevertheless, the use of confidence intervals is not likely to be of value in discussing risks with individual patients and their families, as the general concept of a confidence interval relates to likelihoods under future sampling rather than to an individual case, and the calculated risk is the best estimate for that individual (32).

Because the replication study (EDIPPP) included several community behavioral health centers and intergovernmental managed mental health organizations, the risk calculator appears to be generalizable beyond academic medical centers, at least within the U.S. health care system. The degree to which the risk calculator generalizes to other health care system models remains an open question.

In addition to testing the calculator’s performance in independent data sets, future work could determine whether other variables, including biological tests, can improve prediction over and above the set of clinical, demographic, and cognitive measures evaluated here. Some promising leads on the use of biological assays to predict psychosis among clinical high-risk patients have emerged using empirically based discovery approaches, including machine learning algorithms for gray matter variations in structural brain images (33) and “greedy” regression algorithms for proteomic/metabolic plasma parameters (34). Studies employing discovery-oriented model-optimization methods, with parallel, independent samples, are needed to better inform future versions of this and other risk calculators. However, it is still critical to note that the data used in any risk calculator cannot be the same data that are used in the model optimization phase; as noted above, doing so would invalidate the risk predictions for new cases.

Given that in approximately one-third of clinical high-risk cases, the symptoms that determined their initial risk status remit within 6–12 months of ascertainment (35, 36), it should be possible to develop a complementary tool to predict a new case’s likelihood of remission from a clinical high-risk syndrome. Such an estimate would not necessarily be merely the inverse of the conversion risk, as different predictors may be relevant.

It is also possible that risk calculators could eventually be used to select clients for different treatment regimens or to reclassify risk after completion of a particular intervention. At this stage, the knowledge base for doing so is limited, as only a small number of controlled prevention trials in clinical high-risk cases have appeared. Collectively, the results support the view that any targeted intervention, whether biological or psychological in approach, is associated with better outcomes than less targeted control conditions (37). Results of two small trials with antipsychotic drugs do not support a prophylactic effect on conversion risk beyond the period of active treatment (38, 39). In general, the use of such medications in individuals who are below the threshold of full psychosis is not recommended. Intriguing results have been obtained in an initial trial of omega-3 fatty acid supplementation (40); this finding awaits confirmation by independent studies. Psychosocial interventions such as cognitive-behavior therapy and family-focused psychoeducation may be beneficial in deflecting the course of illness severity and chronicity (41, 42); however, it remains unclear whether such approaches can prevent onset of illness. Future intervention studies are encouraged to use the risk calculator at end-stage analysis to determine whether treatment efficacy is moderated by initial risk level or profile.

Ultimately, the degree of risk estimated by the risk calculator may be useful for weighing the cost-benefit ratios of various treatment options that emerge from clinical intervention research in the clinical high-risk population. Treatments associated with greater risks to the patient (e.g., medication side effects) or greater costs to health care delivery systems (e.g., resource- and time-intensive psychotherapeutic inventions) may best be reserved for those with higher-than-median levels of predicted risk (i.e., ≥0.16), while cost-effective treatments with benign side effect profiles may be the best option for those whose predicted risk for psychosis is in the lower range.

Like a person at risk for cardiovascular disease or cancer, an individual with a prodromal risk syndrome is more interested in receiving information pertinent to his or her personal risk profile than information about the population at large. Publication of this risk calculator is intended to assist clinicians in providing such personalized risk estimates. It is of course possible for untrained individuals to access these tools and approximate their scores on the set of risk variables. If, in so doing, a high predicted risk of conversion were generated, this could lead to significant personal distress. To mitigate this possibility, we have built in a decision tree for the online calculator that requires confirmation of an interview-based SIPS diagnosis of a prodromal risk syndrome and confirmation that the ratings and test scores were obtained by a professional; if either one of these verifications are missing, the decision tree opts out of making a prediction. The risk for loss of privacy or stigmatization based on access of the prediction tool by untrained users is also mitigated for these reasons.

In summary, a well-performing risk calculator for psychosis is available for application to new patients who meet criteria for a psychosis risk syndrome. Challenges to be addressed in the next phase of research include incorporating biological assays into the risk calculations, extending the analysis to predict likelihood of remission, extending the framework to calculate reductions in risk based on particular interventions, and investigating how patients and family members feel about and use this information.