Main

Over the years, in-patient intravenous antibiotic treatment has been demonstrated to be the most effective strategy for patients with postchemotherapy febrile neutropenia (FN). More recently, patients with solid tumours have been seen to present mild episodes associated with clinical stability, particularly when they receive care in the early stages of infection (Carmona-Bayonas et al, 2011); hence, not all of them require prolonged hospitalisation for intravenous antibiotics. Consequently, various groups have sought to stratify patients with FN into high- and low-risk groups, in an attempt to individually predict their clinical course and optimise treatment (Talcott et al, 1992; Klastersky et al, 2000; Ahn et al, 2015; Carmona-Bayonas et al, 2015).

The Continuous Care Working Group of the Spanish Society of Medical Oncology (SEOM, for its acronym in Spanish) has recently devised a prognostic classification, called the Clinical Index of Stable Febrile Neutropenia or CISNE, for patients with solid tumours and seemingly stable episodes. CISNE is capable of classifying stable patients according to their risk of serious complications and death more accurately than previous scales (Carmona-Bayonas et al, 2015).

CISNE is used in the same way as the eligibility criteria of most randomised clinical trials that compared in-patient treatment to ambulatory treatment or oral vs intravenous antibiotics and that excluded patients using pragmatic clinical criteria, such as shock, comorbidity, acute organ failure, pneumonia, or oral intolerance (Teuffel et al, 2011; Vidal et al, 2013). These potentially serious situations comprise a starting point for prognostic evaluation, as recommended by the American Society of Clinical Oncology’s (ASCO) clinical practice guideline (Flowers et al, 2013).

The CISNE model was designed as an adjuvant tool in decision-making, allowing to individually predict the development of serious complications despite the absence of evident risk criteria such as those previously mentioned (Carmona-Bayonas et al, 2015). While CISNE’s prognostic potential has been demonstrated, further applicability and efficacy evaluations are needed to generalise its use in the community.

The aim of this study was to develop a nomogram and online calculator capable of generating individualised predictions, and to externally validate this new decision-making support tool to predict serious complications in an independent series of the University of Salamanca Hospital (USH) FN registry.

Materials and Methods

Patients

The nomogram was developed from a prospective cohort of 1133 patients with seemingly stable FN from the FINITE (Evaluación de Factores Pronósticos en Fiebre Neutropénica, Tumour Sólido y Episodios Estables) study. A description of the FINITE study design, data collection, data quality, and eligibility criteria have been reported previously (Carmona-Bayonas et al, 2015). Briefly put, FINITE was a multicentre study conducted in 22 Spanish centres and one Chilean centre between 2012 and 2014. The key eligibility criteria required that the patient be adult, have a solid tumour treated with mild–moderate intensity chemotherapy, not be hospitalised for another reason, not have acute organ failure (renal, cardiac, respiratory) or decompensation of chronic insufficiency, lack septic shock or hypotension (systolic pressure <90 mm Hg), have no severe infections, and lack other serious complications within the first 3 h of being diagnosed with FN (see definitions in Table 1).

Table 1 Key definitions of the study
Full size table

Validation was carried out with the USH FN registry. The University of Salamanca Hospital is a 900-plus bed tertiary institution, and its Medical Oncology Department is the reference department for an estimated population of 350 000 inhabitants. In 2010, this department launched a project to create a database of FN patients. The objective of this registry was to understand the epidemiological characteristics and outcomes after the centre had instituted an ambulatory programme to manage these patients. This registry has been reported previously by the authors (Matos et al, 2014). The hospital’s Ethics Review Board approved the registry and written informed consent was obtained from the patients.

The criteria for entry into the registry included patients (⩾18 years) with solid tumours or lymphomas treated with chemotherapy who sought emergency medical care for fever ⩾38 °C and neutrophils ⩽500/mm3 (or ⩽1000/mm3 with an expected decrease ⩽500/mm3), regardless of their clinical status. This registry took into account only the first episode suffered by a patient during the study period.

Study design

The FINITE study design has been described previously (Carmona-Bayonas et al, 2015). Insofar as the USH registry is concerned, clinical data sources consisted of information gleaned from both patient and Emergency Department records. This registry was updated periodically by the on-duty oncologists and the people charged with treating these patients. To standardise collection, the USH investigators designed a form containing the following variables: description of patients’ clinical status, including vital signs, other medical comorbidities, tumour type and stage, oncological treatment, FN management, microbiology, duration, and outcome. Missing values for any CISNE explanatory model covariates were disallowed, although all predictors were readily available and there were no exclusions for this reason.

Patients in the USH registry were classified as seemingly stable or unstable, as per the same criterion followed in the CISNE-derived series (see Table 2). All individuals remained in observation for at least 6 h. As per the treating physician’s judgement, clinically stable patients were given the possibility of receiving direct home care (discharge between 6 and 24 h), early discharge (24–72 h), or standard admission (stay ⩾72 h).

Table 2 Comparison of baseline characteristics in the study cohorts
Full size table

The primary end point for this analysis was the occurrence of major FN-associated complications in patients believed to be clinically stable. Major complications were defined as hypotension, acute renal, respiratory, or cardiac failure, arrhythmia, major bleeding, delirium, acute abdomen, disseminated intravascular coagulation, and other events (Table 1).

To minimise subjectivity in data interpretation, two specialists in patient support treatment from USH evaluated the presence of complications and CISNE predictors separately, according to detailed instructions and objective definitions of the variables (Table 2). The investigators were not informed as to the CISNE categories, despite the fact that data collection was not blinded.

Observation was a minimum of 7 days from the time of arrival at the USH Emergency Department until the episode had fully resolved. Resolution, in turn, was defined as the disappearance of all signs of infection, recovery from neutropenia (⩾1500 neutrophils per mm3), and being afebrile for 48 h. In the event that the patient was discharged early or received ambulatory treatment, a home surveillance component was used to cover this period.

Development of the nomogram and web-based calculator

A nomogram is a graphic representation of the solution of an equation that provides a reasonable approximation of the probability of a particular outcome. The CISNE model explanatory covariables consisted of: Eastern Cooperative Group (ECOG) Performance Status ⩾2, chronic obstructive pulmonary disease, cardiovascular disease, mucositis NCI grade ⩾2, monocytes <200/mm3, and stress-induced hyperglycaemia (Carmona-Bayonas et al, 2011). To build the nomogram, the models’ coefficients were updated and retrained, bearing in mind the entire FINITE series (n=1133). The objective definition of clinical stability, issues regarding inferential generalisability, heterogeneity concerning prevalence of the explanatory covariables, and the model’s scope of application have been described previously (Carmona-Bayonas et al, 2015). We used k-fold crossvalidation with k=5 to limit the risk of overfitting when obtaining the scores associated with each of the nomogram’s covariates. We therefore conducted five logistic regression models adjusted using a maximum-likelihood estimation to predict the binary outcome (presence or absence of serious complications) on the training subsets over five different random partitions of the data into five-folds. These models’ coefficients served to construct five nomograms. The goodness of fit for each nomogram was verified by means of the Hosmer–Lemeshow test on each subset. Finally, we averaged estimates of all nomogram parameters from the five models to correct and assess the bias thereby generated. The final nomogram (Figure 2) was used to assign a probability of complication to each patient. The same was carried out to develop an online calculator: http://www.iricom.es/prognostictools/Cisne/inicio.aspx.

To determine the discrimination ability of the final model, we developed 1000 bootstrap replications that were used as internal validation subsets, calculating the model’s bias-corrected concordance (c)-index to predict serious complications. To estimate the optimal cutoff point of the model, predictions were dichotomised by Youden’s J index (Youden, 1950).

External validation of the nomogram in the USH registry

All patients from USH who met the criterion of clinical stability were assessed using this nomogram (Ghanem et al, 2014). The calibration and discriminatory ability were externally validated in clinically stable patients from the USH registry. First, we used a bootstrapping procedure with 2000 replications to get overfitting-corrected estimates of predicted vs observed values to evaluate the nomogram’s calibrations. The absolute mean and squared errors and the 0.9 quantile of the absolute error were calculated. Then, we computed a summary of the c-index of this validation subset as a measure of discrimination. Based on previous research, around 300 patients would be sufficient for this analysis (Peduzzi et al, 1996). All statistical analyses were performed using R software, including the rms package version 4.2-1 (R Development Core Team, 2014; Harrell et al, 2015).

Results

Patients

Between the two registries, a total of 1952 patients with FN were screened, of which 73% of the FINITE cohort (n=1133) and 76% of the USH registry (n=296) were eligible for analysis. Figure 1 is a flowchart of both databases. Table 2 presents the descriptive statistics of each series. There are no differences between both series with respect to age (median of 60 years in both), functional status, and tumour stage. The distribution of tumours and episodes is quite similar. The most remarkable differences between the series lie in the fact that there was greater prophylaxis with granulocyte-colony-stimulating factor (G-CSF) in the FINITE cohort (27% vs 13%; P<0.001) and prior antibiotic use (18% vs 5%; P<0.001). In contrast, the USH patients present a higher rate of direct ambulatory care (30% vs 11%, P<0.0002) or treatment with oral antibiotics (36% vs 10%, P<0.0002).

Figure 1
figure 1

Flow diagram of the FINITE and USH series.

Full size image

Outcomes

The multicentre FINITE series revealed marginally lower rates of serious complications and mortality: 13.4% (95% CI, 11.5–15.5) and 1.8% (95% CI, 1.1–2.7), respectively. In comparison, the rate of infection-related serious complications and death in seemingly stable patients in the USH registry was 18.6% (95% CI, 14.6–23.4) and 2.7% (95% CI, 1.3%–5.2%), respectively. In this registry, the most common complications were shock (n=22), acute respiratory failure (n=11), major bleeding (n=5), and cardiac insufficiency (n=3). In the 108 USH patients who were discharged early or directly, 5% (n=6) were readmitted because of complications. These complications were hypotension in two cases, and respiratory failure, mucositis, anaemia, and intestinal occlusion in one case each. None of the patients who were readmitted died. In contrast, the death rate was significantly higher among the ineligible USH patients (unstable): 18.1% (95% CI, 11.6–27).

Development of a nomogram based on the FINITE data set

First, we developed the prognostic nomogram for estimating the risk of serious complications. Figure 2 illustrates this nomogram. Actual proportions deviated by <1% from the predictions (Figure 3). The Hosmer–Lemeshow test showed good fit of the model for the entire series, and separately for each of the five subsets used to create it. Youden’s J index corresponds to ⩾116 points (equivalent to a risk of serious complications ⩾13%), which is considered the optimal cutoff value for high risk. The distribution of prognostic categories by nomogram scores is presented in Table 3. Across the 1000 bootstrap replications, the bootstrap optimism-corrected c-index for predicting serious complications was 0.855 (95% CI, 0.823–0.887).

Figure 2
figure 2

The CISNE nomogram. COPD=chronic obstructive pulmonary disease; ECOG PS=Eastern Cooperative Oncology Group Performance Status; NCI=National Cancer Institute.

Full size image
Figure 3
figure 3

Calibration plot showing predicted probability vs actual (observed) frequencies of serious complications in clinically stable patients from the FINITE study ( n =1133). (Bootstrap procedure with 2000 repetitions; mean absolute error=0.008; mean squared error=0.00021; 0.9 quantile of absolute error=0.014. Distribution of predicted risks of patients is shown at top of the figure.)

Full size image
Table 3 Distribution of risk classes and outcomes derived from the FINITE and USH seriesa
Full size table

Validation of the nomogram in the USH registry

We then applied the nomogram to the subgroup of 296 apparently stable USH patients. Table 3 shows how the complications and mortality rates go up as the nomogram score increases (P<0.001). With a cutoff of ⩾116, the nomogram has 66% sensitivity, 83% specificity, 3.88 positive likelihood ratio, 0.41 negative likelihood ratio, 48% positive predictive value, and 91% negative predictive value. A c-index of 0.831 (95% CI, 0.782–0.871) was displayed, denoting good discriminatory ability. The nomogram’s predictions appear to be well calibrated compared with those observed in the USH registry with a mean absolute error of 3.6% (Figure 4). The Hosmer–Lemeshow goodness-of-fit test (χ2 6.01; P=0.3) also exhibited appropriate calibration (Figure 4). Notably, in the subgroup that was readmitted after direct or early discharge (n=6), 5 (83%) had a score ⩾116 on the CISNE nomogram (high risk).

Figure 4
figure 4

Calibration plot showing predicted probability vs actual (observed) proportions of serious complications in clinically stable patients from the USH registry ( n =296). (Bootstrap procedure with 2000 repetitions; mean absolute error=0.036; mean squared error=0.00291; 0.9 quantile of absolute error=0.111. Distribution of predicted risks of patients is shown at top of the figure.)

Full size image

Discussion

This study describes the development of a nomogram to predict prognosis in patients with seemingly stable FN. The criterion for application of the CISNE nomogram is initial apparent clinical stability as defined by the absence of serious complications, extensive infection, shock, and acute organ failure at the time of first evaluation. The nomogram assigns a predicted probability of serious complications based on the CISNE model’s explanatory covariables (ECOG PS ⩾2, chronic obstructive pulmonary disease, cardiovascular disease, mucositis NCI grade ⩾2, monocytes <200/mm3, and stress-induced hyperglycaemia). This nomogram has been applied to an external validation series from a registry of consecutive cases of FN at the USH. In this series, it has proven reasonable calibration and good discriminatory ability to predict serious complications.

Although the CISNE nomogram was designed chiefly to predict the likelihood of serious complications in stable patients initiating in-patient therapy, the USH data confirm that a two-stage classification similar to that used in the FINITE study (exclusion criteria plus the CISNE nomogram) comprise a safe method by which to select low-risk patients when the screening takes place within 6 to 24 h from the time of arrival at the Emergency Department.

The USH series reveals that the absence of baseline criteria of severity (apparent clinical stability) is efficacious when selecting patients with relatively low risk of complications and death; for instance, infection-related mortality was significantly lower in seemingly stable patients in comparison with clinically unstable subjects (3% vs 18%). However, the absence of severity criteria at the onset does not fully rule out the possibility that the neutropenic patient will develop serious complications, which occurred in 18% of the patients deemed stable and even in 5.5% of those who received direct or early discharged in the USH cohort.

In light of the aforementioned, we believe that a CISNE-based nomogram can contribute to improve exclusion criteria for management in these patients, as well as their classification, or decision-making regarding oral or outpatient treatment.

The nomogram has several limitations that must be indicated to optimise its use. While the model’s functioning was robust in a subsequent update and its explanatory covariables therefore appear to be appropriate in different contexts, one must always be mindful of the possibility of uncommon risk factors, for example, cirrhosis, malnutrition, chronic renal disease, or other pathologies or situations. Although patients with lymphomas were included, there were so few cases (n=21) as to advise caution when dealing with this population. The USH registry’s sample size depended on patient volume, but the series contains ∼91% of the events generally recommended for studies such as this one (Peduzzi et al, 1996). Similarly, it is possible that the nomogram’s discriminatory ability might be enhanced in the future by incorporating new variables. Given their extraordinary variability, it is important that we attempt to integrate patients’ specific infection and oncological status. Nevertheless, these six variables appear to be germane and discriminatory in the three series analyzed. The model appears to be well calibrated across the spectrum of severity despite which a slight underestimation of risk was seen in the range between 20 and 40% of predicted probabilities of the USH cohort. Although the eligibility criteria for oral, outpatient management in the USH registry were similar to those of the Infectious Disease Society of America (IDSA) clinical practice guidelines (Freifeld et al, 2011), the decision as to where to treat fell ultimately to the treating physician, who had to factor in each patient’s particular risk factors, or other family, psychological, or social aspects. The USH series also suggests that a percentage of serious complications can be detected only by increasing the screening period beyond 3 h (e.g. by in-hospital observation for 6–24 h).

Finally, how the CISNE model would be used within the current classification systems recommended by clinical guidelines must be elucidated, as well as the differences with respect to other existing models, such as the Talcott and Multinational Association for Supportive Care in Cancer (MASCC) score (Talcott et al, 1992; Klastersky et al, 2000). At present, the MASCC score is the most widely validated and recommended to identify low-risk patients, although never as a stand-alone selection method, but rather in combination with exclusion criteria similar to the ones proposed in this article (Flowers et al, 2013). However, our group has demonstrated that the CISNE score is better able to discriminate than the MASCC model when used in apparently stable patients with solid tumours: areas under the receiver operating characteristic curves 0.721 (95% CI, 0.669–0.768) for MASCC and 0.868 (95% CI, 0.827–0.903) for CISNE (P=0.002) (Carmona-Bayonas et al, 2015).

Several methodological issues hinder the applicability of the MASCC model in daily clinical practice. Most notable among them are that the heaviest weighted predictor, hypotension, is also the most common outcome; that a relevant percentage had acute leukaemia or bone marrow transplant, limiting its application to patients with solid tumours; or that many patients were initially haemodynamically unstable and could therefore never be classified as low risk, despite their MASCC score.

The most obvious advantage of the CISNE model over the previously proposed ones is that the two-step triage process improves patient safety, reinforcing decision-making without added complexity. CISNE seeks to decrease the uncertainty linked to the diminished inflammatory response due to immunosuppression, which undermines the clinician’s ability to detect complications early on. As a final point, CISNE is a safe method because, unlike others, its prime purpose is not to release patients early, but just the opposite – to keep them from occurring until the apparent stability has been confirmed as real after an appropriate time of in-hospital observation. Therefore, the results of our study are mainly applicable to avoid early discharge of patients with cancer beginning in-patient therapy. In light of our data, we suggest that patients experiencing seemingly stable episodes with a CISNE nomogram score ⩾116 points (equivalent to a risk of serious complications ⩾13%) should not be sent home until they have proven to be truly stable and blood cultures have been examined. On the contrary, we emphasise that this nomogram should not be used so much to select low-risk patients for direct outpatient treatment. Nevertheless, well-conducted clinical trials are required to endorse that this prognostic classification can be safely and efficaciously associated with therapeutic classes.

In short, we report a nomogram to predict complications in seemingly stable patients with solid tumour who present FN and who commence in-patient treatment. This tool has been developed in two different series that, together, comprise more than 1400 eligible patients and subsequently validated using an external database. The nomogram can therefore be useful for individualising patient care in terms of treatment and place, in addition to interpreting other series and future studies in this field.