Introduction

The Sequential Organ Failure Assessment (SOFA) score is frequently used in the intensive care unit (ICU) to assess the incidence of organ dysfunction, guide management and aid in prognosis [1–3]. The SOFA was one of the first severity of illness scores validated for use in the ICU, initially to assess organ dysfunction in patients with sepsis [2], though recently has emerged as a tool to predict mortality in the acutely admitted ICU patient [4–6]. The score is based on major dysfunction in the following systems: hepatic, cardiovascular, neurologic, renal, respiratory and hematologic. Given the ease of calculation with data available in the daily routine of an ICU, it has become the primary method to stratify severity of illness in the ICU at time of admission [4–7].

A potential critical limitation of this tool, though, may be the assessment of neurological function based on the Glasgow Coma Scale (GCS). This scale aimed to standardize level of consciousness determinations in patients with acute brain injury [8]. However, it has expanded beyond this initial role to be used by emergency medical services and in the ICU. Although GCS is used in several ICU scoring systems, including APACHE III, SAPS III, and SOFA, it is subject to interobserver variability, and lacks verbal assessment in those undergoing mechanical ventilation [8–11]. Prior studies have attempted to modify the SOFA score, either by exclusion of the neurologic component [12,13] and even exclusion of the neurologic and hematologic component [14]. The Richmond Agitation-Sedation Scale (RASS), while designed to guide sedation in the critically ill, has emerged as an alternative tool to assess neurologic function in the ICU with minimal interobserver variability [15]. A recent study demonstrated that modification of the SOFA score by substitution of GCS for the RASS retained predictive mortality in the ICU [16].

The RASS [15,17] and the Full Outline of UnResponsiveness (FOUR) [18,19] are validated and highly reliable alternative methods to assess neurologic function in critically ill patients, irrespective of mechanical ventilation status. Given these alternative methods, we hypothesize that use of RASS and FOUR as measures of neurologic function in the SOFA score will demonstrate improved performance when compared to the GCS based SOFA score. Our ICU adopted routine collection of RASS and FOUR to evaluate neurologic function distinct from GCS. The current study aims to determine the accuracy of a modified SOFA score that replaces the neurologic GCS based component with RASS or FOUR to predict ICU mortality at time of admission of acutely ill patients.

Study design and methodology

This was a prospective observational descriptive study in adult patients over 18 years of age, admitted to the ICU from August 2015 to December 2018, in a 22 bed ICU of an urban hospital in Salvador, Bahia, Brazil. All acutely ill patients admitted to the ICU were eligible for inclusion, while postoperative ICU admissions, those with incomplete data and transfers to other hospitals were excluded from the final analysis. Individuals with missing bilirubin data at the time were included given that levels are unlikely to be significantly altered in a non-jaundiced population.

Data on age, gender, admission source, ICU outcome and length of stay, SOFA, Charlson Comorbidity Index, use of mechanical ventilation, GCS, RASS and FOUR were prospectively recorded by the medical staff as part of routine clinical care, including patients with and without sedation use in the Epimed Monitor system, which contained all other variables of interest for this study. ICU discharge was considered end of follow-up. GCS, RASS, and FOUR values were used to determine the SOFA neurologic component score in the SOFA-NeuroGCS, SOFA-NeuroRASS and SOFA-NeuroFOUR scores as shown in Table 1. Neurologic assessment in a majority of patients undergoing mechanical ventilation was performed following intubation at time of admission to the ICU. In those individuals with RASS greater than zero (suggesting restlessness or agitation), the ICU protocol in place utilizes the CAM-ICU delirium tool to guide use of sedation. There were a limited number of patients with RASS scores greater than 0 (54 patients, 5,4%) of which 11 (1.1%) had a RASS>2. Any individual with a RASS greater than or equal to 0 (ranging from alert to combative), for the purposes of our modified score, was determined to have a SOFA neurologic point equal to 0. SOFA calculations were performed by the original method, designated as SOFA_GCS, a RASS-based method, designated as SOFA_RASS and a FOUR-based method, designated as SOFA_FOUR. SOFA_GCS, SOFA_RASS and SOFA_FOUR were calculated in the first 6 hours of ICU admission [2]. To test our primary hypothesis, accuracy of SOFA_GCS, SOFA_RASS and SOFA_FOUR to predict mortality was determined by comparing the area under the curve (AUC).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Neurologic SOFA score for GCS, RASS and FOUR.

https://doi.org/10.1371/journal.pone.0229199.t001

Median and interquartile ranges (IQR) were used as measures of central tendency. Frequencies were compared using the Pearson’s chi-squared test. Continuous variables were compared using the Mann-Whitney U test (between two groups) or the Kruskal-Wallis test with Dunn’s multiple comparisons (between >2 groups). Correlations were tested using the Spearman’s rank correlation test.

The p-values were adjusted for multiple comparisons using the Holm-Bonferroni’s method [21]. All analyses were pre-specified. Two-sided P value < 0.05 after adjustment for multiple comparisons were considered statistically significant. To test our primary hypothesis, performance of the traditional and modified SOFA scores were compared using a two-tailed Z-test to evaluate the absolute AUC and difference in AUC derived from the empirical ROC curves produced by the NCSS Statistical Software. A Cox proportionate test analysis was performed to determine score performance when adjusted for variables that differed significantly between survivors and non-survivors to quantify risk of mortality predicted by each score. Statistical analyses were performed using SPSS 25.0 (IBM statistics), Graphpad Prism 6.0 (GraphPad Software, San Diego, CA) and JMP 12.0 (SAS, Cary, NC, USA). Ethics approval and waiver of consent to participate was approved by the Research Ethics Committee of Hospital Ana Nery under the number 2.571.265 and CAAE 52892315.1.0000.0045.

Results

Over the course of the study period, 2179 patients were admitted to the ICU with 1180 excluded from final analysis for the following reasons: 380 elective post-operative admissions, 336 missing GCS data, 387 missing FOUR data and 77 who were readmitted to the ICU. The final sample consisted of 999 patients (Fig 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Flowchart of study participants.

https://doi.org/10.1371/journal.pone.0229199.g001

The median age of the included cohort was 72 years old (IQR 57–83) with a female predominance (n = 544, 54.4%) (Table 2). Individuals requiring mechanical ventilation constituted a minority of patients (n = 85, 8.5%). The SOFA modified for RASS and FOUR similarly under-predicted mortality in the intubated population. Moreover, non-survivors were more likely to require vasopressors (n = 17, 14.4%) compared with those who survived (n = 13, 1.4%) for p<0.001. Neurologic scores including GCS, RASS and FOUR were decreased significantly in non-survivors compared to survivors (median n = 14, 9–15 vs n = 15, 15–15, p< 0.001). Mortality was significantly increased in those with an infectious or renal indication for ICU admission.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Study population characteristics by mortality outcome.

https://doi.org/10.1371/journal.pone.0229199.t002

When discriminate performance was assessed, accuracy of the modified SOFA with RASS or FOUR was comparable to SOFA_GCS. However, the SOFA_FOUR demonstrated significantly decreased discriminate function (AUC 0.67) for the overall cohort compared to both SOFA_GCS (AUC 0.74) and SOFA_RASS (AUC 0.71)), p = 0.042 and p = 0.004, respectively as seen in Fig 2A. There was no significant difference in performance between GCS and RASS based SOFA for the overall cohort. All three scores underperformed when evaluated in patients undergoing mechanical ventilation, a population for whom FOUR and RASS were predicted to demonstrate improved performance (Fig 2B). Specifically, SOFA_FOUR (AUC 0.55) performed significantly worse when compared to GCS (AUC 0.70) and RASS (0.70), p = 0.014 and p = 0.034 respectively, while no significant difference was detected between SOFA based on RASS and GCS. The modified and original SOFA scores continued to perform poorly when evaluated in those admitted with sepsis (Fig 2C), though the RASS based SOFA was significantly worse compared to the GCS based SOFA (AUC 0.55 vs 0.66, p = 0.035). Cox proportionate testing confirmed findings in the ROC analysis with all scores marginally predicting mortality, even when adjusting for significant covariates in survivors and non-survivors (Fig 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2.

Comparison of receiver operating characteristic (ROC) curves for prediction of ICU mortality by Sequential Organ Failure Assessment (SOFA) using Glasgow Coma Scale (GCS), Richmond Agitation and Sedation Scale (RASS) and Full Outline of UnResponsiveness (FOUR) neurologic assessment substitutions in the total cohort (A), subset undergoing mechanical ventilation (B) and septic subset (C). Comparisons between the absolute and differences of AUC were considered significant for p<0.05. AUC = area under the curve.

https://doi.org/10.1371/journal.pone.0229199.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Adjusted and unadjusted Cox regression model for ICU mortality.

The effects of traditional and modified SOFA upon survival are constant over time and did not vary when each one was adjusted for age, gender and BMI (body mass index).

https://doi.org/10.1371/journal.pone.0229199.g003

Discussion

This is one of the first prospective cohort studies in the ICU to evaluate whether substitution of the GCS in the SOFA score with RASS and FOUR improved prediction of ICU mortality at the time of admission. In contrast to our initial hypothesis, our findings suggest that substitution with FOUR is inferior to both the RASS and GCS based SOFA, even in a mechanically ventilated population. In contrast, it appears that RASS substitution for GCS may be a more convenient measure of neurologic function with marginal loss of SOFA score performance. Why these alternative methods did not surpass GCS in measurement of neurologic dysfunction is unclear. The reduced performance of the FOUR modified score in particular, compared to either RASS or GCS, suggests that patient populations distinct from a neurocritical cohort in whom this scale was developed may limit the generalizability of the FOUR [18, 19]. It may be that in the absence of severe neurologic dysfunction, the discriminate function of the FOUR score may be inferior to the RASS or GCS, as was the case in our population with primarily cardiac, pulmonary or infectious causes of ICU admission.

The Cox proportionate analysis adjusted for baseline characteristics (including age and comorbidities from the Charlson and Modified Frailty index) demonstrated marginal differences in performance of the modified and the original scores, contrary to our initial hypothesis. There may be additional unmeasured local factors responsible for ICU mortality that explain why the SOFA severity score performs poorly independent of modification of the neurologic component. Our data suggest that only RASS modified SOFA is a possible alternative to SOFA_GCS for prediction of mortality in the first 24 hours in the ICU. Our findings demonstrate similar performance of the RASS modified and original SOFA scores in the general ICU population, suggesting that alternative neurologic scale for SOFA with RASS may be effective in sedated patients and those undergoing mechanical ventilation [15,16].

Despite our study’s strength to support the use of alternatives measures of neurologic function in SOFA, we acknowledge a number of limitations. First, as a single center study there may be unknown confounding factors bias. However, given the significant size of our cohort and similar severity of illness in the general ICU population to other studies, it is unlikely that heterogeneity influenced the results found here. Second, the limited number of participants undergoing mechanical ventilation may have impacted the poor performance of all three scores in the intubated population. Discriminate function may improve in cohorts with increased use of mechanical ventilation, though, our findings highlight the poor performance of SOFA with and without modification. The lack of discriminate performance of all 3 SOFA scores in our population may be a consequence of a primarily elderly cohort whose mortality is not accurately predicted by these scores with unknown patient factors impacting score performance. Further studies are needed to clarify these findings. While 30-day mortality and morbidity following ICU discharge was not available for our cohort, our study focused on the severity of illness at time of ICU admission of an acutely ill cohort with the primary outcome of inpatient ICU mortality. Finally, despite recent studies finding improved predictive ability of SOFA in those with sepsis, our results demonstrated poor performance for both the original and modified SOFA scores, suggesting possible local patient or methodological factors in the sepsis subset [20,21].

Conclusion

Despite routine use in various medical environments, GCS may not be the most effective neurological scale for the ICU. While RASS and FOUR are convenient, our findings only support RASS substitution as a reasonable alternative for the neurologic component in the SOFA score. Further studies are required to determine whether these modifications may demonstrate improved accuracy in ICU mortality prediction in specific subpopulations, including those who are post-operative.

Supporting information

Acknowledgments

K.M.A., N.M.F.F. and B.B.A. were responsible for study design, implementation, manuscript preparation and had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. T.A.C, I.B.B.F., R.C.M., G.P.T, M.L.O., L.P.N and S.A. contributed substantially to the study design, data analysis and interpretation, and writing and review of the manuscript. K.F.F and M.B.A. were responsible for advance statistical analysis, figure generation and manuscript review and preparation. Research groups GEMINI, linked to the Núcleo de Ensino e Pesquisa do Hospital da Cidade, and MONSTER, linked to the Osvaldo Cruz Foundation.