1.

Introduction

Since its emergence in December 2019 in Wuhan, China, the global outbreak of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) and its associated disease—COVID-19—has accrued over 2160 million confirmed cases and over 4,500,000 confirmed deaths.¹ To reduce the overall burden of COVID-19 on healthcare systems, especially in the face of additional variants, diagnosis and prognostication should be accurate, efficient, and accessible. For diagnosis, the reference standard is reverse-transcriptase polymerase chain reaction (RT-PCR). This can be aided by imaging modalities such as computed tomography (CT) and chest radiography (CXR), which have been studied for diagnosis and for monitoring progression using AI methods.^2–6 The most common thoracic imaging manifestations of COVID-19 pneumonia are reported as peripheral, bilateral ground glass opacities with or without consolidation or visible interlobular lines.^7,8

Because COVID-19 can vary widely in initial presentation and disease progression—from asymptomatic to multiorgan failure and death—much research has focused on finding clinical and imaging prognostic indicators.^9,10 These studies have used a variety of endpoints, such as mortality, intensive care (ICU) admission, length-of-stay, and disease severity rating.^10,11 Some studies have identified clinical and demographic factors associated with more severe outcomes, and prognostic algorithms based on these variables have been proposed for clinical use.¹² For prognosis prediction based on imaging, visual semiquantitative scoring systems of CT and CXR have been proposed.^3,13,14 Others have applied AI to CT images for prognosis prediction.^15–18 Due to the large number of potential predictors, machine learning/artificial intelligence methods are used extensively in these studies for feature extraction and classification, with potential for more rapid results over human readers.¹¹

However, there is scarce literature on direct comparisons of relative prognostic abilities between clinical, quantitative CXR, and quantitative CT variables using multiparametric statistical models. Due to inconsistencies in clinical and laboratory data gathered and availability of various imaging modalities across different healthcare systems, such a comparison is crucial in working toward not just an accurate prognostic model but a maximally resource-efficient one.

To compare these many variables efficiently, feature selection is required. Embedded methods, such as random forest, are popular in feature selection due to their robustness to noise and resistance to overfitting.¹⁹ However, both wrapper and embedded methods become computationally intensive with higher feature datasets, lack interpretability as “black box” methods, and have been found biased by correlated features.²⁰ To address these weaknesses, embedded methods can be combined with filter methods, such as minimum redundance-maximum relevance (mRMR) which quickly culls variables by their relevance while penalizing redundancy to other variables.²¹ mRMR has the additional advantage of interpretability of the features filtered. Developed for and applied to genomics analysis, mRMR in combination with advanced embedded/wrapper methods has been found more accurate than using classifiers alone, with lower computational cost.²² Here, we use mRMR in conjunction with random forest for simultaneous feature selection and inferences on the relative prognostic strength of variables.

Our goals in this study were to assess whether severe COVID-19, which we defined as requiring ICU admission, can be accurately predicted using a set of demographics, clinical, qCXR, and qCT variables at initial presentation. ICU admission was chosen to address serious disease manifestations, including but not limited to respiratory failure. Our secondary goal was to assess if AI-read qCXR was a more cost- and time-efficient, noninferior predictor compared with human-read qCT, acknowledging that a direct measure of any efficiency gains in radiologists’ interpretation is beyond the scope of this study. Moreover, we assessed the relative performance of models incorporating subsets of clinical and imaging variables with cross-validation, to rank the most valuable predictors and arrive at a parsimonious optimal model with a minimum set of variables.

2.

Materials and Methods

2.1.

Patient and Image Selection

This single institution, retrospective study obtained IRB approval with waiver of informed consent and was HIPAA compliant. We randomly selected 131 patients with the following inclusion criteria: positive RT-PCR for SARS-CoV-2, a pair of CXR, and chest CT performed within 48 h of each other and within 48 h of presentation, and age older than 18. All scans were obtained within the following date range: March to October, 2020. Figure 1 details study design, inclusion, and exclusion criteria.

Fig. 1

Flow diagram with inclusion and exclusion criteria in our study cohort ($n=131$).

CXR and CT images were de-identified using a standard anonymization profile in Sectra PACS and transferred through a secure file exchange to a computational cluster for imaging processing. Table 1 lists all variables included for analysis for each patient.

Table 1

Variables by category.

Demographics	Vital signs	Laboratory	AD metrics
Age	Respiratory rate	AST (abnormal)	POa CXR CNN
Cardiovascular comorbidities	Symptom-neurologic	ESR (abnormal)	POa DRR CNN
Gender	Temperature	CRP (abnormal)	POv CT ground-truth
Renal comorbidities	BP_abnormal	Creatinine (abnormal)	—
BMI	Symptom-fever	D.Dimer (abnormal)	—
Immunologic comorbidities	Symptom-GI	MCHC (abnormal)	—
Diabetes	Heart rate	ALT (abnormal)	—
Other comorbidities	Symptom-respiratory	RDW (abnormal)	—
Respiratory comorbidities	—	MCV (abnormal)	–
—	—	Hct (abnormal)	—
—	—	Hgb (abnormal)	—
—	—	Ferritin (abnormal)	—
—	—	WBC (abnormal)	—
—	—	MCH (abnormal)	—
—	—	Troponin.T (abnormal)	—
—	—	Plt (abnormal)	—
—	—	RBC (abnormal)	—

2.3.

Airspace Disease Quantification on CXR and Chest CT

All CXRs were performed with AP (anterior posterior) technique and digital acquisition. Chest CTs were performed with variable protocols (on Siemens or GE scanners); however, every scan included contiguous 1-mm axial slices without interslice gap, obtained with iterative reconstruction using high spatial resolution algorithms (tailored for lung evaluation).

Airspace disease (AD) was quantified on CT and on CXR utilizing a combination of expert human annotations on CT and an AI method on CXR, detailed on a previous publication.²³

Two deep convolutional neural networks (CNNs) are used, one for lung segmentation and one for segmentation of airspace opacity from chest DRR and x-ray. Both networks are based on a residual U-net architecture, with the encoder layers initialized from the ResNet18 architecture.^24,25 This results in a nine-layer deep network with 64, 64, 128, 256, and 512 feature maps in the encoding layers (following the ResNet18 structure). Skip connections based on feature map concatenation are used. The final layer is a mapping to two-channel output, describing foreground (i.e., airspace opacity area) and background.

CT images have been annotated at raw resolution and without preprocessing. DRRs are generated as integrals over synthetic projection lines through the CT volume under a parallel projection geometry. Using a deep CNN, we enhance the resolution of the resulting DRR to achieve isotropic resolution that is higher, closer to typical chest radiographs (CXR). DRRs and x-rays (during inference) are feed to both segmentation networks at a resolution of $512\times 512\text{pixels}$. More details can be found in Ref. 23. In terms of preprocessing, for x-ray processing during inference we use the robust intensity normalization technique described in Ref. 26.

Figure 2 shows our method. Figure 3 provides an example of AD quantification on CT. Figure 4 demonstrates a patient example of multiple types of AD quantification according to our method, on CT, CXR, and DRR, by the CNN algorithm, validated by human expert readers.

Fig. 2

Schematic illustration of our method, which generates quantitative AD annotation on CXR by a CNN algorithm, leveraging paired CT POv AD (ground-truth) and validated by two independent expert readers. The set of imaging variables were fed into our statistical models, in conjunction with CVs. Light blue color reflects expert human annotation of AD (two expert readers). Color coded heat map (predominantly red) reflects the CNN prediction of AD, with red in the center representing the highest confidence level, and blue at the edges the lowest.

Fig. 3

CT derived three-dimensional volume (POv) airspace quantification (ground truth), on (a, e) axial, (b, f) sagittal, (c, g) coronal MPRs, and on (d, h) VR masks, in two patients in our cohort. Light blue lines reflect the contours of the segmented AD.

Fig. 4

(a) DRR, (b) AP intensity thickness AD mask from CT, (c) DRR with projected AD mask, (d) DRR CNN prediction, (e) CXR, (f), (g) CXR expert readers for validation, (h) CXR CNN prediction, in a patient example in our cohort. Light blue color reflects expert human annotation of AD [(f) and (g)] two expert readers]. Light blue color in (c) reflects the DRR with projected AD mask from CT ground-truth. Color coded heat map (predominantly red) reflects the CNN prediction of AD, with red in the center representing the highest confidence level, and blue at the edges the lowest (d and h).

The metrics used to assess the severity of AD are as follows:

• • Percentage of opacity–volume (POv): The POv is measured on CT scans and quantifies the percent volume of the lung parenchyma that is affected by AD

  $$\mathrm{POv}=100\times \frac{\text{Volume of Airspace Disease}}{\text{Total Lung Parenchyma Volume}}.$$

• • Percentage of opacity – area (POa): The POa is measured on CXRs and quantifies the percent area of the lung parenchyma that is affected by AD:

  $$\mathrm{POa}=100\times \frac{\text{Area of Airspace Disease}}{\text{Total of Lung Parenchyma Area}}.$$

The Appendix describes characteristics of AI CNN training datasets.

3.

Results

Of the 131 total patients, 48% (63) were male, and 52% (68) female, with a mean age of 60.3 and standard deviation of 18.0. Of the 60 patients who required ICU care, 48% (29) were male, 52% (31) were female, with a mean age of 67.1 and standard deviation of 16.4. Of the 71 patients who did not require ICU care, 48% (34) were male, 52% (37) were female, with a mean age of 54.4 and standard deviation of 17.3 (Table 2).

Table 2

Demographics of cohort (n=131).

In the dimension reduction, age, respiratory rate, abnormal creatinine, POv on CT, and abnormal MCHC were identified as the top five variables by random forest among the top 10 variables filtered by mRMR (Table 3).

Table 3

Top 10 variables identified by mRMR, ranked by MAD on random forest (MCHC, mean corpuscular hemoglobin concentration; CRP, C-reactive protein; AST, aspartate aminotransferase).

The CVs only model (model I) had the poorest performance and accuracy: diabetes, cardiovascular comorbidities, respiratory comorbidities, and BMI were insignificant, only age was significant ($\mathrm{AUC}=0.67$ [0.58 to 0.76]; $\text{accuracy}=77\%$). In the logistic model fit using variables selected from mRMR and random forest dimension reduction (model II), all variables but respiratory rate were significant predictors of ICU ($\mathrm{AUC}=0.78$ [0.70 to 0.86]; $\text{accuracy}=81\%$). A parsimonious model with only age and affected volume on CT (model III) performed similarly without need of laboratory tests ($\mathrm{AUC}=0.75$ [0.67 to 0.84]; $\text{accuracy}=80\%$; AUC difference to model II = 0.03 (95%: −0.02 to 0.07, $p=0.27$)). Both models II and III had significantly higher AUC compared to model I ($\mathrm{AUC}\text{difference}=0.11$ (95%: 0.02 to 0.19, $p=0.01$); $\mathrm{AUC}\text{difference}=0.08$ (95%: 0.01 to 0.15, $p=0.04$), respectively). Neither models II nor III performed significantly differently when POv CT was swapped out for AI CNN-obtained POa on CXR ($\mathrm{AUC}\text{difference}=0.01$ (95%: −0.01 to 0.02, $p=0.23$); $\mathrm{AUC}\text{difference}=0.01$ (95%: −0.01 to 0.03, $p=0.45$), respectively) (Figs. 5 and 6). Tables 4 and 5 contain variable coefficient summaries for all models fit as well as results of the Delong tests comparing correlated AUC’s.

Fig. 5

Vertically averaged LOOCV ROC of models I to III, with parallel models II and III using POa (CXR CNN, model “a”) and POv (CT, model “b”), respectively.

Fig. 6

Vertically averaged LOOCV ROC of model II, with different imaging variables (POa CXR CNN, POv CT, and POa DRR CNN).

Table 4

Individual variable coefficients, confidence intervals, and p-values from models I to III (ROC, receiver operating curve; AUC, area under the curve; LOOCV, leave-one-out cross-validation; BMI, body mass index (Kg/m2); CV comorbidities, cardiovascular comorbidities (0/1 binary, 1 = comorbidities present); MCHC, mean corpuscular hemoglobin concentration (pg.); POv, percent of opacity-volume; CT, computed tomography; POa, percent of opacity-area; DRR CNN, digitally reconstructed radiographs; CNN, convolutional neural network.)

Table 5

Individual variable coefficients, confidence intervals, and p-values from model II with different imaging variables (ROC, receiver operating curve; AUC, area under the curve; LOOCV, leave-one-out cross-validation; BMI, body mass index (Kg/m2), CV comorbidities, cardiovascular comorbidities (0/1 binary, 1 = comorbidities present); CRP, C-reactive protein; POv, percent of opacity-volume; CT, computed tomography; POa, percent of opacity-area; CXR, chest x-ray; DRR, digitally reconstructed radiographs; CNN, convolutional neural network).

4.

Discussion

Our study demonstrates that COVID-19 requiring ICU can be accurately predicted at presentation with just age and extent of lung parenchyma affected by AD, either % volume on CT (POv) or % area on CXR (POa). The noninferiority of this parsimonious model to the complete model with laboratory values and the relatively poor performance of the prediction model based only on comorbidities suggests that imaging—especially the faster and cheaper AI-read qCXR—may be a more resource-efficient way to assess prognosis than obtaining many laboratorial values. It also suggests that imaging may be a better prognostic predictor than some predictors commonly evaluated in literature, such as BMI, diabetes, or cardiovascular and respiratory comorbidities.

These conclusions are corroborated by both mRMR and random forest dimension reduction, where most nonimaging variables were ranked lower. From mRMR, we can infer that these variables were either not as relevant at predicting ICU admission, contained redundant information, or both. This is reflected in the logistic regression coefficients, where AD coefficients are consistently of higher magnitude than other predictors—this implies that while age is a powerful prognosticator in the elderly, in younger COVID-19 patients their AD on imaging is the strongest prognosticator of ICU admission. Moreover, the coefficients suggest that older patients tend to require ICU admission with less AD than younger patients.

Furthermore, it also shows that AI-derived AD quantification on CXR is as strong a prognostic predictor of severe COVID-19 as AD quantification on CT (by expert annotators) and AD quantification on DRR from CT. Moreover, imaging variables add information that improves prognostic performance and is not redundant with other CVs. Using POa from CXR or POv from CT, particularly if computed by AI algorithms, has the additional advantage of simplicity, reproducibility, and speed over semiquantitative scoring systems, besides being applicable in environments that may lack access to expert human radiologists.

Our study has several limitations. Given that we selected a subset of patients who fulfilled multiple inclusion criteria (positive RT-PCR for SARS-CoV-2, and paired chest CT and CXR performed within 48 h of each other and within 48 h of presentation) and due to the single center design, we have a relatively small sample size compared with the total population of COVID-19 patients. Notwithstanding, our models generated statistically significant results and our random sample is representative of the COVID-19 patients admitted to the hospital. The AD annotation on CT was performed by two human readers, without automated algorithms; however, outlier cases were secondarily reviewed by two additional expert radiologists and corrected. While every patient in the cohort had COVID-19 at the time the CXR and CT were obtained, it is possible that not all AD detected was a manifestation of COVID-19, though expert human radiologists made their best effort to exclude likely chronic opacities on CT. Our sample size limited the number of features which could be tested at once, either through random forest or by logistic regression, which we addressed using dimension reduction techniques. Furthermore, this study used binarized clinical laboratory values—as more research establishes relative value ranges associated with severity of outcome, it would be feasible to more fairly assess the relative prognostic value of laboratorial tests compared with imaging. However, given the inconsistent panel of laboratory values obtained per patient—largely dependent on clinician discretion—such binarized laboratory values allowed for better imputation of missing values. Finally, our models’ AUC of 0.75 to 0.78 suggests the existence of other predictors of COVID-19-related ICU admission not accounted for in this analysis. This performance may also suggest that different transformations of existing data—such as the use of different thresholds for laboratory values, as exist for example in other disease states—are more relevant for prognostication.

A future direction of our research includes expanding the sample size to explore more potential predictors and their interactions, as well as expanding the dataset to cover multiple timepoints for longitudinal analysis, which will likely lead to more accurate predictive models. Another direction is to include patients with CXR without CT to expand our sample size, as we have demonstrated prognostic equivalence in imaging modalities.

5.

Conclusions

In conclusion, our study demonstrates that prediction of severe COVID-19 with sufficient accuracy as to be clinically relevant is attainable using only a small set of variables at initial presentation, while emphasizing that quantitative imaging variables derived from CXR and CT are superior to well established CVs without imaging, adding prognostic prediction accuracy without being redundant. Our study also emphasizes the potential of statistical prognostication models, which incorporate data from multiple modalities, including clinical, demographic, laboratory, and imaging variables. Given that AI methods can obtain quantitative imaging metrics with similar accuracy but much faster than expert human readers, AI methods for detection and quantification of AD, particularly when applied to much more widely available CXRs, feeding statistical predictive machine learning models of severe COVID-19, may augment the role of thoracic imaging and impact the management and prognostication of patients affected by COVID-19, fostering better patient outcomes, improving resource allocation and potentially increasing radiologists’ efficiency.

6.

Appendix: Properties of the Training and Validation Data used for Development of the AI Lung and Airspace Disease Segmentation on CT and CXR, Compared to Our Study Cohort

Table 6 contains further details on the properties of the training and validation data used for development of the AI lung and airspace disease segmentation on CT and CXR, compared to our cohort.

Table 6

Demographics, scanner, and image characteristics of data used for AI development and for the study cohort.