Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Angiotensin II receptor blocker or angiotensin-converting enzyme inhibitor use and COVID-19-related outcomes among US Veterans

  • Catherine G. Derington ,

    Roles Conceptualization, Data curation, Investigation, Supervision, Visualization, Writing – original draft, Writing – review & editing

    catherine.derington@utah.edu

    Affiliation Department of Population Health Sciences, Division of Health System Innovation and Research, University of Utah School of Medicine, Salt Lake City, UT, United States of America

  • Jordana B. Cohen,

    Roles Conceptualization, Investigation, Methodology, Visualization, Writing – review & editing

    Affiliations Department of Medicine, Renal-Electrolyte and Hypertension Division, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA, United States of America, Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States of America

  • April F. Mohanty,

    Roles Investigation, Methodology, Visualization, Writing – review & editing

    Affiliations George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT, United States of America, Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, UT, United States of America

  • Tom H. Greene,

    Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Supervision, Writing – review & editing

    Affiliation Department of Population Health Sciences, Division of Health System Innovation and Research, University of Utah School of Medicine, Salt Lake City, UT, United States of America

  • James Cook,

    Roles Data curation, Investigation, Methodology, Visualization, Writing – review & editing

    Affiliations George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT, United States of America, Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, UT, United States of America

  • Jian Ying,

    Roles Formal analysis, Investigation, Methodology, Writing – review & editing

    Affiliations Department of Population Health Sciences, Division of Health System Innovation and Research, University of Utah School of Medicine, Salt Lake City, UT, United States of America, George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT, United States of America

  • Guo Wei,

    Roles Data curation, Formal analysis, Investigation, Software, Writing – review & editing

    Affiliation George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT, United States of America

  • Jennifer S. Herrick,

    Roles Formal analysis, Funding acquisition, Investigation, Writing – review & editing

    Affiliations Department of Population Health Sciences, Division of Health System Innovation and Research, University of Utah School of Medicine, Salt Lake City, UT, United States of America, George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT, United States of America

  • Vanessa W. Stevens,

    Roles Conceptualization, Data curation, Supervision, Validation, Writing – review & editing

    Affiliations Department of Population Health Sciences, Division of Health System Innovation and Research, University of Utah School of Medicine, Salt Lake City, UT, United States of America, George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT, United States of America

  • Barbara E. Jones,

    Roles Conceptualization, Formal analysis, Investigation, Methodology, Supervision, Writing – review & editing

    Affiliations George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT, United States of America, Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, UT, United States of America

  • Libo Wang,

    Roles Investigation, Methodology, Visualization, Writing – review & editing

    Affiliation Department of Medicine, Division of Cardiology, University of Utah School of Medicine, Salt Lake City, UT, United States of America

  • Alexander R. Zheutlin,

    Roles Investigation, Methodology, Project administration, Writing – review & editing

    Affiliation Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, UT, United States of America

  • Andrew M. South,

    Roles Investigation, Methodology, Writing – review & editing

    Affiliations Department of Pediatrics, Section of Nephrology, Brenner Children’s Hospital, Wake Forest School of Medicine, Winston Salem, NC, United States of America, Division of Public Health Sciences, Department of Epidemiology and Prevention, Wake Forest School of Medicine, Winston Salem, NC, United States of America

  • Thomas C. Hanff,

    Roles Investigation, Methodology, Writing – review & editing

    Affiliation Department of Medicine, Division of Cardiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States of America

  • Steven M. Smith,

    Roles Investigation, Methodology, Writing – review & editing

    Affiliation Department of Pharmacotherapy and Translational Research, University of Florida College of Pharmacy, Gainesville, FL, United States of America

  • Rhonda M. Cooper-DeHoff,

    Roles Investigation, Methodology, Writing – review & editing

    Affiliations Department of Pharmacotherapy and Translational Research, University of Florida College of Pharmacy, Gainesville, FL, United States of America, Department of Medicine, University of Florida, College of Medicine, Gainesville, FL, United States of America

  • Jordan B. King,

    Roles Investigation, Methodology, Visualization, Writing – review & editing

    Affiliations Department of Population Health Sciences, Division of Health System Innovation and Research, University of Utah School of Medicine, Salt Lake City, UT, United States of America, Institute for Health Research, Kaiser Permanente Colorado, Aurora, CO, United States of America

  • G. Caleb Alexander,

    Roles Investigation, Methodology, Writing – review & editing

    Affiliation Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD

  • Dan R. Berlowitz,

    Roles Investigation, Methodology, Writing – review & editing

    Affiliations Department of Public Health; University of Massachusetts Lowell, Lowell, MA, United States of America, Edith Nourse Rogers Memorial Veterans Hospital, Bedford, MA, United States of America

  • Faraz S. Ahmad,

    Roles Investigation, Methodology, Writing – review & editing

    Affiliation Department of Medicine, Division of Cardiology, Northwestern University Feinberg School of Medicine, Chicago, IL, United States of America

  • M. Jason Penrod,

    Roles Formal analysis, Funding acquisition, Methodology, Visualization, Writing – review & editing

    Affiliation Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, UT, United States of America

  • Rachel Hess,

    Roles Investigation, Methodology, Writing – review & editing

    Affiliations Department of Population Health Sciences, Division of Health System Innovation and Research, University of Utah School of Medicine, Salt Lake City, UT, United States of America, Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, UT, United States of America

  • Molly B. Conroy,

    Roles Investigation, Methodology, Writing – review & editing

    Affiliations Department of Population Health Sciences, Division of Health System Innovation and Research, University of Utah School of Medicine, Salt Lake City, UT, United States of America, Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, UT, United States of America

  • James C. Fang,

    Roles Investigation, Methodology, Writing – review & editing

    Affiliation Department of Medicine, Division of Cardiology, University of Utah School of Medicine, Salt Lake City, UT, United States of America

  • Michael A. Rubin,

    Roles Investigation, Methodology, Writing – review & editing

    Affiliations George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT, United States of America, Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, UT, United States of America

  • Srinivasan Beddhu,

    Roles Investigation, Methodology, Writing – review & editing

    Affiliation Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, UT, United States of America

  • Alfred K. Cheung,

    Roles Investigation, Methodology, Project administration, Writing – review & editing

    Affiliation Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, UT, United States of America

  • Weiming Xian,

    Roles Investigation, Methodology, Writing – review & editing

    Affiliations Edith Nourse Rogers Memorial Veterans Hospital, Bedford, MA, United States of America, Department of Pharmacology and Experimental Therapeutics, Boston University School of Medicine, Boston, MA, United States of America

  • William S. Weintraub,

    Roles Investigation, Methodology, Writing – review & editing

    Affiliation MedStar Washington Hospital Center, Washington, DC, United States of America

  •  [ ... ],
  • Adam P. Bress

    Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Writing – original draft, Writing – review & editing

    Affiliations Department of Population Health Sciences, Division of Health System Innovation and Research, University of Utah School of Medicine, Salt Lake City, UT, United States of America, George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT, United States of America

  • [ view all ]
  • [ view less ]

Correction

13 May 2021: The PLOS ONE Staff (2021) Correction: Angiotensin II receptor blocker or angiotensin-converting enzyme inhibitor use and COVID-19-related outcomes among US Veterans. PLOS ONE 16(5): e0251897. https://doi.org/10.1371/journal.pone.0251897 View correction

Abstract

Background

Angiotensin II receptor blockers (ARBs) and angiotensin-converting enzyme inhibitors (ACEIs) may positively or negatively impact outcomes in severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection. We investigated the association of ARB or ACEI use with coronavirus disease 2019 (COVID-19)-related outcomes in US Veterans with treated hypertension using an active comparator design, appropriate covariate adjustment, and negative control analyses.

Methods and findings

In this retrospective cohort study of Veterans with treated hypertension in the Veterans Health Administration (01/19/2020-08/28/2020), we compared users of (A) ARB/ACEI vs. non-ARB/ACEI (excluding Veterans with compelling indications to reduce confounding by indication) and (B) ARB vs. ACEI among (1) SARS-CoV-2+ outpatients and (2) COVID-19 hospitalized inpatients. The primary outcome was all-cause hospitalization or mortality (outpatients) and all-cause mortality (inpatients). We estimated hazard ratios (HR) using propensity score-weighted Cox regression. Baseline characteristics were well-balanced between exposure groups after weighting. Among outpatients, there were 5.0 and 6.0 primary outcomes per 100 person-months for ARB/ACEI (n = 2,482) vs. non-ARB/ACEI (n = 2,487) users (HR 0.85, 95% confidence interval [CI] 0.73–0.99, median follow-up 87 days). Among outpatients who were ARB (n = 4,877) vs. ACEI (n = 8,704) users, there were 13.2 and 14.8 primary outcomes per 100 person-months (HR 0.91, 95%CI 0.86–0.97, median follow-up 85 days). Among inpatients who were ARB/ACEI (n = 210) vs. non-ARB/ACEI (n = 275) users, there were 3.4 and 2.0 all-cause deaths per 100 person months (HR 1.25, 95%CI 0.30–5.13, median follow-up 30 days). Among inpatients, ARB (n = 1,164) and ACEI (n = 2,014) users had 21.0 vs. 17.7 all-cause deaths, per 100 person-months (HR 1.13, 95%CI 0.93–1.38, median follow-up 30 days).

Conclusions

This observational analysis supports continued ARB or ACEI use for patients already using these medications before SARS-CoV-2 infection. The novel beneficial association observed among outpatients between users of ARBs vs. ACEIs on hospitalization or mortality should be confirmed with randomized trials.

Introduction

Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) binds to angiotensin-converting enzyme 2 (ACE2) to enter and infect host cells [1]. Under homeostasis, ACE2 converts angiotensin II to angiotensin-(1–7), reducing blood pressure and inflammation [26]. Angiotensin II receptor blockers (ARBs) and ACE inhibitors (ACEIs) inhibit the renin-angiotensin system and may differentially influence ACE2 expression in select organs [7, 8]. During the ongoing pandemic, there has been unprecedented interest in whether ARBs and ACEIs are beneficial or harmful in patients with SARS-CoV-2 infection or coronavirus disease 2019 (COVID-19) [9].

At least twelve randomized clinical trials are testing the effects of ARB or ACEI continuation, initiation, or discontinuation on COVID-19 outcomes, two of which are now completed and reported no increased risk of harm with continuation of ARB or ACEI therapy among hemodynamically stable adults hospitalized for COVID-19 [10, 11]. Numerous retrospective studies examined the association of ARBs and ACEIs with COVID-19 outcomes using case-control and historical cohort study designs and generally reported no excess risk of SARS-CoV-2 infection or COVID-19 severity among ARB or ACEI users compared to non-users, with some suggesting potential benefit from ARBs or ACEIs [12]. However, several of the prior observational studies contain methodological limitations that may reduce their internal validity, and thus, their clinical applicability [13, 14]. Despite the volume of published observational analyses to date (more than 80 as of Feb 15, 2020) [12], focused, high-quality observational studies are needed to complement randomized trials and guide ARB or ACEI use during the growing COVID-19 pandemic. Unfortunately, due to the short duration of the pandemic limiting sample size, it is not feasible to perform a new-user design, which is considered best practice for estimating medication effects outside of randomized trials [15]. However, unlike many prior observational analyses, we utilized several methods to increase robustness of results including an active comparator design, rigorous adjustment for a rich set of covariates, and negative control analyses to assess for residual bias.

ARBs and ACEIs are among the most commonly prescribed medications in the United States (US). The potential risks of abrupt discontinuation are enormous, and their effects on the ongoing COVID-19 pandemic are not clear. Our goal was to estimate the direction and magnitude of association between ARB and ACEI use with COVID-19-related outcomes among US Veterans with treated hypertension in the outpatient and inpatient settings using Veterans Health Administration (VHA) data and robust observational methods.

Methods

Data source and extraction

We used outpatient and inpatient demographic, pharmacy, clinical, and healthcare utilization data from the national VHA to conduct this retrospective cohort study. The study was pre-registered with ClinicalTrials.gov (NCT04467931), where the protocol is publicly available. The University of Utah institutional review board and Salt Lake City Veterans Affairs Health Care System Research and Development Office, approved the study with a waiver of informed consent.

Cohort derivation

We derived separate outpatient and inpatient cohorts for analyses. Veterans were eligible for inclusion as outpatients if they had a positive SARS-CoV-2 test performed between January 19, 2020 (the day before the first confirmed COVID-19 case in the US) and October 15, 2020 but were not hospitalized in the seven days before the positive test (S1 Fig, Panel A); the date of the positive SARS-CoV-2 test was the index date. Veterans were eligible for inclusion as inpatients if they had a positive SARS-CoV-2 test performed between January 19, 2020 and October 15, 2020 and were hospitalized for COVID-19; the index date was the first hospital admission date on or after the date where the Veteran started receiving SARS-CoV-2 testing (S1 Fig, Panel B). We identified COVID-19 cases using the definitions developed by the VHA for public reporting [16, 17]. Nursing home residents could be included in the outpatient and inpatient cohorts if meeting all other eligibility criteria.

We excluded Veterans who did not meet one-year continuous enrollment criteria prior to the index date or who had data inconsistencies (e.g., not alive on index date or multiple death dates). Finally, we restricted the analyses to Veterans with treated hypertension, defined as ≥1 inpatient or ≥2 outpatient encounters with an International Classification of Diseases, Ninth Revision (ICD-9) or Tenth Revision (ICD-10) code (401.x, 403.0x, 403.1x, 403.9x, I10, I12.0, or I12.9) [18] using all available data pre-index, and ≥1 outpatient pharmacy fill for an antihypertensive medication in the 104-day pre-index period (S1 Table). We selected 104 days to account for patients who receive 90-day supply prescriptions, plus a 14-day grace period to account for modest non-adherence.

Exposures

For both outpatient and inpatient cohorts, we assigned Veterans to exposure groups based on ≥1 outpatient pharmacy antihypertensive medication fill in the 104-day pre-index period. We compared two medication exposures among outpatients and inpatients, separately. First, we compared current users of any ARB and/or ACEI, irrespective of concurrent non-ARB/ACEI antihypertensive therapy, to Veterans using neither an ARB nor ACEI (current user of an ARB/ACEI-based regimen vs. non-ARB/ACEI-based regimen, S2 Table). For this comparison, because diagnoses other than hypertension (i.e., compelling indications) for ARB or ACEI use would introduce confounding by indication, we excluded Veterans with compelling indications defined in the 2017 American College of Cardiology/American Heart Association blood pressure guidelines [19] (i.e., diabetes, chronic kidney disease, coronary heart disease, heart failure with reduced ejection fraction, or prior stroke). We choose, a priori, to restrict the study population for this comparison to only those without compelling indications because we believe that the confounding by indication for this specific comparison (ACEI/ARB users compared to non-users) would be insurmountable.

Our second exposure compared current users of an ARB vs. ACEI (current user of an ARB vs. an ACEI). For this comparison, we included Veterans with and without compelling indications for ARBs and ACEIs, and we excluded Veterans who were current users of both an ARB and ACEI. For this comparison, restriction to those without compelling indications is not needed because these compelling indications are shared between both medications; ACEIs and ARBs are used interchangeably as they are thought to be equivalent in benefit and safety [19, 20]. As such, for this comparison, we included Veterans with and without compelling indications for ARBs and ACEIs, and we excluded Veterans who were current users of both an ARB and ACEI. Due to the small sample size of inpatient Veterans without compelling indications who were ARB/ACEI- vs. non-ARB/ACEI-based antihypertensive regimen users, we considered comparisons of these two groups to be exploratory.

Outcomes

For outpatients, the primary outcome was first occurrence of all-cause hospitalization or all-cause mortality, and secondary outcomes were all-cause hospitalization, all-cause mortality, and ICU admission, separately (definitions in S1 Table). For outpatients, Veterans were censored at the first occurrence of an outcome or October 15, 2020. For inpatients, the primary outcome was first occurrence of all-cause mortality, and secondary outcomes were ICU admission, dialysis, or mechanical ventilation, separately. For inpatients, Veterans were censored at the first occurrence of an outcome, 30 days’ follow-up, or October 15, 2020.

Covariates

We determined baseline demographics, vital signs, laboratory measurements, and concomitant medications within the one year prior to the index date (i.e., the “pre-index period”; S1 Table). We defined vital signs and laboratory results using the value closest to the index date in the pre-index period, and we defined comorbid conditions using all available claims and medical record data prior to the index date (S1 Table). We also calculated a Charlson Comorbidity Index [21].

Propensity score estimation

To adjust for measured confounding among outpatients and inpatients separately, we generated propensity scores (PS) to estimate the probability of being an ARB/ACEI- vs. non-ARB/ACEI-based antihypertensive regimen user and, separately, of being an ARB user vs. being an ACEI user, using logistic regression with all baseline covariates included in the PS model (S3 Table) [22]. The regions of common support were assessed using histograms of the PS between medication exposure groups (S2-S3 Figs). Next, using the PS, we calculated each Veteran’s matching weight [23] and verified covariate balance in the matching weighted populations using absolute standardized mean differences [24].

Statistical analyses

All analyses were performed in the outpatient and inpatient cohorts, separately. We compared patient characteristics between current users of ARB/ACEI- vs. non-ARB/ACEI-based antihypertensive regimens and ARB users vs. ACEI users before and after weighting. To account for missing data (S4-S7 Tables), we used multiple imputation with chained equations to provide 10 imputed datasets using Rubin’s formulae [25, 26]. We estimated hazard ratios (HR) for the associations between the medication exposure groups and outcomes using matching-weighted Cox regression. The use of matching-weights provides a weighted analogue to 1:1 paired PS matching. We applied a two-sided alpha of 0.05 for all hypothesis tests, without correction for multiple comparisons.

Subgroup and sensitivity analyses

All analyses, including fitting logistic regression models to estimate PS, were repeated in subgroups according to age, sex, race-ethnicity, body mass index, and the number of antihypertensive medication classes being taken.

In sensitivity analyses, we repeated Cox regression analyses comparing medication regimens using several different covariate adjustment strategies (Supplemental Methods in S1 File). To investigate if the observed associations were sensitive to the definition of the medication exposure window, we repeated all analyses with several alternative definitions (see S1 File). Finally, we repeated analyses using a composite of gastrointestinal bleeding or urinary tract infection as negative control outcomes.

All analyses were completed using R v.4.0.2 (R Foundation for Statistical Computing, Vienna, Austria).

Results

Patient characteristics

After applying all inclusion and exclusion criteria, a total of 4,969 outpatient Veterans without compelling indications were included for analysis, of which there were 2,482 and 2,487 users of an ARB/ACEI- and non-ARB/ACEI-based antihypertensive regimen, respectively (S4 Fig). Of 13,581 outpatient Veterans with and without compelling indications, 4,877 were current users of an ARB, and 8,704 were current users of an ACEI.

Of 485 inpatient Veterans without compelling indications, there were 210 and 275 users (i.e., filled a prescription in the 104 days prior to the admission) of an ARB/ACEI- and non-ARB/ACEI-based antihypertensive regimen, respectively (S4 Fig). Of 3,178 inpatient Veterans with and without compelling indications, 1,164 were current users of an ARB, and 2,014 were current users of an ACEI.

Among both outpatients and inpatients, the majority of ARB/ACEI, non-ARB/ACEI, ARB, and ACEI users were men, obese, and had comorbid depression (S4-S7 Tables). The majority of outpatients and inpatients who were ARB users and ACEI users also had diabetes and/or chronic kidney disease. There were no differences in baseline characteristics after weighting (S5 Fig, Panels A-D, all absolute standardized mean differences <0.1).

Outcomes

Over a median 87 days of follow-up among outpatients, there were 5.0 vs. 6.0 all-cause hospitalizations or deaths per 100 person-months among ARB/ACEI-based users vs. non-ARB/ACEI-based regimen users (HR 0.85, 95% CI 0.73–0.99; Table 1 and Fig 1 Panel A). When comparing ARB users vs. ACEI users in outpatients, there were 13.2 vs. 14.8 all-cause hospitalizations or deaths per 100 person-months over a median 85 days of follow-up (HR 0.91, 95% CI 0.86–0.97; Table 1 and Fig 1 Panel B). There was a 9.0% lower relative risk of all-cause hospitalization (HR 0.91, 95% CI 0.85–0.97) and 16.0% lower relative risk of ICU admission (HR 0.84, 95% CI 0.71–0.98) among outpatient ARB users vs. ACEI users (S6 Fig).

thumbnail
Fig 1.

Matching weight-adjusted cumulative hazard curves for all-cause hospitalization or mortality among outpatient Veterans who were current users of an ARB/ACEI- vs. non-ARB/ACEI-based antihypertensive regimen (Panel A) and ARB user vs. ACEI user (Panel B). Cumulative incidence curves demonstrating the risk for all-cause hospitalization or mortality in outpatients between exposure groups were generated from matching-weighted Cox regression models and bootstrap resampling with 2,000 iterations. Veterans are censored at the first occurrence of all-cause hospitalization, all-cause mortality, or October 15, 2020. Numbers at risk in table are unweighted. Abbreviations: ACEI: angiotensin-converting enzyme inhibitor; ARB: angiotensin II receptor blocker; CI: confidence interval; HR: hazard ratio.

https://doi.org/10.1371/journal.pone.0248080.g001

thumbnail
Table 1. Matching weight-adjusted incidence rates and hazard ratios for the primary and secondary outcomes among outpatient Veterans with treated hypertension who were current users of an ARB/ACEI- vs. non-ARB/ACEI-based antihypertensive regimen and ARB user vs. ACEI user, separately.

https://doi.org/10.1371/journal.pone.0248080.t001

Among inpatients, over a median 30 days of follow-up, there was no statistically significant difference in all-cause mortality events in the exploratory analysis of ARB/ACEI-based users (i.e., filled a prescription in the 104 days prior to the admission) vs. non-ARB/ACEI-based regimen users (HR 1.25, 95% CI 0.30–5.13; Table 2 and Fig 2 Panel A). Over a median 30 days of follow-up, there were 21.0 vs. 17.7 all-cause deaths among inpatient ARB users vs. ACEI users (HR 1.13, 95% CI 0.93–1.38; Table 2 and Fig 2 Panel B). There were no statistically significant differences between exposure groups for secondary outcomes of ICU admission, dialysis, and mechanical ventilation (Table 2 and S7 Fig).

thumbnail
Fig 2.

Matching weight-adjusted cumulative hazard curves for all-cause mortality among inpatient Veterans who were current users of an ARB/ACEI- vs. non-ARB/ACEI-based antihypertensive regimen (Panel A) and ARB user vs. ACEI user (Panel B). Cumulative incidence curves demonstrating the risk for all-cause mortality in inpatients between exposure groups were generated from matching-weighted Cox regression models and bootstrap resampling with 2,000 iterations. Veterans are censored at the first occurrence of all-cause mortality, 30 days’ follow-up, or October 15, 2020. Numbers at risk in table are unweighted. Due to small sample size, results displayed in Panel A are exploratory. Abbreviations: ACEI: angiotensin-converting enzyme inhibitor; ARB: angiotensin II receptor blocker; CI: confidence interval; HR: hazard ratio.

https://doi.org/10.1371/journal.pone.0248080.g002

thumbnail
Table 2. Matching weight-adjusted incidence rates and hazard ratios for the primary and secondary outcomes among inpatient Veterans with treated hypertension who were current users of an ARB/ACEI- vs. non-ARB/ACEI-based antihypertensive regimen and ARB user vs. ACEI user, separately.

https://doi.org/10.1371/journal.pone.0248080.t002

Subgroup and sensitivity analyses

Among outpatients taking less than three antihypertensive medication classes, users of an ARB/ACEI- vs. non-ARB/ACEI-based regimen appeared have a stronger association with all-cause hospitalization or mortality (HR 0.74, 95% CI 0.63–0.88) compared to those taking three or more antihypertensive medication classes (HR 1.52, 95% CI 0.91–2.71, p-value for interaction 0.003, S8 Table). Outpatient non-Hispanic Black Veterans taking an ARB vs. an ACEI also appeared to have a stronger association with all-cause hospitalization or mortality (HR 0.85, 95% CI 0.77–0.93) compared to Veterans of other race-ethnicities (HR 0.94, 95% CI 0.87–1.02, p-value for interaction 0.036, S9 Table). Conversely, inpatient non-Hispanic Black Veterans appeared to have a 48% higher relative risk of all-cause mortality if taking an ARB vs. an ACEI (HR 1.48, 95% CI 1.06–2.08), which was not observed among other race-ethnicities (HR 0.95, 95% CI 0.72–1.24, p-value for interaction 0.025, S10 Table). Sample size was not adequate to allow for subgroup analyses for the inpatient ARB/ACEI- vs. non-ARB/ACEI-based antihypertensive regimen comparison.

Sensitivity analysis using different covariate adjustment strategies and varying definitions of the medication exposure window did not materially change the results in either outpatients or inpatients (S11-S15 Tables). There was no association between any of the main exposures in either outpatients or inpatients for the negative control outcomes (S13-S14 Tables).

Discussion

This historical cohort study of ARB and ACEI use among outpatient and inpatient Veterans with treated hypertension positive for SARS-CoV-2 offers at least two main findings. First, among outpatients, current use of an ARB/ACEI was associated with a 15% lower relative risk of COVID-19-related outcomes compared to a non-ARB/ACEI-based antihypertensive medication regimen. Second, this is one of the first reports demonstrating that among outpatients, ARB users had a 9% lower relative risk of all-cause hospitalization or mortality compared to ACEI users. These findings support continued use of renin-angiotensin system blockade for (1) outpatients already using an ARB or ACEI, and (2) continued use of an ARB for outpatients using an ARB prior to testing positive for SARS-CoV-2. For current outpatient users of ACEIs prior to testing positive for SARS-CoV-2 in clinical practice, the HR of 0.91 for all-cause hospitalization or mortality with ARB vs. ACEI is insufficiently far from 1.0 to recommend converting from an ACEI to an ARB given the prevalent user design limitations. Randomized trials are needed to confirm the observed potential beneficial effects of ARBs compared to ACEIs on COVID-19-related outcomes.

The mechanistic basis for the potential effects of ARB or ACEI use on COVID-19-related outcomes is centered around the physiology of ACE2, the protein which SARS-CoV-2 uses for host cell entry. ACE2 upregulation in the lungs by ARBs or ACEIs could enhance SARS-CoV-2 interaction with its receptor, increase infection of type II alveolar epithelial cells, and increase COVID-19 severity. The contrasting hypothesis maintains that ARBs or ACEIs could mitigate viral-induced inflammatory responses by reducing available angiotensin II or by blocking angiotensin II’s actions via the angiotensin II type 1 receptor (AT1R), thus shifting the balance of the renin-angiotensin system back towards ACE2 and angiotensin-(1–7). This would, thereby, promote anti-inflammatory and anti-fibrotic effects in the lungs and possibly mitigate SARS-CoV-2-induced acute lung injury [2, 27]. Animal experiments and human studies have not conclusively demonstrated the presence, direction, or magnitude of the association between ARBs or ACEIs and ACE2 levels in the circulation or tissue expression, especially in the lungs [7, 8, 2836].

Two small, randomized trials reported no increased risk of harm with continuation of ARB or ACEI therapy among hemodynamically stable adults hospitalized for COVID-19 [10, 11]. The current report adds to these smaller trials by reinforcing the randomized trial results in a larger scale observational study in a diverse population. Also, our analysis evaluated SARS-CoV-2+ outpatients’ risk of developing COVID-19, a population not tested in these trials. Furthermore, our inpatient cohort was older, and our analysis directly compared ARB vs ACEI users, which these smaller trials were not adequately powered to evaluate. High-quality observational studies have potential to augment trial results and provide helpful insights into the association of ARB and ACEI use and COVID-19 outcomes.

The current analysis may be the first observational study to show a protective association among outpatients between current users of an ARB compared to ACEI on COVID-19-related outcomes. Current users of an ARB in the outpatient cohort had a 9% lower relative risk of all-cause hospitalization or mortality and 16% lower relative risk of ICU admission compared to ACEI users. There is a mechanistic basis for the differential effects between ARBs and ACEIs. It has been hypothesized that ARBs and ACEIs have differential effects on ACE2 [7, 8]. In healthy adult humans, ACEIs, but not ARBs, may increase duodenal ACE2 expression [37]. In animals, ARB-mediated AT1R antagonism reduces peripheral and pulmonary inflammation, fibrosis, and edema [38, 39]. In addition, angiotensin II can induce beneficial effects through the AT2R, and ARBs and ACEIs differentially activate secondary pathways within the renin-angiotensin system, leading to alternative angiotensin II generation and metabolism [2]. Recent data suggest that bradykinin-mediated inflammation may contribute to adverse outcomes in patients with COVID-19 [40]. Because ACEIs inhibit bradykinin metabolism, this is another plausible mechanism for differences in COVID-19-related outcomes between ACEI users and ARB users.

Understanding the extent to which use of ARBs and ACEIs is beneficial or harmful in COVID-19 is a complicated yet critical knowledge gap because hypertension is highly prevalent among Americans with more severe COVID-19-related outcomes [9]. Ideally, we would have larger randomized trials to address these gap. The recently completed trials and prior observational investigations have generally found no association between users of an ARB/ACEI vs. non-ARB/ACEI and COVID-19-related outcomes [12]. However, in contrast to the prior analyses largely conducted among inpatients or those with and without COVID-19, the current analysis identified a significant 15% reduction in all-cause hospitalization or mortality among SARS-CoV-2-positive outpatients with hypertension taking ARB/ACEI- vs. non-ARB/ACEI-based regimens. The difference in results may be due to a larger sample size in the current analysis (i.e., increased power), more specific population (i.e., only those with treated hypertension), use of a PS matching-weighted analysis, or differences in exposure definitions compared to prior analyses. The heterogeneity in study design, study populations, exposure definitions, and statistical analyses may explain contrasting results between observational research investigating ARB and ACEI use and COVID-19-related outcomes.

In analyses evaluating ARB vs. ACEI use among inpatients, we did not find evidence of a significant association between ARB vs. ACEI users and all-cause mortality. This is in contrast to the significantly lower risk of all-cause hospitalization or mortality among ARB users compared to ACEI users in outpatients. One explanation for this inconsistency is the concept of “depletion of the susceptibles,” in which ARB users who are susceptible to the beneficial effects of ARBs on all-cause hospitalization or mortality were less likely to be hospitalized, and therefore, were less likely to be included in the inpatient analysis [14]. The restriction of the inpatients to those who did not benefit from ARBs as outpatients could bias results towards observing a benefit in the ACEI group. Accordingly, we caution against over-interpretation of the directionality of the relationship between ARBs vs. ACEIs in inpatients with COVID-19. Additionally, because we ascertained ARB and ACEI use in the three months prior to SARS-CoV-2 infection, our results should not be applied to acute changes in medication therapy at the time of COVID-19 infection or during hospitalization.

The current analysis has several strengths. This was a highly focused report with two primary exposures and two cohorts utilizing detailed and validated electronic health record and pharmacy data from the largest integrated health care delivery system in the US. We included several sensitivity analyses to assess for residual confounding and robustness of results. The current study also has several limitations. The external validity of our findings may be limited by a predominantly male and white population. Because we restricted the analysis comparing ARB/ACEI- vs. non-ARB/ACEI-based regimens users to only those without a compelling indication for an ARB or an ACEI to increase internal validity, our results may not be generalizable to adults with diabetes, chronic kidney disease, coronary heart disease, heart failure with reduced ejection fraction, or prior stroke. The study question necessitated a prevalent-user design, which could introduce bias. Our active-comparator design and methods of covariate adjustment mitigate these sources of bias, as is evident through the balance in exchangeability achieved between exposure groups before and after weighting. In addition, the results of the negative control analyses suggested a low risk of substantial residual bias. We were not able to estimate the impact of initiating ARB or ACEI in patients not previously receiving treatment, stopping treatment in inpatients already receiving treatment, or, finally, switching patients between ARB and ACEI after developing COVID-19; our findings can be interpreted within the clinical question of continuing an ARB vs. continuing an ACEI during infection with SARS-CoV-2. We did not capture healthcare utilization external to the VHA, however, we do not believe there would be differential ascertainment by exposure groups. We did not adjust for multiple comparisons. Instead, consistent with the recommendations from Rothman [41], Perneger [42], Feise [43], and Althouse [44], we provided extensive information and detail on the study design, methods, and statistical analyses and report effect sizes, confidence intervals, and p-values so that the readers can make their own interpretation of our findings in the context of the other recent reports on this topic. Data on symptoms at the time of SARS-CoV-2 testing were unavailable. Due to regional differences in the acceleration of COVID-19 cases and resource availability in the US, decisions on which patients with COVID-19 were admitted likely varied by facility and over time according to clinical judgment, local bed availability and patient acuity. Therefore, the severity of hospitalized patients with COVID-19 included in the inpatient analysis may have varied by region and over time.

Conclusion

COVID-19 is a persistent public health crisis that has infected more than 27 million and killed more than 475,000 US adults as of February 15, 2020. More than 45 million US adults use ARBs and ACEIs annually. Unlike prior randomized trials and observational analyses, we provide novel evidence that for SARS-CoV-2-positive outpatients, use of an ARB compared to ACEI in the three months prior to infection was associated with significantly lower risk of all-cause hospitalization or mortality. Randomized trials are needed to confirm the signal of potential beneficial effects of ARBs compared to ACEIs on COVID-19-specific outcomes. We also confirmed results of prior randomized trials and observational reports which found no evidence of a harmful association of ARBs or ACEIs with adverse COVID-19-related outcomes compared to use of non-ARB/ACEI antihypertensive medications. In light of the serious risks of discontinuation of ARBs and ACEIs in patients with hypertension, our findings support professional society recommendations to continue ARB or ACEI treatment among patients with hypertension in the absence of an established indication to discontinue them.

Acknowledgments

The authors would like to acknowledge the team who developed and deployed the VA’s National Surveillance Tool, which informed many of the data elements for this research. This work was also supported using resources and facilities at the VA Salt Lake City Health Care System with funding from the VA Informatics and Computing Infrastructure. Finally, the support and resources from the Center for High Performance Computing at the University of Utah are gratefully acknowledged.

References

  1. 1. Hoffmann M, Kleine-Weber H, Schroeder S, Kruger N, Herrler T, Erichsen S, et al. SARS-CoV-2 Cell Entry Depends on ACE2 and TMPRSS2 and Is Blocked by a Clinically Proven Protease Inhibitor. Cell. 2020;181(2):271–80 e8. Epub 2020/03/07. pmid:32142651; PubMed Central PMCID: PMC7102627.
  2. 2. South AM, Diz DI, Chappell MC. COVID-19, ACE2, and the cardiovascular consequences. Am J Physiol Heart Circ Physiol. 2020;318(5):H1084–H90. Epub 2020/04/02. pmid:32228252; PubMed Central PMCID: PMC7191628.
  3. 3. Xue H, Zhou L, Yuan P, Wang Z, Ni J, Yao T, et al. Counteraction between angiotensin II and angiotensin-(1–7) via activating angiotensin type I and Mas receptor on rat renal mesangial cells. Regul Pept. 2012;177(1–3):12–20. Epub 2012/05/09. pmid:22561449.
  4. 4. Rice GI, Thomas DA, Grant PJ, Turner AJ, Hooper NM. Evaluation of angiotensin-converting enzyme (ACE), its homologue ACE2 and neprilysin in angiotensin peptide metabolism. Biochem J. 2004;383(Pt 1):45–51. Epub 2004/07/31. pmid:15283675; PubMed Central PMCID: PMC1134042.
  5. 5. Ren Y, Garvin JL, Carretero OA. Vasodilator action of angiotensin-(1–7) on isolated rabbit afferent arterioles. Hypertension. 2002;39(3):799–802. Epub 2002/03/19. pmid:11897767.
  6. 6. Andreatta-van Leyen S, Romero MF, Khosla MC, Douglas JG. Modulation of phospholipase A2 activity and sodium transport by angiotensin-(1–7). Kidney Int. 1993;44(5):932–6. Epub 1993/11/01. pmid:8264152.
  7. 7. Ferrario CM, Jessup J, Gallagher PE, Averill DB, Brosnihan KB, Ann Tallant E, et al. Effects of renin-angiotensin system blockade on renal angiotensin-(1–7) forming enzymes and receptors. Kidney Int. 2005;68(5):2189–96. Epub 2005/10/14. pmid:16221218.
  8. 8. Ferrario CM, Jessup J, Chappell MC, Averill DB, Brosnihan KB, Tallant EA, et al. Effect of angiotensin-converting enzyme inhibition and angiotensin II receptor blockers on cardiac angiotensin-converting enzyme 2. Circulation. 2005;111(20):2605–10. Epub 2005/05/18. pmid:15897343.
  9. 9. Harky A, Chor CYT, Nixon H, Jeilani M. The controversy of using angiotensin-converting enzyme inhibitors and angiotensin receptor blockers in COVID-19 patients. J Renin Angiotensin Aldosterone Syst. 2021;22(1):1470320320987118. Epub 2021/01/09. pmid:33412991; PubMed Central PMCID: PMC7797594.
  10. 10. Lopes RD, Macedo AVS, de Barros ESPGM, Moll-Bernardes RJ, Dos Santos TM, Mazza L, et al. Effect of Discontinuing vs Continuing Angiotensin-Converting Enzyme Inhibitors and Angiotensin II Receptor Blockers on Days Alive and Out of the Hospital in Patients Admitted With COVID-19: A Randomized Clinical Trial. JAMA. 2021;325(3):254–64. Epub 2021/01/20. pmid:33464336; PubMed Central PMCID: PMC7816106.
  11. 11. Cohen JB, Hanff TC, William P, Sweitzer N, Rosado-Santander NR, Medina C, et al. Continuation versus discontinuation of renin-angiotensin system inhibitors in patients admitted to hospital with COVID-19: a prospective, randomised, open-label trial. Lancet Respir Med. 2021. Epub 2021/01/11. pmid:33422263; PubMed Central PMCID: PMC7832152.
  12. 12. Mackey K, Kansagara D, Vela K. Update Alert 7: Risks and Impact of Angiotensin-Converting Enzyme Inhibitors or Angiotensin-Receptor Blockers on SARS-CoV-2 Infection in Adults. Ann Intern Med. 2021. Epub 2021/01/05. pmid:33395346; PubMed Central PMCID: PMC7791405.
  13. 13. Edmonston DL, South AM, Sparks MA, Cohen JB. Coronavirus Disease 2019 and Hypertension: The Role of Angiotensin-Converting Enzyme 2 and the Renin-Angiotensin System. Adv Chronic Kidney Dis. 2020;27(5):404–11. Epub 2020/12/15. pmid:33308506; PubMed Central PMCID: PMC7334971.
  14. 14. Cohen JB, D’Agostino-McGowan L, Jensen ET, Rigdon J, South AM. Evaluating sources of bias in observational studies of angiotensin-converting enzyme inhibitor/angiotensin II receptor blocker use during coronavirus disease 2019: beyond confounding. J Hypertens. 2020. Epub 2020/11/14. pmid:33186321.
  15. 15. Lund JL, Richardson DB, Sturmer T. The active comparator, new user study design in pharmacoepidemiology: historical foundations and contemporary application. Curr Epidemiol Rep. 2015;2(4):221–8. Epub 2016/03/10. pmid:26954351; PubMed Central PMCID: PMC4778958.
  16. 16. Stratford D. National Surveillance Tool assesses readiness across VA’s health system. VAntage Point Off. Blog US Dep. Veterans Aff. 2020. Available at: https://www.blogs.va.gov/VAntage/74896/national-surveillance-tool-assesses-readiness-across-vas-health-system/. Accessed September 3, 2020.
  17. 17. US Department of Veterans Affairs. COVID-19 National Summary—VA Access to Care. 2020. Available at: https://www.accesstocare.va.gov/Healthcare/COVID19NationalSummary.
  18. 18. Quan H, Khan N, Hemmelgarn BR, Tu K, Chen G, Campbell N, et al. Validation of a case definition to define hypertension using administrative data. Hypertension. 2009;54(6):1423–8. Epub 2009/10/28. pmid:19858407.
  19. 19. Whelton PK, Carey RM, Aronow WS, Casey DE Jr., Collins KJ, Dennison Himmelfarb C, et al. 2017 ACC/AHA/AAPA/ABC/ACPM/AGS/APhA/ASH/ASPC/NMA/PCNA Guideline for the Prevention, Detection, Evaluation, and Management of High Blood Pressure in Adults: Executive Summary: A Report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines. Hypertension. 2018;71(6):1269–324. Epub 2017/11/15. pmid:29133354.
  20. 20. Williams B, Mancia G, Spiering W, Agabiti Rosei E, Azizi M, Burnier M, et al. 2018 ESC/ESH Guidelines for the management of arterial hypertension. Eur Heart J. 2018;39(33):3021–104. Epub 2018/08/31. pmid:30165516.
  21. 21. Quan H, Sundararajan V, Halfon P, Fong A, Burnand B, Luthi JC, et al. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care. 2005;43(11):1130–9. Epub 2005/10/15. pmid:16224307.
  22. 22. Rosenbaum PR, Rubin DB. Reducing bias in observational studies using subclassification on the propensity score. J Am Stat Assoc. 1984;79(387):516–24.
  23. 23. Li L, Greene T. A weighting analogue to pair matching in propensity score analysis. Int J Biostat. 2013;9(2):215–34. Epub 2013/08/02. pmid:23902694.
  24. 24. Franklin JM, Rassen JA, Ackermann D, Bartels DB, Schneeweiss S. Metrics for covariate balance in cohort studies of causal effects. Stat Med. 2014;33(10):1685–99. Epub 2013/12/11. pmid:24323618.
  25. 25. White IR, Royston P, Wood AM. Multiple imputation using chained equations: Issues and guidance for practice. Stat Med. 2011;30(4):377–99. Epub 2011/01/13. pmid:21225900.
  26. 26. Rubin DB, Schenker N. Multiple imputation in health-care databases: an overview and some applications. Stat Med. 1991;10(4):585–98. Epub 1991/04/01. pmid:2057657.
  27. 27. Sparks MA, South A, Welling P, Luther JM, Cohen J, Byrd JB, et al. Sound Science before Quick Judgement Regarding RAS Blockade in COVID-19. Clin J Am Soc Nephrol. 2020;15(5):714–6. Epub 2020/03/30. pmid:32220930; PubMed Central PMCID: PMC7269218.
  28. 28. Wysocki J, Lores E, Ye M, Soler MJ, Batlle D. Kidney and Lung ACE2 Expression after an ACE Inhibitor or an Ang II Receptor Blocker: Implications for COVID-19. J Am Soc Nephrol. 2020;31(9):1941–3. Epub 2020/07/17. pmid:32669323; PubMed Central PMCID: PMC7461678.
  29. 29. Burchill LJ, Velkoska E, Dean RG, Griggs K, Patel SK, Burrell LM. Combination renin-angiotensin system blockade and angiotensin-converting enzyme 2 in experimental myocardial infarction: implications for future therapeutic directions. Clin Sci (Lond). 2012;123(11):649–58. Epub 2012/06/22. pmid:22715807.
  30. 30. Sukumaran V, Veeraveedu PT, Lakshmanan AP, Gurusamy N, Yamaguchi K, Ma M, et al. Olmesartan medoxomil treatment potently improves cardiac myosin-induced dilated cardiomyopathy via the modulation of ACE-2 and ANG 1–7 mas receptor. Free Radic Res. 2012;46(7):850–60. Epub 2012/04/14. pmid:22497476.
  31. 31. Zhong JC, Ye JY, Jin HY, Yu X, Yu HM, Zhu DL, et al. Telmisartan attenuates aortic hypertrophy in hypertensive rats by the modulation of ACE2 and profilin-1 expression. Regul Pept. 2011;166(1–3):90–7. Epub 2010/09/22. pmid:20854846.
  32. 32. Velkoska E, Dean RG, Burchill L, Levidiotis V, Burrell LM. Reduction in renal ACE2 expression in subtotal nephrectomy in rats is ameliorated with ACE inhibition. Clin Sci (Lond). 2010;118(4):269–79. Epub 2009/08/25. pmid:19698082; PubMed Central PMCID: PMC2782317.
  33. 33. Soler MJ, Ye M, Wysocki J, William J, Lloveras J, Batlle D. Localization of ACE2 in the renal vasculature: amplification by angiotensin II type 1 receptor blockade using telmisartan. Am J Physiol Renal Physiol. 2009;296(2):F398–405. Epub 2008/11/14. pmid:19004932.
  34. 34. Epelman S, Tang WH, Chen SY, Van Lente F, Francis GS, Sen S. Detection of soluble angiotensin-converting enzyme 2 in heart failure: insights into the endogenous counter-regulatory pathway of the renin-angiotensin-aldosterone system. J Am Coll Cardiol. 2008;52(9):750–4. Epub 2008/08/23. pmid:18718423; PubMed Central PMCID: PMC2856943.
  35. 35. Hamming I, van Goor H, Turner AJ, Rushworth CA, Michaud AA, Corvol P, et al. Differential regulation of renal angiotensin-converting enzyme (ACE) and ACE2 during ACE inhibition and dietary sodium restriction in healthy rats. Exp Physiol. 2008;93(5):631–8. Epub 2008/01/15. pmid:18192334.
  36. 36. Ishiyama Y, Gallagher PE, Averill DB, Tallant EA, Brosnihan KB, Ferrario CM. Upregulation of angiotensin-converting enzyme 2 after myocardial infarction by blockade of angiotensin II receptors. Hypertension. 2004;43(5):970–6. Epub 2004/03/10. pmid:15007027.
  37. 37. Vuille-dit-Bille RN, Camargo SM, Emmenegger L, Sasse T, Kummer E, Jando J, et al. Human intestine luminal ACE2 and amino acid transporter expression increased by ACE-inhibitors. Amino Acids. 2015;47(4):693–705. Epub 2014/12/24. pmid:25534429.
  38. 38. Benicky J, Sanchez-Lemus E, Honda M, Pang T, Orecna M, Wang J, et al. Angiotensin II AT1 receptor blockade ameliorates brain inflammation. Neuropsychopharmacology. 2011;36(4):857–70. Epub 2010/12/15. pmid:21150913; PubMed Central PMCID: PMC3055735.
  39. 39. Sanchez-Lemus E, Benicky J, Pavel J, Saavedra JM. In vivo Angiotensin II AT1 receptor blockade selectively inhibits LPS-induced innate immune response and ACTH release in rat pituitary gland. Brain Behav Immun. 2009;23(7):945–57. Epub 2009/05/12. pmid:19427376; PubMed Central PMCID: PMC2749886.
  40. 40. Garvin MR, Alvarez C, Miller JI, Prates ET, Walker AM, Amos BK, et al. A mechanistic model and therapeutic interventions for COVID-19 involving a RAS-mediated bradykinin storm. Elife. 2020;9. Epub 2020/07/08. pmid:32633718; PubMed Central PMCID: PMC7410499.
  41. 41. Rothman KJ. No adjustments are needed for multiple comparisons. Epidemiology. 1990:43–6. pmid:2081237
  42. 42. Perneger TV. What’s wrong with Bonferroni adjustments. Bmj. 1998;316(7139):1236–8. pmid:9553006
  43. 43. Feise RJ. Do multiple outcome measures require p-value adjustment? BMC medical research methodology. 2002;2(1):1–4. pmid:12069695
  44. 44. Althouse AD. Adjust for Multiple Comparisons? It’s Not That Simple. Ann Thorac Surg. 2016;101(5):1644–5. Epub 2016/04/24. pmid:27106412.