Inter-observer reliability of DSM-5 substance use disorders☆
Introduction
The fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5), published in May 2013, included several changes to the Substance Use Disorders (SUD) diagnostic category, which were intended to address limitations in the DSM-IV SUD diagnoses. Specifically, DSM-5 omitted the criterion related to “Illegal Acts,” added a “Craving” criterion, and combined abuse and dependence into a single SUD diagnosis with three severity levels: mild (2–3 criteria), moderate (4–5 criteria), and severe (6 or more criteria) (Hasin et al., 2013).
Several studies supported the decision to omit the “Illegal Acts” criterion on the basis of low endorsement rates (Gilder et al., 2014, Hasin et al., 2012, Kerridge et al., 2011, Saha et al., 2012). Although the Craving criterion did not provide additional information in the context of the other changes (Keyes et al., 2011), it was added to ensure that the DSM-5 SUD diagnosis was comparable to the International Classification of Diseases (ICD) diagnosis, which includes craving as a diagnostic criterion (Hasin et al., 2013, Keyes et al., 2011, Mewton et al., 2013). Studies that used Item Response Theory (IRT) analyses consistently demonstrated a single dimension underlying the 11 DSM-5 SUD criteria, the severity of which varied on a continuum, justifying the use of a single diagnosis of SUD (Borges et al., 2010, Gilder et al., 2014, Hasin et al., 2012, Kerridge et al., 2011, Preuss et al., 2014, Saha et al., 2012). Studies that examined the impact of these changes on the estimated prevalence of SUDs showed that the performance of the DSM-5 criteria was similar to that of the DSM-IV criteria, though DSM-5 captured DSM-IV “diagnostic orphans” (i.e., individuals meeting only 1 or 2 DSM-IV dependence criteria and no DSM-IV abuse criteria) (Kelly et al., 2014, Mewton et al., 2013, Peer et al., 2013, Proctor et al., 2012).
The studies that led to the development of DSM-5 focused primarily on psychometric properties of the DSM-5 SUD diagnoses, rather than assessing their reliability. The DSM-IV substance dependence diagnoses have been shown to be reliable, with substance abuse diagnoses less so (Hasin et al., 1997, Hasin et al., 2006, Pierucci-Lagha et al., 2007, Pierucci-Lagha et al., 2005). Because DSM-5 criteria include 10 of the 11 DSM-IV abuse and dependence criteria, DSM-5 SUD criteria are also likely to be reliable (Hasin et al., 2013). Moreover, the empirical basis for the changes embodied in DSM-5 would predict that the reliability of SUD diagnoses would be at least as good as for DSM-IV diagnoses (Kraemer et al., 2012). However, a formal evaluation of the reliability of the DSM-5 SUD diagnoses is needed, as evidence of the reliability of DSM-5 would increase the utility of DSM-5 for research and policy purposes (Hasin et al., 2013). To date, the only published study of the reliability of any DSM-5 SUD was from the DSM-5 field trials in the United States and Canada, where the reliability of the DSM-5 diagnosis of alcohol use disorder was considered to be good (intraclass kappa (κ) = 0.40, 95% CI = 0.27–0.54) (Regier et al., 2013).
The primary objectives of the current study were (1) to examine the inter-observer reliability of DSM-5 alcohol, opioid, cocaine, and cannabis use disorder diagnoses and criteria measured using the Semi-Structured Assessment for Drug Dependence and Alcoholism (SSADDA) and (2) to compare the reliability of these diagnoses with that of SSADDA-derived DSM-IV diagnoses in the same sample of 173 individuals recruited from a variety of settings. Consistent with the results of other studies, we found that SUDs commonly co-occurred with other SUDs (Regier et al., 2013, Stinson et al., 2005) and psychiatric disorders (Compton et al., 2007, Conway et al., 2006). Because diagnostic criteria for one disorder could mimic those of another disorder, thereby confounding the diagnoses, we also evaluated the effect of comorbid disorders on the reliability of DSM-5 SUD diagnoses.
Section snippets
Sample
The study was a secondary analysis of data from a study conducted in 2004–2005 in a sample of 173 subjects in which the inter-observer reliability of DSM-IV SUD diagnoses (Pierucci-Lagha et al., 2007, Pierucci-Lagha et al., 2005) was evaluated. The subjects were recruited from addiction treatment facilities at the University of Connecticut Health Center (UConn) and the Yale University School of Medicine (Yale) (n = 103), inpatient and outpatient psychiatric services at UConn (n = 29), and community
Sample characteristics
The sample comprised 87 men (50.3%) and 86 women (49.7%), with an average age of 37.5 years (SD = 10.4). Subjects were predominantly European-American (38.7%), African-American (43.9%) or Hispanic (11.2%), and never married (62.4%). Most subjects (68.2%) had at least a high school education (mean years of education = 12.9, SD = 2.4), and 19.2% reported having a current full-time job. Most subjects (75.3%) had at least one SUD, which most commonly involved alcohol, followed by cocaine, cannabis and
Discussion
The inter-observer reliability of the DSM-5 diagnoses of alcohol, opioid, cocaine and cannabis use disorders was good-to-excellent. When evaluated in relation to the severity of SUD (mild, moderate, or severe), the reliabilities, although good to excellent for all substances except cannabis, were modestly lower than for any SUD. Most of the discrepancies between the two interviews occurred between the categories of mild and moderate SUD severity: the greater the prevalence of mild SUD
Role of funding source
The sponsors had no role in the study design, collection, analysis or interpretation of the data, or in the writing of the manuscript.
Contributors
Cécile Denis, Joel Gelernter, Amy Hart, and Henry Kranzler had full access to all of the data in the study and take responsibility for the integrity of the data and accuracy of the data analysis.
Study concept and design: Henry Kranzler, Joel Gelernter. Analysis and interpretation of data: Cécile Denis, Henry Kranzler, Amy Hart. Statistical analysis: Cécile Denis. Obtained funding: Henry Kranzler, Joel Gelernter. Study supervision: Henry Kranzler. All authors contributed to and approved the
Conflict of interest statement
Cécile Denis, Joel Gelernter, and Amy Hart have no disclosures to make. Henry Kranzler has been a consultant or advisory board member for Alkermes, Lilly, Lundbeck, Otsuka, and Pfizer and is a member of the American Society of Clinical Psychopharmacology's Alcohol Clinical Trials Initiative, which is supported by AbbVie, Ethypharm, Lilly, Lundbeck, and Pfizer.
Acknowledgments
The authors thank the COGA investigators for generously allowing them to modify the SSAGA interview for our use. We also thank the participants and interviewers at the participating sites and Joseph Cubells, M.D., Ph.D. and Lindsay Farrer, Ph.D. for their contributions to the initial data collection and management, respectively. This study was supported by NIH grants DA12422, DA12849, DA12690, DA028874, and AA13736.
References (34)
- et al.
Item response theory analyses of DSM-IV and DSM-5 stimulant use disorder criteria in an American Indian community sample
Drug Alcohol Depend.
(2014) - et al.
The Alcohol Use Disorder and Associated Disabilities Interview Schedule-5 (AUDADIS-5): reliability of substance use and psychiatric disorder modules in a general population sample
Drug Alcohol Depend.
(2015) - et al.
The Alcohol Use Disorder and Associated Disabilities Interview Schedule (AUDADIS): reliability of alcohol and drug modules in a clinical sample
Drug Alcohol Depend.
(1997) - et al.
Analyses related to the development of DSM-5 criteria for substance use related disorders: 2. proposed DSM-5 criteria for alcohol, cannabis, cocaine and heroin disorders in 663 substance abuse patients
Drug Alcohol Depend.
(2012) - et al.
Concordance between DSM-5 and DSM-IV nicotine, alcohol, and cannabis use disorder diagnoses among pediatric patients
Drug Alcohol Depend.
(2014) - et al.
Dimensionality of hallucinogen and inhalant/solvent abuse and dependence criteria: implications for the diagnostic and statistical manual of mental disorders—fifth edition
Addict. Behav.
(2011) - et al.
Prevalence of DSM-IV and DSM-5 alcohol, cocaine, opioid, and cannabis use disorders in a largely substance dependent sample
Drug Alcohol Depend.
(2013) - et al.
Reliability of DSM-IV diagnostic criteria using the semi-structured assessment for drug dependence and alcoholism (SSADDA)
Drug Alcohol Depend.
(2007) - et al.
Diagnostic reliability of the semi-structured assessment for drug dependence and alcoholism (SSADDA)
Drug Alcohol Depend.
(2005) - et al.
Compatibility of current DSM-IV and proposed DSM-5 diagnostic criteria for cocaine use disorders
Addict. Behav.
(2012)
Analyses related to the development of DSM-5 criteria for substance use related disorders: 1. Toward amphetamine, cocaine and prescription drug use disorder continua using Item Response Theory
Drug Alcohol Depend.
Comorbidity between DSM-IV alcohol and specific drug use disorders in the United States: results from the National Epidemiologic Survey on Alcohol and Related Conditions
Drug Alcohol Depend.
Statistical methods for assessing agreement between two methods of clinical measurement
Lancet
The dimensionality of alcohol use disorders and alcohol consumption in a cross-national perspective
Addiction
A new, semi-structured psychiatric interview for use in genetic linkage studies: a report on the reliability of the SSAGA
J. Stud. Alcohol
Reliability of individual diagnostic criterion items for psychoactive substance dependence and the impact on diagnosis
J. Stud. Alcohol
Developing criteria for establishing interrater reliability of specific items: applications to assessment of adaptive behavior
Am. J. Ment. Defic.
Cited by (24)
Identification of opioid use disorder using electronic health records: Beyond diagnostic codes
2023, Drug and Alcohol DependenceConstruct validity of DSM-5 cannabis use disorder diagnosis and severity levels in adults with problematic substance use
2022, Journal of Psychiatric ResearchReliability and validity of DSM-IV and DSM-5 methamphetamine use disorder diagnoses using the Chinese Version of the Semi-Structured Assessment for Drug Dependence and Alcoholism (SSADDA)
2021, Drug and Alcohol DependenceCitation Excerpt :The information collected with the SSADDA enabled us to generate both DSM-IV and DSM-5 diagnoses and compare the psychometric performance of the two diagnostic systems with respect to methamphetamine (MA) diagnoses for the SSADDA Chinese version. Previously Denis et al. (2015) compared the two diagnostic systems for alcohol, opioid, cocaine, and cannabis use disorders using the English version of the SSADDA and found that DSM-5 SUD diagnoses showed reliability that was comparable or greater than that of DSM-IV diagnoses. However, there is currently no study examining the reliability or validity of DSM-5 lifetime diagnoses for MA using SSADDA.
Test-retest reliability and cross-cultural applicability of DSM-5 adopted diagnostic criteria for ketamine use disorders
2021, Drug and Alcohol DependenceCitation Excerpt :These differences may be attributed to low base rates for mild and moderate diagnoses. However, severe ketamine use disorder had a low base rate and the highest level of reliability, which is consistent with prior studies that have shown that lower substance use disorder severity in baseline interviews is associated with more discordant interviews at retest, since a difference of one or two criteria can result in the presence or absence of a diagnosis for mild disorders (Denis et al., 2015; Hasin et al., 2006, 2020). Our findings also demonstrate reliability both within and between study sites for the diagnosis of an overall ketamine disorder and severe ketamine disorder, with no significant differences found between sites for these diagnoses.
Test-retest reliability of DSM-5 substance disorder measures as assessed with the PRISM-5, a clinician-administered diagnostic interview
2020, Drug and Alcohol DependenceCitation Excerpt :Of many exploratory tests, few were significant, suggesting that these factors had little impact on reliability and supporting the generalizability of the results. However, low SUD severity in the initial interview did predict discordant interviews at retest, consistent with earlier studies (Hasin et al., 2006b; Hasin et al., 1996; Denis et al., 2015; Grant et al., 2015a; Hasin et al., 1997a; Grant et al., 1995). In mild disorders, a difference of only one or two criteria can determine the presence or absence of a diagnosis, which is not the case for severe disorders.
- ☆
Supplementary material can be found by accessing the online version of this paper at http://dx.doi.org/10.1016/j.drugalcdep.2015.05.019.