Reliability and Validity of the Substance Abuse Outcomes Module

Abstract

Back to table of contents Previous article Next article ArticleFull AccessReliability and Validity of the Substance Abuse Outcomes ModuleG. Richard Smith M.D.M. Audrey Burnam Ph.D.Cynthia L. Mosley B.B.A.Jan A. Hollenberg M.S.Mike Mancino M.D.Wen Grimes M.A.G. Richard Smith M.D.Search for more papers by this authorM. Audrey Burnam Ph.D.Search for more papers by this authorCynthia L. Mosley B.B.A.Search for more papers by this authorJan A. Hollenberg M.S.Search for more papers by this authorMike Mancino M.D.Search for more papers by this authorWen Grimes M.A.Search for more papers by this authorPublished Online:1 Oct 2006AboutSectionsPDF/EPUB ToolsAdd to favoritesDownload CitationsTrack Citations ShareShare onFacebookTwitterLinked InEmail The estimated annual cost for substance abuse treatment in the United States ranges from $16 to $18 billion ( 1 , 2 ). In 2001 annual expenditures for alcohol treatment services and drug abuse treatment were estimated at $9.7 billion and $8.5 billion, respectively ( 1 ). As of March 31, 2003, approximately 1.1 million patients were in treatment ( 3 ). To better serve these patients and keep costs to a minimum, clinicians must be aware of treatments that work effectively and of outcomes expected from these treatments. Observational studies indicate that, on average, substance abuse treatments are effective ( 4 , 5 ), and controlled studies support the efficacy of specific treatment interventions or approaches ( 6 ). However, a wide variety of substance abuse treatments exist, and costs are substantial. Additional information about types of patients who would benefit from specific treatments and the amount of treatment that is appropriate to achieve the best patient outcomes at an affordable cost could be very helpful. Thus another assessment tool for routine care could provide knowledge about what works in substance abuse treatment settings, which, in turn, could help improve patient outcomes. Posttreatment follow-up of patients in substance abuse treatment is important because of the disorder's chronic, relapsing nature. Thus a self-report measure is particularly important. To address this concern, we evaluated the Substance Abuse Outcomes Module (SAOM), a self-report instrument developed several years ago that does not require a trained interviewer, is brief, assesses substance use and various social and functioning consequences, and collects data about substance use diagnosis, relapse, and remission.This article presents field test results about SAOM's ability to assess the outcomes of substance abuse treatment as part of routine clinical care. A preliminary, prospective field test was used to examine the SAOM's utility, specifically the reliability and construct validity of its key patient characteristics and patient outcome measures and the SAOM's responsiveness to change over time among individuals treated for substance abuse and dependence. The SAOM also has the potential to improve care on a broader scale and more indirectly—through systems of care or through measures of competencies or performance. Although routine assessment of outcomes in substance abuse treatment settings is not common, some provider organizations have started incorporating this information technology into their systems of care. Two examples are the Veterans Affairs clinical programs for substance abuse treatment ( 7 ) and the treatment outcomes and performance pilot studies supported by the Center for Substance Abuse Treatment ( 8 ), which require routine outcome assessment with the Addiction Severity Index (ASI) ( 9 ). The Accreditation Council on Graduate Medical Education started requiring residency training programs, including psychiatry programs, to document resident competencies in practice-based learning and improvement ( 10 ). The American Board of Psychiatry and Neurology has plans to begin requiring documentation of competencies for psychiatrists who participate in the Maintenance of Certification Program ( 11 ). In addition, the Centers for Medicare and Medicaid Services of the U.S. Department of Health and Human Services is about to implement a pay-for-performance plan in which physicians will be reimbursed at a higher rate for meeting certain practice-based performance measures ( 12 ). Routine outcomes assessments, such as the SAOM, will likely be one way to measure outcomes for systems of care and to document competencies and performance measures. MethodsSubstance Abuse Outcomes Module The SAOM is the latest in a series of six modules ( 13 , 14 , 15 , 16 , 17 , 18 ) developed to improve the understanding of relationships between patient characteristics, processes of care, and patient outcomes of care by using the minimum amount of information possible ( 19 ). The SAOM, like the other five outcomes modules, has both clinician and patient baseline assessment forms, which are completed upon diagnosis of a new episode of substance abuse. The patient follow-up assessment form is completed three and six months after baseline. A user's manual provides details about recommended protocols, scoring instructions, and instrument development. The SAOM is available for unlimited free use by permission: an electronic copy can be obtained at the authors' Web site (www.netoutcomes.net) and elsewhere ( 20 ). The SAOM uses the "tracer condition" approach, in which a single disorder is closely scrutinized in a given treatment setting. This disease-specific approach is the standard throughout medicine ( 21 ). These characteristics make the SAOM most appropriate for use in substance treatment settings and also for use with patients who have primary substance problems in general psychiatric settings. The SAOM would not be an appropriate assessment tool for patients with a primary diagnosis other than substance abuse or dependence who have a comorbid or secondary substance use disorder. A comprehensive review of the literature on key patient characteristics, processes of care, and outcomes of care for substance abuse treatment was the basis for the SAOM's development. The module includes constructs that have been shown in multiple studies to be related to treatment outcomes; the module emphasizes collection of the minimum amount of data needed to understand key relationships in order to improve care ( 19 ). The SAOM was also designed to measure the types and outcomes of care received for substance abuse and characteristics that influence outcomes and types of care given; it includes criteria to identify diagnostically similar groups of patients. Table 1 lists the domains measured by each SAOM component as well as the instruments used to construct the module ( 22 , 23 , 24 , 25 , 26 , 27 , 28 ). Table 1 Domains and content of the Substance Abuse Outcomes ModuleTable 1 Domains and content of the Substance Abuse Outcomes ModuleEnlarge tablePreliminary feasibility testing included field testing of a prototype module in multiple substance abuse treatment settings. After revision based on field test results, the prototype module was used for the study reported here. The version evaluated in this study is a 113-item self-report assessment designed to be used in a paper-and-pencil format, although it can be administered orally in person or via telephone and also by computer. Written at a third-grade reading level and currently available only in English, the SAOM takes 20 minutes, on average, to complete, according to preliminary feasibility testing. The SAOM makes current diagnoses (substance abuse or dependence) according to the particular substance of abuse and includes a variety of response formats (Likert scales, yes-no formats, and list selections).Design of the field test Data on the SAOM's reliability and validity were collected in 1999 and 2000 as part of a longitudinal study of 100 patients who were identified at the beginning of treatment for an episode of DSM-IV alcohol or drug abuse or dependence. At baseline patients completed the SAOM, and a trained research assistant administered accompanying validation instruments. Approximately three months later, patients were recontacted to complete the SAOM's self-administered follow-up portion and a research assistant again administered the accompanying validation instruments. The approval of the University of Arkansas for Medical Sciences Institutional Review Board was obtained before study initiation. Patient recruitmentClinicians conducting intake evaluations of consecutive admissions at an outpatient methadone treatment program and at two private inpatient substance abuse treatment facilities in Little Rock, Arkansas, referred eligible patients to the study. To restrict the study to those beginning a new episode of treatment, the admission had to be the first within the past six months and patients could not have had any outpatient treatment in the previous six months. Quota samples were drawn at admission to test the module across conditions in five groups on the basis of type of drug problem identified—alcohol, heroin or other opiates, cocaine or crack, marijuana, and hallucinogens. Eligible patients were at least 18 years old, had a primary diagnosis of alcohol or drug dependence or abuse, were able to understand and speak English, were medically stable, had no active psychoses, and had sufficient cognitive function to report on lifetime drinking and drug use. A research assistant met with each newly referred patient to fully explain the study's purpose and requirements and to obtain written informed consent to participate in the research. Patients were assured that all information collected from personal interviews and administered instruments would not be shared with the treatment team, that study results would be presented for groups of patients only, and that their course of treatment was in no way connected to information provided to the study researchers. Each participant agreed to be contacted for follow-up assessment regardless of treatment success. After meeting with the research assistant, six referred patients declined to participate in the study. The research assistant administered the Neurobehavioral Cognitive Status Examination ( 29 ) to verify that referred patients demonstrated sufficient cognitive function to report about substance use during the past three months. One person who scored in the range of mild to moderate impairment was excluded from the study. Validation instrumentsWidely used and highly regarded instruments that collect information on SAOM domains (substance use, diagnoses, and consequences) were used to evaluate the SAOM's content validity. The substance abuse module of the Composite International Diagnostic Interview (CIDI-SAM) ( 30 ) is a highly structured, detailed interview designed to ascertain specific diagnoses of substance abuse and dependence. The Addiction Severity Index, 5th edition (ASI), is a comprehensive, semistructured, interviewer-administered assessment tool ( 31 ). The Alcohol Use Disorder Identification Test (AUDIT) is an alcohol screening procedure developed by the World Health Organization that quantifies alcohol consumption and harmful consequences ( 32 , 33 ). The Diagnostic Interview Schedule (DIS) is a highly reliable, structured, lay-administered interview developed to assess psychiatric conditions ( 34 ). The DIS sections on depressive disorders and antisocial personality traits and disorders were used. The Timeline Follow-Back Assessment (TFBA) is a reliable, valid, interviewer-administered method to estimate alcohol consumption ( 35 ). The Inventory of Drug Use Consequences (INDUC-2R) is a self-report instrument designed to assess lifetime and recent consequences of drug or alcohol use, including physical consequences, intrapersonal consequences, interpersonal consequences, and impulse control ( 25 ). The Medical Outcomes Study Social Support Survey (MOS-SSS) is a self-report instrument developed as part of the Medical Outcomes Study to assess the extent of respondents' social support networks ( 36 ). An experienced, trained research assistant administered the CIDI-SAM, ASI, DIS sections, and TFBA. Patients were asked to complete the AUDIT, INDUC-2R, and MOS-SSS.Data collectionAt baseline patients were randomly assigned to complete the SAOM's patient baseline assessment either before or after the accompanying validation instruments to control for order effects; one-half (N=50) of the patients received the SAOM first. One-half of the sample, chosen by independent random selection from the groups with a drug problem, was asked to complete the module a second time on day 3 or 4 after the initial SAOM administration to examine test-retest reliability. Baseline assessments occurred in face-to-face interviews or were self-administered, depending on the administration protocol for each instrument. All patients completing the baseline assessment received a check for $30, and those completing the retest received an additional $30.At the time of follow-up, the SAOM and the validation instruments were readministered. Patients were considered lost to follow-up if they did not respond after several attempts to make phone or mail contact. All follow-up assessments had a target date three months after the baseline interview. Follow-up was conducted within a six-week window (two weeks before and four weeks after the target date). Follow-up interviews were completed by telephone, personal interview, or by mail, as appropriate for each assessment. As in the baseline assessment, the SAOM was administered to one-half of the patients before the other validation instruments and to the other half after the other validation instruments.Data analysisCronbach's alpha coefficients were calculated to examine the internal consistency and reliability of the multi-item SAOM. Test-retest reliability statistics were calculated for SAOM outcome measures with intraclass correlation coefficients (ICCs), and kappa coefficients were calculated for continuous and dichotomous SAOM measures. Concurrent validity of outcome measures was examined by analyzing the association between SAOM measures and the appropriate validation measures with Pearson or Spearman correlation coefficients for continuous measures and chi square and kappa statistics for dichotomous measures. To examine the extent of predictive utility for SAOM measures of case mix, changes in SAOM outcome measures between baseline and follow-up were examined as a function of case-mix measures with ordinary least-squares multiple regression. Sensitivity of SAOM outcome measures to change was examined by calculating effect size scores ( 37 ). ResultsSampleOf the 100 patients recruited at baseline, 32 were women. Ages ranged from 18 to 75 years, with a mean age of 40.7±9.6. The sample included 23 African-American patients, one Hispanic patient, and 76 white patients. Thirty-two were married or cohabitating, 19 were single and never married, 28 were divorced, 18 were separated, and three were widowed. Forty-six reported an annual household income of less than $20,000, 42 had incomes between $20,000 and $80,000, 11 reported incomes higher than $80,000, and one refused to answer this question. Twenty-nine had not completed high school, 22 were high school graduates, 28 attended some college, and 21 were college graduates. Of the 100 patients, who were all beginning treatment for a new episode of substance abuse or dependence, 64 were new admissions to inpatient treatment program A, 26 were new admissions to inpatient treatment program B, and ten were new admissions to the outpatient methadone treatment program. Many patients had multiple substance abuse and dependence diagnoses as is typical in substance abuse treatment settings. Ninety-one patients had a CIDI-SAM diagnosis of substance dependence, and 42 had a diagnosis of substance abuse (see Table 2 for specific diagnoses). Four did not meet DSM-IV criteria for dependence or abuse according to the CIDI-SAM interview, even though they had a clinical diagnosis of substance abuse or dependence when admitted to the treatment program. One patient who entered the study died before the three-month follow-up assessment, three could not be located, and three refused follow-up assessment. The remaining 93 patients completed the follow-up assessment. Table 2 Number of patients with diagnoses of substance abuse and dependence among 100 patients on admission to two inpatient treatment facilities or a methadone programTable 2 Number of patients with diagnoses of substance abuse and dependence among 100 patients on admission to two inpatient treatment facilities or a methadone programEnlarge tableAccuracy of self-reports of diagnostic criteriaThe accuracy of the SAOM's diagnostic measure was compared with the accuracy of diagnoses obtained with the research standard, the CIDI-SAM. At baseline, the SAOM's diagnostic component's 17 items had high internal consistency, with a Cronbach alpha coefficient of .89. Agreement between the SAOM diagnostic measure and the CIDI-SAM about the presence of a substance use diagnosis (abuse or dependence) was 93 percent. Nonagreement between the measures was evenly balanced between over- and underdiagnoses. The SAOM diagnostic measure was highly sensitive (96 percent) in detecting whether any substance abuse diagnosis was present. Because the probability of not having a diagnosis in this baseline clinical sample was very low (4 percent), calculation of specificity was inappropriate. A similar analysis was conducted for the accuracy of the SAOM's diagnostic measure at follow-up, when some patients' substance use disorders had remitted. The percentage agreement with the CIDI-SAM was high, with 90 percent agreement on the presence of any diagnosis and 89 percent agreement on the presence of abuse or dependence. The kappa coefficient (.76) suggested substantial agreement on the presence of any substance use diagnosis, based on the scoring criteria of Cohen ( 38 , 39 ). Reliability of key constructsTable 3 shows results of assessments of internal reliability (alpha) and test-retest reliability (ICC or kappa coefficient) for key constructs in the SAOM domains of patient characteristics and patient outcomes of care. Table 3 Reliability of key constructs of the Substance Abuse Outcomes ModuleTable 3 Reliability of key constructs of the Substance Abuse Outcomes ModuleEnlarge tablePatient characteristics . In addition to a diagnostic measure, the SAOM includes case-mix variables that have been shown in prior studies to predict the outcomes of substance abuse treatment. These variables include severity of abuse and dependence, parental abuse or dependence, age of onset of abuse or dependence, social support, comorbid medical conditions, previous treatment for abuse or dependence, support for sobriety, and presence of antisocial personality traits. Generally, the internal reliability of all case-mix variables was good to excellent, with alpha values from .58 to .90 and all but two values reaching .85 or above. The ICCs or kappa coefficients, as appropriate, were also strong, ranging from .56 to .96, with the values for most case-mix variables above .90 except for diagnosis, previous treatment, and support for sobriety. Patient outcomes. Patient outcome measures included assessments of substance use and consequences of substance use, as shown in Table 3 . Substance use variables included multiple measures of quantity and frequency of substance use. Because use measures were single items, only test-retest reliability is shown in Table 3 for those measures. Measures of consequence were multi-item scales that assessed physical, intrapersonal, interpersonal, impulse control, and social consequences of substance use. All consequence scales demonstrated high internal consistency, with alpha values from .72 to .83. Both categories showed moderate to high test-retest reliability, with most ICCs or kappa coefficients above .80 (range= .47-.99). Items below .60 were total number of drinks on heavy drinking days, polydrug use days in the past month, intrapersonal consequences, and impulse consequences. Validity of key constructsConcurrent validity.Table 4 presents concurrent validity data for key patient case-mix characteristics and patient outcome variables assessed in the SAOM. Correlations of the SAOM case-mix variables with the corresponding research standard ranged from .36 to .94. Correlations were .54 or higher, except for severity of drug abuse or dependence at baseline and comorbid medical conditions. All chi square values were highly significant, and all kappa coefficient values were in the moderate to high range ( 35 , 36 ). Table 4 Concurrent validity of patient variables at baseline and follow-up as measured by the Substance Abuse Outcomes ModuleTable 4 Concurrent validity of patient variables at baseline and follow-up as measured by the Substance Abuse Outcomes ModuleEnlarge tableFor patient outcomes, correlations between the module variables and the research standards were high, ranging from .47 to .99. All values exceeded .60, except the values for alcohol quantity and frequency at baseline. The chi square and kappa coefficient comparisons were also highly significant. The SAOM's validity in determining remission of substance use disorders according to DSM-IV criteria is important, because follow-up remission rates are often used as an indication of quality of care. The SAOM's follow-up diagnostic assessment was compared with that of the CIDI-SAM in a 2×2 analysis of remission. Agreement (no-no and yes-yes) occurred in 69 of the 81 cases (85 percent). In one case (1 percent), the SAOM diagnostic assessment indicated remission, but the CIDI-SAM did not. In 11 cases (14 percent), the SAOM assessment underreported remission compared with the CIDI-SAM. The SAOM diagnostic assessment, compared with the CIDI-SAM, had 83 percent sensitivity to determine remission, 94 percent specificity, 98 percent positive predictive value, 39 percent negative predictive value, and a kappa coefficient of .64. Predictive validity. The predictive validity of baseline patient case-mix characteristics in regard to patient outcomes at follow-up was tested. The SAOM's case-mix variables in the baseline assessment included severity of dependence or abuse, age of onset of dependence or abuse, parental substance abuse or dependence, previous substance abuse treatment, social support, support for sobriety, number of comorbid medical conditions, depression, and antisocial traits. Table 5 lists patient outcomes and notes which case-mix variables significantly predicted them. Statistically significant findings (p≤.01) and trends (p<.01 and p≤.05) are reported for each case-mix measure, along with the overall multiple correlation coefficient (R 2 ) value when the entire set of case-mix predictors was included. More stringent statistical criteria were used because of multiple comparisons. The R 2 value represents the total amount of variance in patient outcomes explained by all case-mix measures. Table 5 Validity of baseline patient characteristics as predictors of patient outcomes at follow-up, as measured by the Substance Abuse Outcomes ModuleTable 5 Validity of baseline patient characteristics as predictors of patient outcomes at follow-up, as measured by the Substance Abuse Outcomes ModuleEnlarge tableEach case-mix measure significantly predicted at least one patient outcome. The majority of case-mix measures predicted outcomes in hypothesized directions. Two predicted in a direction opposite of what was expected, but the values were not statistically significant (between .05 and .10). These two measures were parental substance abuse, which predicted fewer social consequences (rather than more), and severity of substance abuse, which predicted fewer reduced-activity days (rather than more).Sensitivity of key constructs to clinical change Analyses examined the SAOM's sensitivity to clinically important change. Change scores between baseline and follow-up outcome measures were calculated, then effect-size scores for each outcome variable were calculated as the mean change score divided by the standard deviation of the baseline mean score ( 34 ). The effect size was interpreted as defined by Cohen ( 34 ): .20 or less is a small effect, .20 to .50 is a moderate effect, and .80 or greater is a large effect. A positive effect size represents improvement, and a negative effect size denotes worsening of the condition. All SAOM effect-size scores were positive, indicating that the SAOM variables changed in the clinically expected direction. Most scores were in the moderate or large ranges, indicating that the SAOM measures were sensitive to important clinical changes. DiscussionOn the basis of this preliminary study, the SAOM appears to be a reasonably reliable and valid instrument for use in routine substance abuse treatment settings, particularly those that are similar to settings reported here for patients with substance abuse or dependence as a primary diagnosis. The SAOM provides self-report data regarding the presence of any diagnosis of substance abuse or dependence, key patient case-mix characteristics, and specific outcomes related to substance abuse or dependence, including substance use, consequences of use, relapse, and remission. The SAOM was not designed as a clinical research instrument; it was specifically designed to systematically assess outcomes of care in routine clinical settings in order to monitor or improve patient outcomes ( 19 ). These types of clinical assessment tools need to be brief, preferably based on patient self-report, and to measure key case-mix characteristics as well as outcomes. An ideal outcome module would take only moments of the patient's time and no clinician time. The 20 minutes for patient completion of the SAOM at baseline and follow-up and the 1.5 minutes for clinician completion (clinician assessment) at baseline appear to be a reasonable trade-off for the potential improvement in care and assurance of good treatment outcomes. Future studies are needed to determine the SAOM's usefulness in monitoring and improving patient outcomes. Advantages of the SAOM Although making diagnoses is the province of clinicians, estimates of diagnoses can be helpful in some quality improvement efforts. The SAOM's ability to estimate substance use diagnoses and remission according to DSM-IV criteria eliminates the expense of training and labor for outcomes assessment interviewers. Other advantages are the convenience to patients, who can select times to complete the SAOM; diagnostic confirmation, which can help in the reimbursement process; and determination of remission, which is becoming an increasingly important measure of successful treatment in other areas of medicine and may also become a helpful concept in substance abuse treatment. Assessing patient case-mix characteristics is critical when treatment programs are compared in nonexperimental evaluations. Systematic assessment of these variables is needed to show that differences in programs are due to differences in treatment effectiveness rather than differences in disease severity of patient populations. Outcome assessment that fails to take case-mix characteristics into account is, therefore, of little use in efforts to improve care. Tools such as the SAOM have the potential to shed light on this important concept and to focus on processes of care that determine patient outcomes.LimitationsThis study had several limitations. First, the study was primarily of inpatients in substance abuse treatment who were beginning a new episode of care (ten patients were from an outpatient methadone clinic), and most substance abuse treatment in the United States is outpatient treatment. Although this design concept was intentional to provide a relatively homogeneous group of patients with sufficient diversity to have good external validity (generalizability), the number of outpatients brings into question the SAOM's applicability or utility in outpatient care.In addition, most assessments were of patients who had insurance for at least detoxification treatment and were initially treated as inpatients in substance abuse treatment units. It is unknown whether these results can be generalized to public-sector or uninsured outpatients and their treatment programs.Further research is also needed to clearly understand the effects of variations in case mix o