PROTOCOL: Effectiveness of adult employment assistance services for persons with autism spectrum disorders

Abstract

The first paper identifying a condition called autism was published in 1943 (Kanner, 1943). Today, the condition is referred to as autism spectrum disorders (ASD) and refers to a range of mild to severe forms of neurological disorders marked by impairment in social functioning, communication, and repetitive and unusual patterns of behavior. According to the Centers for Disease Control and Prevention (CDC), the rate of autism is increasing among children and now indicates that 1 in 150 children are identified with ASD (CDC, 2008). Functional limitations caused by ASD continue into adulthood and often create barriers to independent living and stable long-term employment (Autism Society of America, 2008). As the number of individuals diagnosed with ASD rises and the number of students with ASD exiting public school settings rises, attention is increasingly focused on potential employment outcomes for individuals with ASD. Individuals without severe disabilities are eight times more likely to be employed than individuals with very severe disabilities (National Organization on Disability, 2000). Individuals with ASD are among those least likely to be employed (Cameto, Marder, Wagner, & Cardoso, 2003; Dew & Alan, 2007). Although economic conditions and employer attitudes affect employment opportunities, employment outcomes for individuals with ASD can be improved by appropriately addressing specific behaviors common among individuals with ASD (Schaller & Yang, 2005). Studies indicate that people with ASD are more likely to lose their employment for behavioral and social interaction reasons rather than their inability to perform work tasks (Dew & Alan, 2007; Hurlbutt & Chalmers, 2004; Unger, 1999). Behavior management challenges affecting employment must be understood and addressed consistently by employment support service providers in order to effectively facilitate obtaining and maintaining employment by individuals with ASD. Relatively few employment support service providers including state vocational rehabilitation counselors have an in-depth understanding of services that are associated with developing successful employment outcomes for individuals with ASD (Dew & Alan, 2007). Each state in the United States has at least one state agency that is charged with the provision of vocational rehabilitation services to facilitate employment outcomes for eligible individuals with disabilities. Data accumulated through the national network of state vocational rehabilitation agencies indicate that few individuals with autism are requesting services and of those that do most are not successfully employed as an outcome of the service. However, in its latest report, the national vocational rehabilitation service system indicates in fiscal year 2007 it provided services that successfully placed 1,774 individuals with ASD into employment situations that continued for 90 or more days (Rehabilitation Services Administration, 2009). Contribution of the Review There has been research done in the area of autism and employment, but such research has not been reported in a comprehensive review format before. Thus, a systematic review on the effectiveness of adult employment assistance for individuals with ASD would not only provide ideas for further research but also show findings that can inform implementation of such research. More needs to be known about strategies that are successful in securing employment outcomes with individuals with ASD. Enhanced awareness of effective approaches is likely to increase the availability of more effective employment support services for individuals with ASD, such as job coaching, vocational rehabilitation, or supported employment. A review of studies on adult employment assistance would not only synthesize the effects and techniques of successful or unsuccessful interventions, but also call to attention the gaps in experimental research in this area. To determine the effectiveness of adult employment assistance in securing and maintaining employment for adults (18 years and older) with autism spectrum disorders. A two-stage process will be used to determine inclusion or exclusion of studies: (1) title and abstract stage and (2) full text stage. Studies will be assessed and selected for advancement to the next stage of inclusion if evaluation meets at least one of the two following criteria: The participant sample of study must include only adults 18 years or older and no longer enrolled in a school-to-work program or secondary-level education program, with a diagnosis of Autism Spectrum Disorder. The review is intended to identify effective interventions that could be implemented by adult service program agencies; therefore, secondary school and other employment programs conducted in school settings with students will be excluded. The authors plan to conduct a second review on the effectiveness of employment assistance for youth and transition-age participants currently enrolled in an educational setting context. The focus of the study intervention must center on the topic of employment. The types of employment to be included are competitive, supported, or integrated employment. Sheltered work or non-integrated work will be excluded from the review. Studies that report a range of effectiveness standards from high to low will be included in the review to determine the overall effectiveness of adult employment assistance. Studies that exclusively address the acquisition of job-related skills/behaviors without demonstrating actual employment placement and maintenance of employment will not be included in this review. Studies focusing on job skill acquisition and subsequent employment will be included in this review. If these criteria are not clear from the title or abstract, the study may be advanced for retrieval of a full text to determine eligibility. A full-text of all citations/abstracts advanced from Stage 1 will be retrieved for a final determination for inclusion in the review and analysis. All of the following criteria must be met by each study in order to be included in the review and analysis. Studies will use an experimental (RCT), quasi-experimental design (QED), or single subject experimental design (SSED) design to report the effects of the intervention. In addition, qualitative studies (primarily case studies) and correlational design studies (COR) will be retrieved and reported to enrich description of intervention effects. The recipients of the intervention must be individuals with an ASD who are 18 years of age or older and are voluntarily seeking assistance in obtaining employment. Study participants with ASD that are eligible for inclusion are individuals with Asperger Syndrome, Autism, Rhett Syndrome, Childhood Disintegrative Disorder, and Pervasive Developmental Disorder-Not Otherwise Specified, as defined in the DSM-IV-TR and diagnosed by a physician, psychiatrist, or neurologist. Participants who are not employed at the time of the study intervention are the focus of this review. Authors will not differentiate between participants who have had an employment history from those who do not. Individuals who are employed prior to an intervention studied and continue to be employed during and after the study intervention will not be included in this review. Study participants with ASD and other disabilities will be included; however, study participants with solely mental retardation, schizophrenia, attention deficits or other non-autism related conditions will not be included. Persons with ASD and secondary conditions will be included. The treatment under investigation must be directed towards the provision of adult employment assistance intended to produce employment outcomes for individuals with ASD. The intervention may address social, behavioral, and/or cognitive dimensions related to the acquisition and maintenance of employment among the study participants. The intervention should be focused and involve relatively specific and structured experiences designed to support employment placement, for example, providing guidance in completion of applications, resumes, and engaging in interviews; shaping of work skills and appropriate employment setting social skills; teaching of work-related communication skills; or working directly with employers in the structuring of work and work setting features for individuals with ASD. The study must report subsequent attainment of an employment placement and provide specific data about the duration and/or retention of that placement. Gainful employment includes competitive, integrated, or supported employment; sheltered work or non-integrated work will not be considered as an outcome measure. Ten (10) or more hours of employment per week will be considered as gainful employment. Employment will be considered to encompass full or part-time (25% time or more) placements. Interventions of any length duration will be included provided adequate description is provided. Eligible studies may be either published or unpublished. Studies may be conducted in any country. We will not exclude studies reported in languages other than English, but we will not specifically search for non-English literature. Assistance in reading non-English studies will be obtained as needed. The search strategy for identification of relevant studies is highlighted below. Computerized database searches will be conducted. We will consult database thesauri, where they are available to assure that the universe of appropriate synonyms have been included in the intervention and outcome search term categories. Search terms and search strategies will be modified to fit individual databases. The keywords to be used in the computerized bibliography searches will be divided into three categories: population, treatment, and domain. The searches will cover the period from 1943 through 2008. All search terms will be truncated using the DIALOG Database conventions so that they will include variations in endings of words and in spelling. Terms from the three categories will be connected with “or” within each category and will be connected with “and” between categories. POPULATION TREATMENT DOMAIN Our efforts to find studies from outside the United States will include corresponding with researchers in countries known for their efforts in autism research (e.g. United Kingdom, Canada, and Australia). Correspondence will ask for their suggestions and help in locating eligible studies, particularly unpublished and fugitive literature. Recent reviews on autism research outside the U.S. will provide additional leads and addresses of contact persons. Grey literature that is identified through electronic searches will be submitted to the same inclusion criteria as other studies. A time range for these types of studies has not been specified in order to maximize consideration of all relevant grey literature. Reference lists from other systematic reviews and individual studies will be searched for potential studies to consider for inclusion. The references in relevant journal articles, systematic reviews, and other reports of research results will be scanned for new additions to our literature for review. Conferences that would be reviewed include, for example, Asia Pacific Autism Conference 2009 and the PENN Autism Network Conference. Studies will be coded for inclusion/exclusion decisions at two stages, Stage 1: citation and abstract and Stage 2: full-text. Two coders will serve as independent reviewers at both stages. A third party will be used in the case that the reviewers disagree and cannot resolve a final coding value. If the reviewer is ‘Unsure’ the citation/abstract will be advanced to the Full-Text for a final inclusion decision. At the Full-Text Stage 2 level, a full text of all citations advanced from Stage 1 will be obtained and coded for an inclusion/exclusion decision. The decision for advancing the retrieved full-text studies to an inclusion status will be made by two reviewers for each study, independently evaluating each study. An inclusion decision for advancement to the coding stage of the process will require that a study meets all the criteria presented earlier. Inter-rater reliability will be established prior to initiating coding activities, thus minimizing coding disagreements. Any disagreements in the determination of the presence of these criteria will be resolved by discussion of the two reviewers. In the event reviewer differences cannot be resolved via consensus building, a third reviewer will be asked to render a final decision. At the Full-Text Stage 2 level, two reviewers will record all excluded studies and the reasons for exclusion independently. Multiple studies that use the same sample or data will not be included. The most rigorous study focusing on our desired intervention outcome will be selected for inclusion. Design type These dimensions will be analyzed as moderator or sub-group analyses to assess their potential explanatory impact on the effect of the intervention. Other data that will be extracted from the primary studies include: publication source, subject characteristics, sample source, employment setting, intervention characteristics, type of employment, and outcome measurement (see Appendix B for Coding Form). The magnitude of the intervention effect will be calculated using the commonly accepted statistical formulae and dedicated programs available. While the outcome data may be available in different formats, we believe that most of the data will be presented in such a way as to require one of the following statistical approaches. Studies that report means and standard deviations for the experimental and control groups, the standardized mean difference statistic (d-index) will be used with the pooled standard deviation being the variance statistic for all calculations. For studies that report statistics such as t, F, or p-values and the accompanying sample sizes only, conversion formula will be used to calculate the d-index for the effect size estimate. In addition, the calculated d value will be corrected for sample size using Hedge's g, which accounts for studies with small sample sizes. All effect sizes will be calculated using a 95% confidence interval. Studies reporting binary data in which mean outcomes are compared in the experimental and control (or comparison groups) will be summarized using the odds ratio derivative statistic. A 95% confidence interval for the odds ratio, risk difference, or risk ratio statistics will be used to report all effect sizes. The most commonly used statistic for calculating an effect size using single subject design data is the PND. The PND provides for the calculation of the non-overlapping difference of measured data points when comparing baseline and subsequent treatment (or generalization) phases. The procedure involves identifying the highest baseline data point and calculating the proportion of data points in the intervention phase that are higher than the data points in the baseline. The PND metric is a simile percentage value that scales the magnitude of the intervention impact as follows: 90%+ = highly effective, 70% to 90% = moderately effective, 50% to 70% =questionable effectiveness, and less than 50% = ineffective (Scruggs, Mastropieri, Cook, & Escobar, 1986). Authors will report a PND calculation as not being possible if study data are inappropriate for such calculations. Only studies using a two group experimental or quasi-experimental design will be included in the data synthesis. Studies reporting either a pre-experimental single group pre- posttest or single subject experimental design will be analyzed separately and not combined with other design types. Studies that utilize PND and PZD will be included in the synthesis efforts. In anticipation that some studies will report multiple outcomes using the same experimental and control group comparisons, we will calculate an effect size for all comparisons individually. The synthesis of effect sizes within a study will be conducted by the aggregation of all effect sizes in order to produce an overall measure of treatment effect for that particular study. Because extracting multiple effect sizes from a single study may result in a violation of independence among the data, if a single study reports multiple outcomes, then the overall intervention effect size will be represented by the mean value of all study outcomes in a single study effect size. Individual multiple effect sizes from the same study will not be synthesized in order to avoid the violation of the principle of data independence in the analysis. The synthesis of effect sizes across conceptually similar constructs will be conducted in order to determine the magnitude of the effect when combining similar outcome effects from several studies. However, all cross study synthesis will utilize only one comparison effect size per study for any summary synthesis so that no single study outcome is represented more than one time in any analysis. For any group study (e.g., RCT, QED) any odds ratio effect sizes will be converted to a d-index effect size so that all effect sizes for all group studies will be presented in the same metric. For the single subject design studies, the PND will not be combined with the d-index metric but will be presented and analyzed separately. For the analysis of dichotomous and continuous data, an assessment of heterogeneity will be conducted. Both fixed and random effects models will be calculated and reported in order to assess the difference in the magnitude of the intervention effect under different sampling error assumptions. Interpretation of the results can then be gauged in the context of the underlying model assumptions sampling homogeneity. A sensitivity analysis will be conducted to assess the impact of a single study on the magnitude of an overall observed effect size. The sensitivity analysis will be conducted for overall study effect size and as appropriate, the impact of moderating variables (e.g., attrition, type of treatment, missing data, sample size, study design). Publication bias will be assessed for published vs. unpublished included studies through the visual inspection of a funnel plot and/or a moderator analysis. For studies reporting incomplete outcome data, we will first contact the senior author of the study and request the missing information to include in the analysis. If we are unable to obtain the needed information, we will attempt to use a method of imputation in which we will set the effect for the missing data to zero and then calculate the outcome effect size. Reported results will be presented both with and without the imputed data in order to assess the magnitude of the impact of the imputation process. Qualitative research will not be used in the analysis of the intervention research. However, we will collect and summarize relevant qualitative research such as survey, correlational studies, and case studies to provide a context for the interpretation of the quantitative results of any data synthesis or meta analysis. Once the protocol is approved, the authors anticipate submitting a first draft of a completed review to the Education Coordinating Group within six months. The authors will examine the review every three years for update. The concept for this systematic review was included in a proposal submitted to and funded by the National Institute on Disability and Rehabilitation Research in the Office of Special Education and Rehabilitative Services in the U.S. Department of Education. No conflicts of interest are known. Full Citation (APA style): Is this an Intervention Study? Were the Participants at least 18 Years Old? Were the Participants out of secondary school or not in a school-to-work transition program? I. Publication Source: II. Subject Characteristics (pg. ) Comments: III. Sample Source (pg. ) Comments: IV. SES (pg. ) Comments: V. Education (pg. ) Comments: VI. Study Community Setting (pg. ) Comments: Geographic Setting: VII. Employment Setting (pg. ) Comments: VIII. Participant Classification (pg. ) Comments: IX. Classification Severity Level (pg. ) Comments: X. Race/Ethnicity (pg. ) Comments: Intervention Characteristics (pg. ) XI. Describe Intervention Characteristics (pg. ) Details of Intervention intended for treatment/comparison groups including how and when administered. Average Length of Intervention Program (pg. ): Length of time of participation activity (pg. ): per Number of Sessions (pg. ): Primary Type of Employment: Wholesale Trade Goods-processing Industries Public Administration Comments: XII. Outcome Measure(s): Comments: Design Characteristics (pg) XIII. Research Design Characteristics: Which of the following research design types were used to examine the impact of program effects for employment placement? Comments: XIV. Method of Random Assignment (pg ) Comments: XV. Recruitment Pool (pg ) Comments: XVI. Blinding Comments: XVII. Fidelity of Implementation Comments: XVIII Effect Size Characteristics (Use d-Index Value if Provided) Outcomes Method of Analysis (pg ) Comments: