The State of the Evidence: ASHA Develops Levels of Evidence for Communication Sciences and Disorders

Abstract

You have accessThe ASHA LeaderFeature1 Mar 2007The State of the Evidence: ASHA Develops Levels of Evidence for Communication Sciences and Disorders Rob Mullen Rob Mullen Google Scholar More articles by this author https://doi.org/10.1044/leader.FTR4.12032007.8 SectionsAbout ToolsAdd to favorites ShareFacebookTwitterLinked In For the first time, clinicians and researchers in speech-language pathology and audiology will have a common framework for characterizing the scientific evidence underlying clinical practice. This framework will offer a look at the state of the scientific evidence in different areas of communication sciences and disorders—and demonstrate areas in which more research is needed—in an era when practicing clinicians, researchers, government agencies, and third-party payers are evaluating the evidence to improve the quality and effectiveness of service delivery. “SLPs regularly make decisions about treatment plans while under constraints of government, education, and financial agencies. Information from this LOE system, which is a trusted source for evidence, will aid clinicians making those decisions,” said Janet Patterson, a member content expert who appraised research studies using ASHA’s LOE system in an evidence-based systematic review. “The entire discipline will be able to speak with one voice about the extent of scientific evidence underpinning clinical practice.” To describe the evidence in communication sciences and disorders, ASHA’s Advisory Committee on Evidence-Based Practice (ACEBP) and National Center for Evidence-Based Practice in Communication Disorders (N-CEP) developed a system of levels of evidence (LOE) specific to communication sciences and disorders. LOE is an approach to evaluating evidence based on its quality, using criteria such as protection from bias and confounding, effect size, and precision (visit ASHA’s evidence-based practice section for a glossary of terminology). LOE is useful as a way to communicate and compare the relative strength or weakness of scientific support for a specific intervention or diagnostic technique. Speech-language pathologists and audiologists often look to levels of evidence as a factor (along with clinical expertise and client values) in making clinical decisions. The LOE system also builds member awareness of the principles involved in evaluating research and provides practicing clinicians with a means for evaluating relevant research when making clinical decisions. Externally, Medicare and third-party payers are increasingly demanding information about the level of evidence to make reimbursement decisions. Insurance companies have characterized the evidence by using various LOE systems and have developed reviews of evidence to justify reimbursement decisions. In addition, government agencies seek input on research needs in setting the agenda for the direction of research and establishing funding priorities. Levels of evidence also can be used to identify gaps in the research for prioritization of future research needs. In Search of a System Identification of a system of LOE for use throughout ASHA was part of the initial charge to ACEBP when it was established in 2005. The original intent was to identify the existing LOE that best fit the particular needs of speech-language pathology and audiology. But no best fit could be found because of two factors that complicated the search. One issue was study quality. ACEBP and N-CEP felt very strongly that poorly done and well-done studies that used similar study designs should not be given equal weight. While this caveat seems obvious, many LOE systems were eliminated because they did not take study quality into consideration. Still others were eliminated because they did not define high and low quality. While these systems would give more weight, for example, to a high-quality cohort study than to a low-quality cohort study, no objective criteria were provided to determine high versus low quality. The second complicating factor was study design. Most LOE systems were developed by and/or for physicians and were based on a medical model. As a result, very few of these systems include single-subject design studies as even a potential source of scientific evidence. This design, however, constitutes a very important part of the research in communication sciences and disorders, and often offers the best ways to address some of the research questions of interest to ASHA members. Because of their focus on an individual subject and in-depth examination of behavior, this type of experimental design may be suited to developing and testing specific treatments. This design is especially valuable in relatively low-prevalence areas such as augmentative and alternative communication or fluency. Only a small number of LOE systems took sufficient account of study quality, and even fewer included single-subject designs. As no single system met both criteria, ACEBP and N-CEP undertook development of an LOE system specific to speech-language pathology and audiology in 2005. ASHA’s Levels of Evidence ASHA’s LOE system is a four-step process that involves assessing the extent to which a study meets quality indicators, deciding the stage of research to which a study applies, “scoring” the quality of the study in the context of the stage of research, and finally, synthesizing this information for all studies to give an overall picture of the level of evidence available on a particular clinical issue. Step 1: Appraise the quality of individual studies related to the topic under review. Eight factors are considered in the appraisal process (see Figure 1). Step 2: Characterize the research stage into which a particular study falls. In communication sciences and disorders, a single-subject design study could, in some circumstances, constitute the best evidence on a particular question. The ASHA LOE is unique in that strong consideration is given to the appropriateness of a study design in the particular context of the clinical question, rather than placing it on a rigid hierarchy of study designs regardless of context. Drawing heavily from Randall Robey’s 2004 article on stages of clinical research, a set of research stages was adopted to provide that context. See Figure 2 [PDF] for the ASHA LOE four stages of research. Step 3: Assess the quality of a study in the context of the research stage to which it applies. For example, the exploratory stage of research includes studies that lead to an eventual decision on the appropriateness of conducting a very expensive and time-consuming randomized controlled trial to test a hypothesis. It would not be appropriate to assess these studies on a criterion such as “intention to treat,” which is a criterion specifically related to randomized The stage of research, once determined, in turn determines the highest possible level of quality for each Step 1 quality criterion for the study designs that fall into that research stage. These levels constitute the standard against which the individual studies are judged. Studies that meet the highest relevant level within each of the seven quality criteria are rated as highest quality. There are seven, rather than eight, quality criteria because “intention to treat” is used only when the study design is a controlled trial, and study design is used as a factor only when the study is not a controlled trial. Studies scoring a five or six are rated as good quality; studies scoring below five are rated as poor quality. Step 4: Synthesize this information into a single table as a snapshot to illustrate the state of the evidence. Combining the information as in the hypothetical table (Figure 3) allows the body of evidence in a given area to be described in terms of research stage and quality. The hypothetical table shows that a number of high-quality efficacy studies have been conducted. One could conclude that research priorities should include an emphasis on well-designed effectiveness research, as there might be little to be gained from additional exploratory or efficacy research related to the same question. Refining the System The LOE system in its current form has been piloted in an evidence-based systematic review on the frequency and intensity of treatment of stroke patients with aphasia. This topic was selected because of the many questions about the most effective way to deliver aphasia treatment, and the lack of a high-quality synthesis of the evidence related to frequency and intensity. ASHA began the review by identifying thousands of abstracts for aphasia research. The ASHA N-CEP staff reviewed the abstracts, narrowing them down to fewer than two dozen pertinent articles, and then obtained the full-text journal articles. These articles were then reviewed using the LOE system by member volunteers Janet Patterson, Stacie Raymer, and Leora Cherney of Division 2, Neurophysiology and Neurogenic Speech and Language Disorders. As the system is applied to a greater number of topics and used by more individuals and groups, refinements will be made to make it as relevant and useful as possible. ACEBP and N-CEP hope that this tool can and will be improved with additional input from ASHA members. All members are invited and encouraged to visit the Web discussion forum on evidence-based practice; a thread has been started on which members may voice opinions and/or post questions about the system. Figure 3: The State of the Evidence on a Fictitious Topic The hypothetical LOE table summarizes the quality of evidence and the types of research questions that have been addressed. In a systematic review, a table such as this will include every individual study contained in that review. This particular example indicates that a number of high-quality efficacy studies have been conducted. One might then conclude that research priorities should include an emphasis on well-designed effectiveness research, as there might be little to be gained from additional exploratory or efficacy research related to the same question. Exploratory, Efficacy, Effectiveness, Policy High Quality, Washington 1994, Eisenhower 1998, Grant 1999, Truman 1999 Good Quality, Lincoln 1992, Jefferson 1997, Nixon 2000, Kennedy 1999 Lower Quality, Roosevelt 1994, Adams 2001 Author Notes Rob Mullen, is director of ASHA’s National Center for Evidence-Based Practice in Communication Disorders. Contact him by e-mail at [email protected]. Advertising Disclaimer | Advertise With Us Advertising Disclaimer | Advertise With Us Additional Resources FiguresSourcesRelatedDetailsCited ByAmerican Journal of Audiology21:2 (295-312)1 Dec 2012An Evidence-Based Systematic Review of Directional Microphones and Digital Noise Reduction Hearing Aids in School-Age Children With Hearing LossRyan W. McCreery, Rebecca A. Venediktov, Jaumeiko J. Coleman and Hillary M. LeechLanguage, Speech, and Hearing Services in Schools42:3 (246-264)1 Jul 2011Auditory Processing Disorder and Auditory/Language Interventions: An Evidence-Based Systematic ReviewMarc E. Fey, Gail J. Richard, Donna Geffner, Alan G. Kamhi, Larry Medwetsky, Diane Paul, Deborah Ross-Swain, Geraldine P. Wallach, Tobi Frymark and Tracy SchoolingAmerican Journal of Audiology20:2 (159-170)1 Dec 2011Audiologic Management of Auditory Neuropathy Spectrum Disorder in Children: A Systematic Review of the LiteraturePatricia Roush, Tobi Frymark, Rebecca Venediktov and Beverly WangAmerican Journal of Speech-Language Pathology19:4 (321-340)1 Nov 2010Evidence-Based Systematic Review: Effects of Oral Motor Interventions on Feeding and Swallowing in Preterm InfantsJoan Arvedson, Heather Clark, Cathy Lazarus, Tracy Schooling and Tobi FrymarkAmerican Journal of Speech-Language Pathology18:4 (361-375)1 Nov 2009Evidence-Based Systematic Review: Effects of Neuromuscular Electrical Stimulation on Swallowing and Neural ActivationHeather Clark, Cathy Lazarus, Joan Arvedson, Tracy Schooling and Tobi FrymarkAmerican Journal of Speech-Language Pathology18:2 (133-145)1 May 2009When “Simon Says” Doesn’t Work: Alternatives to Imitation for Facilitating Early Speech DevelopmentLaura S. DeThorne, Cynthia J. Johnson, Louise Walder and Jamie Mahurin-Smith Volume 12Issue 3March 2007 Get Permissions Add to your Mendeley library History Published in print: Mar 1, 2007 Metrics Downloaded 2,810 times Topicsasha-topicsleader_do_tagleader-topicsasha-article-typesCopyright & Permissions© 2007 American Speech-Language-Hearing AssociationLoading ...