Alternating treatments designs: Interpretation challenges and design solutions for validating and comparing interventions

Abstract

AbstractThe purpose of this paper is to demonstrate and describe why alternating treatments design (ATD) researchers should consider employing a no‐treatment series during the alternating treatments phase. To identify limitations associated with excluding this design element, we obtained examples of ATD studies published in five school psychology journals published over 10 years. Next we read, scored, and analyzed these articles with our primary focus being authors' isolated and comparative effectiveness conclusions. After taking into account authors' relative effectiveness conclusions, we describe why some isolated treatment effectiveness conclusions may not be warranted because threats to internal validity were not controlled. Also, we describe how including a no‐treatment series during the alternating treatments phase can address these limitations by allowing researchers to assess and possibly rule out many threats to internal validity. Discussion focuses on how including this design element may increase the number of studies validating specific interventions, yield better relative effectiveness conclusions, and prevent researchers and consumers of research from concluding that an intervention was effective when observed changes were caused by uncontrolled threats to internal validity.