Reconciling Data from Different Sources: Practical Realities of Using Mixed Methods to Identify Effective High School Practices

Abstract

Background/Context Mixed methods research conveys multiple advantages to the study of complex phenomena and large organizations or systems. The benefits are derived from drawing on the strengths of qualitative methods to answer questions about how and why a phenomenon occurs and those of quantitative methods to examine how often a phenomenon occurs and establish generalizable, empirical associations between variables and outcomes. Though the literature offers many strategies, designing mixed methods research can be challenging in large scale projects when trying to balance reliability, validity, and generalizability. By supporting the findings with multiple forms of evidence mixed methods designs lend greater validity than mono-method ones. However to draw on the comparative advantages of these two paradigms, researchers must grapple with the challenges of working with more than one method. Focus of Study This paper discusses the benefits and challenges of collecting and interpreting mixed methods data in a large scale research and development project. Drawing on existing frameworks, we refect on our strategies of mixed methods design, data collection, and analysis. We discuss the quandaries faced by researchers when discrepant findings emerge. Research Design The data come from a large, mixed methods case study focused on the practices that explain why some high schools in large urban districts are particularly effective at serving low income students, minority students, and English language learners. Undertaken in several phases, the work included sequential and concurrent designs. Incorporating a sequential explanatory design element, we first used quantitative data to identify schools in the district that were more and less effective at improving student achievement in English/language arts, mathematics, and science. We then used a combination of interviews, focus groups, surveys, classroom observations, and district administrative data—in a concurrent design—to try to understand what differentiated between the most and least effective schools in the district. Conclusions Based on our analyses, we provide examples of when mixed methods data converge, when they diverge but are complementary, and when they diverge and introduce a methodological quandary for researchers who must confront seemingly discrepant findings. In so doing, we discuss the tradeoffs encountered between the study design and the implications as we confronted them during analysis and suggest ways to balance the methodological demands of complex research studies. Seemingly discrepant findings, while challenging to reconcile, when considered for their potential complementarity, actually lead to a more complete understanding of the phenomena under study.