Aligning sampling and case selection in quantitative-qualitative research designs: Establishing generalizability limits in mixed-method studies

Anjuli Verma,Black Hawk Hancock,Bryan L Sykes

doi:10.1177/1466138117725341

Anjuli Verma, Black Hawk Hancock + Show 1 more

Open Access

https://doi.org/10.1177/1466138117725341

Copy DOI

Abstract

Quantitative researchers increasingly draw on ethnographic research that may not be generalizable to inform and interpret results from statistical analyses; at the same time, while generalizability is not always an ethnographic research goal, the integration of quantitative data by ethnographic researchers to buttress findings on processes and mechanisms has also become common. Despite the burgeoning use of dual designs in research, there has been little empirical assessment of whether the themes, narratives, and ideal types derived from qualitative fieldwork are broadly generalizable in a manner consistent with estimates obtained from quantitative analyses. We draw on simulated and real-world data to assess the bias associated with failing to align samples across qualitative and quantitative methodologies. Our findings demonstrate that significant bias exists in mixed-methods studies when sampling is incongruent within research designs. We propose three solutions to limit bias in mixed-methods research.

Highlights

Simmering debates among ethnographers about representation, reproducibility, generalizability and the role of theory have come to a boil over the past decade
Despite the burgeoning use of dual research designs, there has been little empirical assessment of whether the themes, narratives, and ideal types derived from qualitative fieldwork are broadly generalizable in a manner consistent with estimates obtained from quantitative analyses based on sample surveys
Against the backdrop of recent controversies concerning race, law, and social inequality research that blends quantitative and qualitative methods (e.g. Goffman, 2014; Klinenberg, 2002), we offer a systematic examination of the conditions under which qualitative-quantitative research designs undermine scientific definitions of validity

Summary

Introduction

Simmering debates among ethnographers about representation, reproducibility, generalizability and the role of theory have come to a boil over the past decade. We retain only race and parental incarceration status from these data to draw attention to how sampling in non-probabilistic fieldwork and interviews has significant import for evaluating mechanisms and processes in MMR – for both quantitative and qualitative researchers.

Results

Conclusion