“Cherry Picking”: Writing From Thin Data

Abstract

“Cherry Picking” is a style of data analysis used when a researcher has inadequate data. Basically, rather than working with large categories, the researcher has terminated data collection with a minimal data set, yet forges ahead nonetheless, completing the analysis. When writing the results, the theoretical structure of the analysis is presented, with commentary, and the best (and sometimes the only) quotations available to illustrate the commentary. I call the analysis for this “documentary-style,” and this is where we see cherry picking used as a deliberate strategy. In a documentary, the commentator says something such as, “His death was devastating news for his daughter,” and the daughter then appears on screen, and says sorrowfully, “I was simply devastated.” So it is with cherry picking: Data that support the commentary are deliberately selected to endorse that same commentary. No other data are presented. Although variation is very limited, the article appears neat and convincing. What is so dangerous about cherry picking? First, the researcher may simply be wrong. Further data would illuminate missing idiosyncrasies, reveal hidden complexities, and even redirect the analysis. Or, if the researcher was right the first time, additional data that may confirm that whatever has been presented is uncommon. If these cherry picking researchers are smart, they will realize that they have no confidence in the validity of the analysis. At best the findings are tentative, and certainly not ready for presentation or publication. How do we recognize a case of cherry picking? First of all, the research has few participants who were each interviewed once. In fact, this is the main problem with smallsample research, tightly scoped research, or research that uses a restrictive, semistructured questionnaire; only a small amount of data obtained from a few participants are available for analysis. When talking about these data, the researcher may use the same example over and over, for if the study was saturated, the researcher could provide numerous examples of whatever is being discussed, and speak about the groups in generalities (“These people do this and that, and others do thus and so”). If those who are cherry picking use such generalities, they are getting onto thin ice, overgeneralizing, and usually recognize this themselves. Another problem is missing perspectives. We become suspicious when a study appears too neat and tidy, or when we think, “How come no one talked about thus and so?” Missing perspectives sometimes result in missing categories, components, or important meanings—in other words, this may be enough of a problem to invalidate the study. Sometimes this situation can be forced on a study, from the researchers defining (or scoping) the study population or boundaries too narrowly or too tightly. Using such rigid criteria for participation is a carryover from quantitative inquiry, and we must always be able to justify why or where we have drawn the boundaries that include and exclude. Less commonly, cherry picking may result from including too much variation in the study, and while there are plenty of participants in the study, the researcher has, for instance, allowed many ethnic groups into the study— and even compared their responses—but has not internally saturated each group. And the worst case (dare I write this?) is the lazy researcher, who has not analyzed but just selected the most interesting, the best articulated, or even the most peculiar quotations, has skipped the process of categorization, and ignored what “most” have said. Perhaps this researcher has not bothered to transcribe, and is working from memory, using recollections of the interviews or from listening to the tapes. All of these are threats to validity. All contribute to poor qualitative inquiry. This is a plea to increase sample size to increase certainty, and when writing, to produce rich description and use the best representative quotations.