MORE THAN FIGURES? QUALITATIVE RESEARCH IN HEALTH ECONOMICS

Abstract

When attending last year's (2011) International Health Economics Association (iHEA) conference in Toronto, we were struck by the notion that almost all papers we listened to used a very similar quantitative methodology, whereas qualitative methods barely seemed to play a role. This is unfortunate: (Health) economists could effectively integrate combinations of qualitative and quantitative methods into their research toolkit, without having to give up the formal modeling approach they are accustomed to. The population and institutions studied will rarely be identical with the population for which policy recommendations are derived, so that, to the extent that the two differ, the recommendations may only partially hold. However, acquiring knowledge about the compatibility of populations and institutions is a complex task, one that may require the type of data generated by qualitative research. There is a limited debate on the standing of qualitative methods in economics and health economics in particular. A general presumption in this debate is that the vast majority of economists share a certain conception of the process of scientific inquiry, that is, the use of increasingly sophisticated mathematics to derive formal statements in mathematical language. Qualitative research, by contrast, is mostly ignored in this framework. By qualitative research, we here mean techniques, which describe ‘in words rather than numbers the qualities of social phenonmena through observation…, unstructured interviews […], diary methods, life histories […] group interviews and focus group techniques, analysis of […] records, documents and cultural products’ (Bowling, 2009, p. 380). Some of mainstream economics' fiercest critics couple their call for qualitative research with an attack on the very foundations of scientific inquiry in economics (Lincoln, 1992; Small and Mannion, 2005). Such criticism is unlikely to resonate with a large number of economists because following the recommendations would entail a complete departure from established paradigms. Moreover, it is also unclear what remains that is specific to economics: economists may no longer have a ‘comparative advantage’ over other social scientists in researching any social phenomena (Coast et al., 2004). An alternative would be to try to accommodate qualitative methods within the currently dominant paradigm. Coast (1999) suggests three uses: First, qualitative studies could be useful for explorative studies before an economic model has even been specified, providing an empirical grounding for the specification of any assumptions. Second, qualitative studies could be used as a precursor to quantitative analysis as a check on the internal validity of the study design. Third, qualitative studies could be conducted ex post to help with the interpretation of confusing or seemingly contradictory evidence in the quantitative analysis. Another potential application of qualitative methods presents itself in the burgeoning field of discrete choice experiments to elicit preferences over health states and treatment options from respondents. Here, qualitative methods could help design the attributes over which respondents choose to ensure that attributes reflect all concept of relevance to respondents and are not misleadingly worded (Coast et al., 2012). Baker et al. (2006) suggest that mixed methods may be suitable to replace the standard quantitative analysis of discrete choice experiments altogether. Finally, Smith et al. (2009) argue that health economists should employ qualitative research when translating economic analysis into organizational practice. As far as we are concerned, these suggestions have not (yet) taken up by a large number of researchers. To our knowledge, no study has empirically investigated the use of qualitative methods, whether in health economics or any other economics subdiscipline. Weiner et al. (2011) reviewed all articles published in nine health services and management journals over 10 years and found qualitative methods in only 9% of articles. We analyzed all papers with abstracts presented at the 8th World Congress on Health Economics, held in Toronto, Canada, from 10–13 July 2011. Our final sample size was 1232. The iHEA spans the entire spectrum of health economics today, both in terms of subspecializations within the health economics field and in terms of geographic and cultural diversity of its membership. We believe that the research presented at this conference provides a valuable sample that is approximately representative of the current research activity in health economics. More than half of the research presented in Toronto came from scholars outside North America and one in six from outside the western world. This is remarkable not only because the conference was held in Canada but also because North American researchers have traditionally dominated the health economics literature, at least those published in the English language (Wagstaff and Culyer, 2011). Two reviewers independently read the papers' abstracts and classified them as theoretical, empirical, literature reviews, or discussion papers. We further subclassified theoretical and empirical papers as qualitative, quantitative, or using mixed methods. There was a very high degree of inter-rater reliability: the reviewers' assessments were identical for 1175 papers, or over 95% of cases. Cohen's Kappa, a measure of inter-rater reliability, was 0.883, which is considered high (Landis and Koch, 1977). For the remaining papers, different categorizations could be resolved in discussion between the two reviewers. Table 1 presents the results of this exercise. Just 78 empirical and seven theoretical papers consider any type of qualitative methods – a mere 6.9% of our total sample or 7.5% if one excludes discussion papers and literature reviews. The small number of papers using partly or fully qualitative methods allowed us to analyze each of these papers' abstracts in greater depth to infer what we perceived to be the authors' motivation for the use of qualitative methods. Again, two reviewers independently read the abstracts and tried to gather the reasons the authors opted for qualitative methods. Once the reviewers had independently classified all papers, they met to discuss all disparities between their groupings. The inter-rater reliability was lower at 64% (Cohen's kappa: 0.495) because the analysis involved some degree of speculation, as the authors' motivations were not always fully apparent from the abstracts alone. Table 2 presents the results with 39 of the papers using either unorthodox economic frameworks outside the economic mainstream, or were conducted by non-economists who presented on health-related topics. Most interestingly, the single largest category of economics papers using qualitative methods (29 papers) sought to evaluate a program and supplemented their quantitative analysis with qualitative methods to identify not just the size of any effects but also the specific channels through which these effects came into play. Moreover, such papers were often interested in the institutional and cultural context in which such policies were implemented. All but two of these studies were conducted in developing countries, which stands in stark contrast to papers of other types. Policy evaluations are more useful if they provide information not only on the size of effects but also on the mechanisms that led to such effects and the context in which these effects took place. This argument would have special force if the lessons from one policy experiment were to be applied in a vast range of institutional and cultural contexts. One such example is the evaluation of a policy from one developing country to be applied in other developing countries, which could be problematic given the great diversity of the many different countries usually lumped together as ‘developing countries’. Here, local institutional knowledge and knowledge about the interplay of institutions, the regulatory framework, and the soon-to-be-introduced policy become critical. The issue of transferring evidence from one setting to another and the importance of evidence on mechanisms, not just outcomes, has been recognized and hotly debated in the development economics community (see e.g., Deaton, 2010). In this context, combining qualitative and quantitative methods has been suggested as one promising remedy (Bamberger et al., 2010). To illustrate, consider the work by El-Khoury et al. (2011), which uses both qualitative and quantitative methods to research the effect of removing user fees for cesarean sections in Mali. The paper thus concludes that the policy has been quite successful overall, but that barriers remain, particularly pertaining to drug costs, transportation, and facility equipment. Such a nuanced and policy-relevant argument would have been difficult to make by looking at the quantitative data alone. But does the argument for intimate knowledge of processes and context, and not just outcomes, apply to health care economics field broadly and not only in developing countries? Health economists frequently argue that the well-known difficulties of uncertainty and incomplete information that characterize the health care sector have given rise to the development of a highly complex and diverse set of institutions that differ markedly across time and space. In general, researchers will wish to generalize their findings and to draw lessons for future policies indicative of the effects we can expect for other populations in other institutional settings. Such arguments derive their strength from knowledge of the context and processes, which generated the observed effects, and it is here that qualitative research may plausibly help in generating this knowledge. We would like to point out that we do not seek to propose some form of radical change or entirely different view on the subject, but we strongly feel that at present, in health economics, far too little qualitative work is performed, and thus major sources of important information go untapped. We suggest enrichment of research methodologies. The authors report no conflicts of interest. The authors thank Nasir Umar, Lara Shore-Sheppard and seminar participants at the Mannheim Institute for Public Health for fruitful discussions and Julia Gedeon for excellent research assistance. All remaining errors and inaccuracies are our own. The authors did not receive any specific funding for this research.