Abstract
This mixed methods approach study investigates the potential of introducing Generative AI (ChatGPT 4 and Bard) as part of a deductive qualitative research design that requires coding, focusing on possible gains in cost-effectiveness, coding throughput time and inter-coder reliability (Cohen’s Kappa). This study involved semi-structured interviews with five domain experts and analyzed a dataset of 122 respondents that required categorization into six pre-defined categories. The results from using Generative AI coders were compared with those from a previous study where human coders carried out the same task. In this comparison, we evaluated the performance of AI-based coders against two groups of human coders, comprising three experts and three non-experts. Our findings support the replacement of human coders with Generative AI ones, specifically ChatGPT for deductive qualitative research methods of limited scope. The experimental group, consisting of three independent Generative AI coders, outperformed both control groups in coding effort, with a fourfold (4x) efficiency and throughput time (15x) advantage. The latter could be explained by leveraging parallel processing. Concerning expert vs. non-expert coders, minimal evidence suggests a preference for experts. Although experts code slightly faster (17%), their inter-coder reliability showed no substantial advantage. A hybrid approach, combing ChatGPT and domain experts shows the most promise. This approach reduces costs, shortens project timelines, and enhances inter-coder reliability, as indicated by higher Cohen's Kappa values. In conclusion, Generative AI, exemplified by ChatGPT, offers a viable alternative to human coders, in combination with human research involvement, delivering cost savings and faster research completion without sacrificing notably reliability. These insights, while limited in scope, show potential for further studies with lager datasets, more inductive qualitative research designs and other research domains.
Published Version
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call