Hallucination or vision? Establishing reliability and validity of AI-based qualitative analysis of the unified model of activism in the Scottish independence Twitter debate

Abstract

PurposeThe present study explores both the validation of the unified model of activism and the methodological reliability of the LlamaParsing approach to natural language processing. Theoretically, it applies the unified model of activism within the context of the Scottish independence movement, evaluating its effectiveness in social media environment.Design/methodology/approachMethodologically, it addresses the reliability and validity challenges associated with AI analyses, particularly the issue of AI hallucinations—instances where AI generates seemingly accurate but incorrect information. By employing the LlamaParsing approach and then comparing and contrasting it with a quantitative content coding process, the study demonstrates how context-specific instructions can enhance the accuracy of AI analyses.FindingsThe findings indicate this approach not only tests and extends the unified model of activism but also offers a robust methodological framework for using NLP and RAG in qualitative research. This dual focus underscores the potential of AI to provide systematic and theoretically valuable insights while highlighting the importance of mitigating its limitations.Originality/valueThis study represents a cutting-edge approach to qualitative data analysis, theory development, and theory testing in communication using a tool that was developed in 2024.