Exploring Generative AI as a Proxy User for Early Stage User Research - Preliminary Findings

Abstract

The potential of generative AI has exploded of late, largely due in part because of the improved accessibility that tools like ChatGPT afford for non-data-scientist / developer users. One potential area of application is for Generative AI models to serve as proxy users in early stage user research. User research is a crucial component of product development, helping to understand user needs, preferences, and behaviors. However, conducting user research can be time-consuming, resource-intensive, and may require access to a user population that is challenging to access (e.g., military users). Generative AI models have shown remarkable progress in generating human-like text and simulating user interactions based on a significant corpus of training materials that serves as the knowledge base for the AI’s reasoning. This paper provides preliminary findings from explorations on the feasibility of leveraging generative AI as a proxy user to inform early stage user research. Using the GPT-4.0 architecture and the Open-AI ChatGPT user interface (chat.openai.com), we conducted preliminary research for six different candidate end user populations. This was accomplished by generating generic product descriptions, notional user personas each respective product, contextualizing ChatGPT to act as the user persona, and then asking a series of generic user experience research (UXR) questions of the GPT model. Responses from ChatGPT were then scored by three UXR / Human Factors subject-matter experts to evaluate the perceived utility of ChatGPT’s responses in terms of supporting early stage product design as a proxy human user. By evaluating the effectiveness of generative AI as a proxy user, this research aims to shed light on its potential benefits and limitations in supporting early stage user research efforts. While additional research is still needed (e.g., comparing the results of ChatGPT to responses generated by actual end users, having SME’s evaluate the accuracy and completeness of ChatGPT’s responses), preliminary findings are promising for the potential that generative AI models hold to serve as early stage proxy users to inform research and product design efforts in domains where significant corpuses of data already exist for model training, and where access to human end users may be restricted our otherwise prohibited.