The TrollLabs open hackathon dataset: Generative AI and large language models for prototyping in engineering design

Abstract

The TrollLabs Open dataset includes comprehensive information that offers a comparison of design practices and outcomes between human participants and Generative AI during a hackathon event. The dataset was curated through the running of a prototyping hackathon designed to assess the abilities and performance of generative AI, specifically ChatGPT, in the early stages of engineering design. This assessment involved comparing ChatGPT's performance to that of experienced engineering students in a hackathon setting, where participants competed by making a prototype that fires a NERF dart as far as possible. In this setup, all ideas, concepts, strategies, and actions undertaken by the AI-controlled team were autonomously generated by the ChatGPT, without human intervention or guidance, but implemented by two participants. Five self-directed baseline teams competed against the AI team. The dataset comprises 116 prototype entries and 433 edges (connection) that enable comparative analysis of design practices and performance between the team instructed solely by generative AI and baseline teams of experienced engineering design students. Prototypes and their attribute data were captured using Pro2booth, an online prototype capture platform running on participants' phones and computers. The dataset includes a transcript of the conversation between ChatGPT and the team responsible for implementing its recommendations, featuring 97 exchanges of prompts and responses. It contains the initial prompt used to instruct the AI, the objective and rules of the hackathon and the objective performance of teams, showing the ChatGPT team finishing 2nd among six teams. To the authors' knowledge, the TrollLabs Open dataset is the first and only open resource that directly compares the performance of generative AI with human teams in an engineering design context. Thus, it is intended to be a valuable resource to design researchers, engineering and design students, educators, and industry professionals seeking to find strategies for implementing generative AI tools in their design processes. By offering a comprehensive data collection, the dataset enables external researchers to conduct in-depth analyses that could highlight the practical implications of integrating generative AI in design practices, possibly providing an overview of its limitations and presenting recommendations for improved integration in the design process.