ChatGPT and finetuned BERT: A comparative study for developing intelligent design support systems

Abstract

Large Language Models (LLMs), like ChatGPT, have sparked considerable interest among researchers across diverse disciplines owing to their remarkable text processing and generation capabilities. While ChatGPT is typically employed for tasks involving general knowledge, researchers increasingly explore the potential of this LLM-based tool in specific domains to enhance productivity. This study aims to compare the performance of a finetuned BERT model with that of ChatGPT on a domain-specific dataset in the context of developing an intelligent design support system. Through experiments conducted on classification and generation tasks, the knowledge transfer and elicitation abilities of ChatGPT are examined and contrasted with those of the finetuned BERT model. The findings indicate that ChatGPT exhibits comparable performance to the finetuned BERT model in sentence-level classification tasks but struggles with short sequences. However, ChatGPT's classification performance significantly improves when a few-shot setting is applied. Moreover, it can filter out unrelated data and enhance dataset quality by assimilating the underlying domain knowledge. Regarding content generation, ChatGPT with a zero-shot setting produces informative and readable output for domain-specific questions, albeit with an excessive amount of unrelated information, which can burden readers. In conclusion, ChatGPT demonstrates a promising potential for application in facilitating data labeling, knowledge transfer, and knowledge elicitation tasks. With minimal guidance, ChatGPT can substantially enhance the efficiency of domain experts in accomplishing their objectives. The findings suggest a nuanced integration of artificial intelligence (AI) with human expertise, bridging the gap from mere classification models to sophisticated human-analogous text generation systems. This signals a future in AI-augmented engineering design where the robust capabilities of AI technologies integrate with human creativity and innovation, creating a dynamic interactions to redefine how we tackle design challenges.