Can AI Get a Degree in Geoscience? Performance Analysis of a GPT-Based Artificial Intelligence System Trained for Earth Science (GeologyOracle)

Abstract

Generative artificial intelligence systems based on the GPT model have shown groundbreaking capabilities in producing scientific texts, revolutionising how geoscientists research and teach. However, there is a noticeable absence of generative AI systems tailored specifically for geoscience, and the efficacy of GPT technology has not been examined within the Earth Science realm. To fill these gaps, we propose a new artificial intelligence system (GeologyOracle) built upon the GPT-4o model and trained on Earth Science data. It is designed to simulate a conversation with a geoscientist, having the capabilities to analyse geologic datasets, suggest new geoscience hypotheses, explain Earth-Science concepts, and interpret geosites. To evaluate the performance of GeologyOracle, the system was tested with 152 geoscience questions provided and evaluated by a panel of geoscience academics on a scale of 0–10. Performance analysis revealed that 79.6% of the answers scored equal to or above the passing mark of 5, with a tendency to deliver high-quality responses (mean: 6.5; median = 7; interquartile range: 5–8). The results demonstrate that GeologyOracle is effective in performing complex geoscience tasks, such as identifying rocks, fossils, and minerals, and interpreting outcrop, core data, and quantitative datasets. The new AI system has exhibited noteworthy ability in the interpretation of internationally-acknowledged geosites situated within geoparks and nature reserves. The performance of the AI system is comparable to that of trained geoscience specialists, suggesting its potential as an assistant in various geoscience fields, including structural geology, palaeontology, geomorphology, sedimentology, and economic geology. While AI does not render geoscientists obsolete, it offers significant potential for accelerating scientific discovery, automating geoscience research, and assisting educators, students, and geotourists.