Background and aimsThe assessment of adverse events from endoscopic procedures is essential for successful interventions, ensuring accurate follow-up, adverse event management and processing for quality control. Despite the critical need for structured documentation, the current practice often relies on free-text recordings, which poses challenges for scalable intervention analysis; however, the introduction of Large Language Models (LLMs) offers a promising solution by enabling the automatic extraction of adverse event details from procedural reports, without altering existing documentation practices. MethodsWe analyzed 672 endoscopy reports, using OpenAI’s GPT-4 and Llama-2 based models to structure the data in JSON for efficient analysis. We used an automated LLM-pipeline to extract adverse events such as bleeding, perforation, and aspiration. The dataset was divided into a proof of concept set (PoC-S) with n=171 reports, on which we explored prompt engineering to improve the performance of the models. The final analysis was run on an additional external test set of 501 reports (TS). ResultsGPT-4 showed high accuracy, with a sensitivity of 97% and specificity of 92% in the PoC-S and 91% and 96% respectively in the TS. GPT-4 use in real-world settings is limited by privacy concerns. Conversely, Llama-2 based models, especially the Llama-2 variants fine-tuned for German language, demonstrated comparable performance (PoC-S: sensitivity 94%, specificity 92%, CS: sensitivity 89%, specificity 93%) and offered a viable, privacy-compliant alternative. These models' effectiveness was further influenced by the method of prompt engineering, with experiments showing that the specificity and sensitivity could vary substantially based on the inclusion of specific prompt features, underscoring the importance of tailored prompt design. ConclusionApplying LLMs to extract structured medical information, particularly from endoscopy reports, offers an efficient, scalable, and adaptable documentation method that captures adverse events accurately with low error rate. It facilitates immediate quality reporting and reduces manual documentation efforts.