Is artificial intelligence getting too much credit in medical genetics?

Abstract

Artificial intelligence has lately proven useful in the field of medical genetics. It is already being used to interpret genome sequences and diagnose patients based on facial recognition. More recently, large-language models (LLMs) such as ChatGPT have been tested for their capacity to provide medical genetics information. It was found that ChatGPT performed similarly to human respondents in factual and critical thinking questions, albeit with reduced accuracy in the latter. In particular, ChatGPT's performance in questions related to calculating the recurrence risk was dismal, despite only having to deal with a single disease. To see if challenging ChatGPT with more difficult problems may reveal its flaws and their bases, it was asked to solve recurrence risk problems dealing with two diseases instead of one. Interestingly, it managed to correctly understand the mode of inheritance of recessive diseases, yet it incorrectly calculated the probability of having a healthy child. Other LLMs were also tested and showed similar noise. This highlights a major limitation for clinical use. While this shortcoming may be solved in the near future, LLMs may not be ready yet to be used as an effective clinical tool in communicating medical genetics information.