Enhancing pre-trained contextual embeddings with triplet loss as an effective fine-tuning method for extracting clinical features from electronic health record derived mental health clinical notes

Abstract

The development and application of real-world evidence in the field of mental health trails other therapeutic areas like oncology and cardiovascular diseases, largely because of the lack of frequent, structured outcomes measures in routine clinical care. A wealth of valuable patient-level clinical data resides in an unstructured format in clinical notes documented at each clinical encounter. Manual extraction of this information is not scalable, and heterogeneity in recording patterns and the heavily context-dependent nature of the content renders keyword-based automated searches of little practical value. While state-of-the-art natural language processing (NLP) models based on the transformer architecture have been developed for information extraction tasks in the mental health space, they are not trained on unstructured clinical data that capture the nuances of different dimensions of mental health (e.g., symptomology, social history, etc.). We have developed a novel transformer architecture-based NLP model to capture core clinical features of patients with major depressive disorder (MDD). Initialized on MentalBERT model weights, we pre-trained our model further on clinical notes from routine mental health care and fine-tuned using triplet loss, an effective feature embedding regularizer which boosts classification and extraction of 3 specific features in patients with MDD: anhedonia, suicidal ideation with plan or intent (SP), and suicidal ideation without plan or intent (SI) or where plan or intent are unknown. Training and testing data were annotated by mental health clinicians. Using triplet loss for fine tuning led to improvement in model performance benchmarked against other standard models (MentalBERT and BioClinicalBERT) on the same tasks, achieving F1 scores of 0.99 for anhedonia, 0.94 for SP, and 0.88 for SI. Model robustness was tested by testing sensitivity of model predictions on modifications to test sentences. The application of such an NLP model can be further scaled to capture clinical features of other disorders as well as other domains like social history or history of illness.