Medical subdomain classification of clinical notes using a machine learning-based natural language processing approach

Kavishwar B Wagholikar,Peter Szolovits,Wei-Hung Weng,Alexa T Mccray,Henry C Chueh

doi:10.1186/s12911-017-0556-8

Abstract

BackgroundThe medical subdomain of a clinical note, such as cardiology or neurology, is useful content-derived metadata for developing machine learning downstream applications. To classify the medical subdomain of a note accurately, we have constructed a machine learning-based natural language processing (NLP) pipeline and developed medical subdomain classifiers based on the content of the note.MethodsWe constructed the pipeline using the clinical NLP system, clinical Text Analysis and Knowledge Extraction System (cTAKES), the Unified Medical Language System (UMLS) Metathesaurus, Semantic Network, and learning algorithms to extract features from two datasets — clinical notes from Integrating Data for Analysis, Anonymization, and Sharing (iDASH) data repository (n = 431) and Massachusetts General Hospital (MGH) (n = 91,237), and built medical subdomain classifiers with different combinations of data representation methods and supervised learning algorithms. We evaluated the performance of classifiers and their portability across the two datasets.ResultsThe convolutional recurrent neural network with neural word embeddings trained-medical subdomain classifier yielded the best performance measurement on iDASH and MGH datasets with area under receiver operating characteristic curve (AUC) of 0.975 and 0.991, and F1 scores of 0.845 and 0.870, respectively. Considering better clinical interpretability, linear support vector machine-trained medical subdomain classifier using hybrid bag-of-words and clinically relevant UMLS concepts as the feature representation, with term frequency-inverse document frequency (tf-idf)-weighting, outperformed other shallow learning classifiers on iDASH and MGH datasets with AUC of 0.957 and 0.964, and F1 scores of 0.932 and 0.934 respectively. We trained classifiers on one dataset, applied to the other dataset and yielded the threshold of F1 score of 0.7 in classifiers for half of the medical subdomains we studied.ConclusionOur study shows that a supervised learning-based NLP approach is useful to develop medical subdomain classifiers. The deep learning algorithm with distributed word representation yields better performance yet shallow learning algorithms with the word and concept representation achieves comparable performance with better clinical interpretability. Portable classifiers may also be used across datasets from different institutions.

Highlights

The medical subdomain of a clinical note, such as cardiology or neurology, is useful content-derived metadata for developing machine learning downstream applications
In the iDASH dataset, combining the hybrid features of bag-of-words + Unified Medical Language System (UMLS) concepts restricted to five semantic groups, with tf-idf weighting and linear support vector machine (SVM) algorithm yielded the best performing classifier for medical subdomain classification (F1 score of 0.932, area under receiver operating characteristic curve (AUC) of 0.957), followed by bag-ofwords + all UMLS concepts or using the bag-of-words + UMLS concepts restricted to 15 semantic types as the feature representation with tf-idf and linear SVM
In the Massachusetts General Hospital (MGH) dataset, the linear SVM classifier with tfidf weighting and the hybrid feature representation of bag-of-words + UMLS concepts restricted to five semantic groups yielded the best performance (F1 score of 0.934, AUC of 0.964), which significantly outperformed the baseline naïve Bayes (NB) classifier with the term frequency and bag-of-words combination (Table 3, Fig. 2 for F1 score, Additional file 1: Figure S2 for AUC)

Summary

Introduction

The medical subdomain of a clinical note, such as cardiology or neurology, is useful content-derived metadata for developing machine learning downstream applications. Automated document classification is generally helpful in further processing clinical documents to extract these kinds of data. Training on specialist reports and applying the subdomain models on notes written by generalists, such as general practitioners and internists, will help identify the major problems of the patient that are being described. This can be useful in studying the practice and validity of clinical referral patterns, and in helping to focus attention on the most pressing medical problem subdomain of the patient

Methods

Results

Discussion

Conclusion