Comparing General and Locally-Learned Word Embeddings for Clinical Text Mining

Abstract

Dense vector-representations of words, referred to as word embeddings, capture word relations differently depending on the type and the size of the corpus they are trained on. Choosing which word embeddings to use poses a problem when applying machine learning on clinical text which contains many specialized words. In this work, we explore the effects of different embedding sources for clinical text classification using various cohort sizes on three medical prediction tasks: Clostridium Difficile infections, MRSA infections, and in-hospital mortality. We compare three embedding sources: pre-trained embeddings from large and general corpora, pre-trained embeddings from large and domain-related corpora, and locally-learned embeddings trained on the task-specific training data. We experiment with several cohort sizes ranging from 20 patients to 2,500 patients. Our results indicate that pre-trained domain-related embeddings are superior for medium-sized cohorts greater than 150 patients, while locally-learned embeddings become increasingly competitive as cohort size grows.