Leveraging MapReduce to efficiently extract associations between biomedical concepts from large text data

Abstract

Large biomedical text data represents an important source of information that not only enables researchers to discover in-depth knowledge about biological systems, but also helps healthcare professionals do evidence-based medicine in clinical settings. However, investigating and analyzing these data is often both data-intensive and computation-intensive. In this paper, we investigate how to use MapReduce, a parallel and distributed programming paradigm, to efficiently mine the associations between biomedical concepts extracted from a large set of biomedical articles. First, biomedical concepts were obtained by matching text to Unified Medical Language System (UMLS) Metathesaurus, a biomedical vocabulary and standard database. Then we developed a MapReduce algorithm that could be used to calculate a category of interestingness measures defined on the basis of a 2×2 contingency table. This algorithm consists of two MapReduce jobs and takes a stripes approach to reduce the number of intermediate results. Experiments were conducted using Amazon Elastic MapReduce (EMR) with an input of 33,960 articles from TREC (Text REtrieval Conference) 2006 Genomics Track. Performance test indicated that our algorithm had approximately linear scalability and was more efficient than a ‘‘pairs’’ approach in the literature. The physician in our project team evaluated a subset of the association mining results related to drug-disease treatment and found that meaningful association rules ranked high.