Development and Evaluation of a Natural Language Processing Annotation Tool (NAT) to Facilitate Phenotyping of Cognitive Status in Electronic Health Records

Abstract

AbstractBackgroundElectronic Health Records (EHR) with large sample sizes and rich information offer great potential for dementia research, but current methods of phenotyping cognitive status are either not scalable or suffer from inaccuracies. Our objective is to evaluate whether deep learning Natural Language Processing (NLP)‐powered semi‐automated annotation can improve the speed and reliability of chart reviews for phenotyping cognitive status.MethodIn this study we developed and evaluated a semi‐automated NLP‐powered annotation tool (NAT) to facilitate phenotyping of cognitive status. Clinical experts adjudicated the cognitive status of 627 patients at Mass General Brigham (MGB) Healthcare using NAT or traditional chart reviews. Patient charts contained EHR data from two datasets: (1) Records from January 1, 2017 to December 31, 2018 for 100 Medicare beneficiaries from the MGB Accountable Care Organization (ACO), and (2) Records from 2‐years pre‐COVID diagnosis to the date of COVID diagnosis for 527 MGB patients. All EHR data from the relevant period were extracted; diagnosis codes, medications, and laboratory test values were processed and summarized. Clinical notes were processed through a deep learning NLP algorithm and a web tool was developed to present an integrated view of all data. Cognitive status was rated as cognitively normal, cognitively impaired, or undetermined. Assessment time and interrater agreement of NAT compared to manual chart reviews for cognitive status phenotyping was evaluated.ResultNAT adjudication provided higher interrater agreement (Cohen k = 0.89 vs. k = 0.80) and significant speed up (time difference mean [SD]: 1.4 [1.3] minutes, P < 0.001; ratio median [min, max]: 2.2 [0.4, 20]) over manual chart reviews. There was moderate agreement with manual chart reviews (Cohen k = 0.67). In the cases that exhibited disagreement with manual chart review, NAT adjudication was able to produce assessments that had broader clinical consensus due to its integrated view of highlighted relevant information and semi‐automated NLP features.ConclusionNAT adjudication improves the speed and reliability for phenotyping cognitive status compared to manual chart reviews. This study underscores the potential of an NLP‐based clinically adjudicated method to build large‐scale dementia research cohorts from EHR.