Academy of Nutrition and Dietetics Nutrition Research Network: Validation of a Novel Nutrition Informatics Tool to Assess Agreement Between Documented Nutrition Care and Evidence-Based Recommendations

Abstract

More evidence regarding registered dietitian nutritionist implementation of evidence-based nutrition practice guidelines (EBNPGs) is needed. We assessed the utility of an automated informatics tool to evaluate congruence of documented nutrition care with 13 individual recommendations in the diabetes mellitus (DM) EBNPG and with the guideline overall. A concurrent validation study was conducted using Nutrition Care Process Terminology documentation entered in the Academy of Nutrition and Dietetics Health Informatics Infrastructure by registered dietitian nutritionists caring for patients with DM. A 15% subset (n = 115) of the 790 patient encounters recorded were selected randomly, and the documented care was evaluated using the automated DM Expected Care Plan (ECP) Analyzer and expert audit. Recommendation-level congruence, as determined by each method, was compared using Cohen’s κ analysis, and the accuracy, sensitivity, and specificity of the DM ECP Analyzer for assessing overall guideline-level congruence was calculated with expert audits as the “gold standard.” For recommendation-level congruence, the DM ECP Analyzer identified more instances of recommendation implementation in the patient encounters, and classified more encounters as including partial or full recommendation implementation for 10 of the 13 recommendations, compared with the expert audit. There was slight to fair agreement between the DM ECP and the expert audit for most individual recommendations, with a mean ± standard deviation level of agreement of κ = .17 ± .19 across all eligible recommendations. At the guideline level, the DM Analyzer had high accuracy (98.3%) and sensitivity (99.1%) and low specificity (0%; no true negatives detected). The DM ECP Analyzer is acceptable for conducting automated audits of nutrition documentation to assess congruence of documented care with recommendations for evidence-based care. Future changes to the EBNPG, Nutrition Care Process Terminology, Academy of Nutrition and Dietetics Health Informatics Infrastructure, and the DM ECP Analyzer could potentially improve recommendation-level performance. The DM ECP Analyzer can be modified for other EBNPGs to facilitate automated assessment of guideline implementation.