Evaluation of a clinical decision support algorithm for patient-specific childhood immunization

Abstract

To evaluate the effectiveness of a clinical decision support system (CDSS) implementing standard childhood immunization guidelines, using real-world patient data from the Regenstrief Medical Record System (RMRS). Study subjects were age 6-years or younger in 2008 and had visited the pediatric clinic on the campus of Wishard Memorial Hospital. Immunization records were retrieved from the RMRS for 135 randomly selected pediatric patients. We compared vaccine recommendations from the CDSS for both eligible and recommended timelines, based on the child's date of birth and vaccine history, to recommendations from registered nurses who routinely selected vaccines for administration in a busy inner city hospital, using the same date of birth and vaccine history. Aggregated and stratified agreement and Kappa statistics were reported. The reasons for disagreement between suggestions from the CDSS and nurses were also identified. For the 135 children, a total of 1215 vaccination suggestions were generated by nurses and were compared to the recommendations of the CDSS. The overall agreement rates were 81.3% and 90.6% for the eligible and recommended timelines, respectively. The overall Kappa values were 0.63 for the eligible timeline and 0.80 for the recommended timeline. Common reasons for disagreement between the CDSS and nurses were: (1) missed vaccination opportunities by nurses, (2) nurses sometimes suggested a vaccination before the minimal age and minimal waiting interval, (3) nurses usually did not validate patient immunization history, and (4) nurses sometimes gave an extra vaccine dose. Our childhood immunization CDSS can assist providers in delivering accurate childhood vaccinations.