Using Evaluation to Inform the BioSense Platform: Results from a 2018 Survey

Abstract

ObjectiveTo assess the present status of utility, functionality, usability and user satisfaction of the BioSense Platform.IntroductionSince 2015, CDC’s Division of Health Informatics and Surveillance staff have conducted evaluations to provide information on the utility, functionality, usability and user satisfaction associated with the National Syndromic Surveillance Program’s BioSense Platform tools. The BioSense Platform tools include: 1) Access and Management Center (AMC), a tool that enables site administrators to manage users and data permissions; 2) Electronic Surveillance System for Early Notification of Community-based Epidemics (ESSENCE), a software application that enables syndromic surveillance related data visualization and analysis; 3) Adminer, a tool that allows users to access site data on the datamart; and 4) Rstudio, an application that can be used for data analysis and visualization. The evaluation findings have informed activities that led to improvements in functionality, development or procurement of platform associated tools, and development of resource materials. In May 2018, NSSP conducted an evaluation with eight jurisdictions that participated in the first user acceptance testing (UAT) evaluations in 2015. The purpose of the evaluation was to assess the present status of utility, functionality, usability and user satisfaction of the tools on the BioSense Platform, and delineate progress since 2015.MethodsCDC’s evaluation framework and utilization-focused evaluation were used to inform and engage stakeholders, develop the evaluation questions, metrics, and methodology. Eight selected jurisdictions participated in an online, Epi-Info survey that captured quantitative and qualitative information. Prior to the survey, participants received a presentation about the evolution of the BioSense Platform since 2015, and were provided an overview of components to evaluate. The participants were asked to assess the following key areas based on use of the BioSense Platform within the past 30 days: 1) the utility, functionality and usability of the AMC, ESSENCE, Adminer and Rstudio; 2) how well the enhanced data flow has enabled them to conduct syndromic surveillance activities; 3) usefulness of the quick start guides. Additionally, participants were asked to provide suggestions for other improvements to the BioSense Platform, and to indicate their overall satisfaction. Descriptive statistics were generated and thematic analysis was conducted to identify themes from qualitative responses.ResultsOverall, participant’s responses remained positive about the utility, functionality, usability and overall satisfaction of the BioSense Platform. Participants indicated using the BioSense Platform regularly (e.g. daily, weekly and/or monthly) within those 30 days. Certain functions have been used more than others across the various tools to conduct syndromic surveillance, with at least 50% of participants reporting use. These included creating data access rules, viewing and verifying raw and processed data, running time series, conducting free-text queries, and assessing data details and total ER visit counts by hospital, county/region, or state. The challenges ranged from tool performance to user interpretation of the function. Participants reported that the enhanced data flow improved their data quality and helped identify issues. Although participants scored ESSENCE to have average usability per the system usability scale (SUS score=63.5 in 2018), the BioSense Platform and its tools were reported as useful by 88% of participants. Further, participants continue to be comfortable using the AMC, however creating data access rules that are outside of simple use cases continue to be a challenge. Participants comfort level with Adminer improved from 2016 to 2018 with all participants reporting comfortable in using the tool. The use of each tool’s quick start guides varied. Of those who used the guides, all of the participants agreed that the Adminer and Data Dictionary guides were useful. There was a smaller number of participants agreeing that the other guides were useful. Lastly, participants provided recommendations to improving the BioSense Platform. The most frequent recommendations were improving the data access control architecture, and sharing aggregate data with hospitals in their state.ConclusionsThe development and operationalization of the BioSense Platform and associated tools has been in an environment of continuing advancements in technology and changing public health needs and priorities. Up-to-date evaluation activities have helped to ensure that BioSense is best suited to address these challenges and meet the syndromic surveillance needs of users. Overall, the findings outlined above indicate that the functionality and utility of BioSense are well suited to meet user needs.ReferencesBangor A, Kortm P, Miller J. Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale. Journal of Usability Studies. 2009; 4:114-123.