System Usability Scale Survey Research Articles

User experience analysis in virtual museums

The paper presents an analysis of user experience in virtual museums. The objects of interest are museums that offer a virtual walk which allows the user to visit and view exhibitions without leaving home. Among the selected objects for the study were the Museum of Auschwitz-Birkenau and the Malbork Castle Museum. The interfaces of these two virtual museums were subjected to eye tracking analysis using the Gazepoint GP3 HD eye tracker and an expert analysis using Nielsen heuristics. Additionally, a survey consisting of questions from the System Usability Scale and self-reported questions was then conducted to help gather information on usability and to collect opinions about online museums. The research group consisted of sixteen students from the Lublin University of Technology, who were presented with the same tasks to perform in the virtual museums. As a result of the research, it turned out that both according to the Nielsen heuristics and the System Usability Scale survey, the Auschwitz-Birkenau Museum was rated better, while in the eye tracking experiment both museums obtained similar results.

Development of a telemedicine virtual clinic system for remote, rural, and underserved areas using user-centered design methods.

Virtual clinics are an emerging form of telemedicine which can positively impact healthcare accessibility in remote, rural, and underserved areas. However, a virtual clinic system for these contexts must be designed appropriately. The user-centered design method can be employed to develop an appropriate virtual clinic. The development of the virtual clinic used the user-centered design method. First, a situational analysis was conducted to understand the context of public primary healthcare facilities in South Africa. Literature review, observations, and interviews were conducted, which then informed system requirements. A virtual clinic system was then developed and its usability was evaluated with doctors and nurses in a lab setting using healthy participants acting as patients. Doctors and nurses completed system usability scale surveys and provided interview feedback. The situational analysis revealed 10 key themes which were translated into a problem statement and 10 system requirements. A virtual clinic system was then developed based on these requirements. 5 doctors and 11 nurses were recruited to complete usability testing with the system. The system received an average of 80.6 scores (good to excellent) out of 100 on the system usability scale. Feedback from participants revealed key areas for improvement of the virtual clinic system, as well as opportunities for further implementation. The developed virtual clinic system demonstrated the application of the user-centered design method to telemedicine technologies for remote, rural, and underserved areas. The positive feedback received from the participants demonstrated the importance of the user-centered design method in developing technologies for enhancing service delivery in health systems. Further work will implement this system in real-world clinical settings.

Bariatric Patient Engagement in a Pre-surgery Virtual Patient Navigation Platform (VPNP).

This study analyzed patient engagement with a virtual patient navigation platform (VPNP) designed to help guide bariatric surgery candidates through the complex pre-operative workup for surgery. Data on baseline sociodemographic and medical history was collected on patients enrolled in the bariatric program at a single academic institution between March and May 2021. The System Usability Scale (SUS) survey was administered to assess VPNP usability. Two groups emerged: "engaged" (ENG; n = 30) who activated their accounts and completed the SUS, and "not-engaged" (NEG; n = 35) who did not activate their accounts (n = 13) or did not use the app (n = 22) and were therefore ineligible for the SUS survey. Analyses demonstrated that only insurance status differed between groups (private insurance: 60% versus 34.3% for ENG versus NEG, respectively; p = 0.038). SUS survey analysis demonstrated high perceived usability (median score = 86.3), corresponding to the 97th percentile of usability. The top three reasons for disengagement included being too busy (22.9%), not being interested (20%), and being unsure about the purpose of the app (20%). The VPNP scored in the 97th percentile of usability. However, given a majority of patients did not engage with the app, and engagement was associated with completing pre-surgery requirements faster (unpublished), future work will focus on mitigating identified reasons for patients not engaging.

Incorporating an Evidence-Based Bundle to Standardize the Heart Failure Discharge Process

BackgroundThis project aimed to improve adherence to postdischarge follow-up among all-cause heart failure (HF) veterans by implementing the evidence-based Focused Discharge Handoff Tool (FDHT). MethodsParticipating inpatient providers supported implementing the FDHT and enrolling veterans in the 7-day post-hospitalization telehealth and primary care appointments. ResultsTotal compliance rate with the use of FDHT was 67.3% (n = 35); remote patient monitoring home telehealth follow-up appointments improved from 18.8% (n = 8) to 50% (n = 26). The System Usability Scale survey scored below average. DiscussionUsing the FDHT to standardize the federal system’s discharge process provided insightful and promising results for supporting veterans. LimitationsDue to the small sample size, the discharge tool was not embedded into electronic medical records. Conclusion and Sustaining PlansThese data reflect positive outcomes for increasing veteran compliance with HF treatment plans by promoting 7-day telehealth appointments and safer transition of care.

Mixed-methods study of development and design needs for CommitFit, an adolescent mHealth App.

Adolescent obesity remains a significant public health issue within the United States. Mobile application technology growth and popularity offer new opportunities for research and health improvement. The development of a consolidated mobile health application (mHealth app) for adolescents on these platforms has the potential to improve health outcomes. Thus, this study describes the co-development process working with adolescent users. The aims are as follows: (I) to explore the visual design and functional requirements when developing the CommitFit mHealth app, (II) to uncover the gamification techniques that incentivize adolescents to set and achieve healthy lifestyle goals, and (III) to identify adolescent expectations when using the CommitFit mHealth application. In this mixed method study, we used semi-structured interviews/task analysis and surveys of adolescents (aged 13 to 15 years) to understand their user requirements and design preferences during the development of the CommitFit mHealth app. Interviews were conducted online, via Zoom. The survey included the user design industry-standard System Usability Scale (SUS) paired with a supplemental questionnaire on the specific features and functionalities of the CommitFit mHealth app. Participants were recruited from the electronic health record from the University of Missouri Healthcare system. Ten adolescents, aged 13 to 15 years (average of 13.6 years), were interviewed and surveyed to explore adolescent preferences with visual app design and functionality. Our inductive thematic analysis found that adolescents preferred colorful, user-friendly interfaces paired with gamification in the CommitFit mHealth app. Our analysis of SUS survey data validated our user-centered and human-system design and adolescents confirmed their design, feature, and functionality preferences. Overall, adolescent users were able to confirm their preference to have educational resources, goal recommendations, leaderboard, points, reminders, and an avatar in the app. Adolescent feedback is crucial in the successful development of our adolescent-targeted mHealth app, CommitFit. Adolescents preferred vibrant colors, easy-to-use interface, gamification, customizable and personalized, and mature graphics. Adolescents were especially motivated by gamification techniques to maintain their interest in the application and their health behavior goals. Additional research is now needed to explore the clinical effectiveness of the CommitFit mHealth app, as a health and lifestyle intervention.

Digital cancer genetic risk assessment in a gynecologic oncology clinic.

10612 Background: Population-based screening for hereditary cancer syndromes and referral of those who screen positive for genetic testing is a cost effective, evidence-based national health priority. However, under-recognition of hereditary cancer syndromes remains a critical concern, with fewer than 20% of individuals with Hereditary Breast and Ovarian Cancer Syndrome and Lynch Syndrome identified. Here, we evaluate the Ambry Genetics Comprehensive, Assessment, Risk, and Education (CARE) Program, a risk-stratification tool to assess eligibility for National Comprehensive Cancer Network (NCCN) genetic testing criteria and lifetime risk for breast cancer based on the Tyrer-Cuzick (version 8.0) risk algorithm. Methods: Between November 2022 and February 2023, new patients in a gynecologic oncology clinic were prompted by text message to complete CARE prior to their appointment. Those using CARE then completed a System Usability Scale (SUS), a validated 10-item questionnaire for the assessment of perceived usability. Results: Sixty patients presented for a new patient gynecologic oncology appointment and were prompted to complete CARE. The median patient age was 55 years (interquartile range, 40-68). The patients self-identified as Non-Hispanic White (44, 73%), Non-Hispanic Black (5, 8%), Asian (5, 8%), Hispanic (3, 5%), other (2, 3%), and unknown (1, 2%). All identified as female. Forty-three patients (72%) completed CARE. Among CARE users, 11 (26%) were identified as meeting criteria for genetic testing; of these, 9 (82%) proceeded with point-of-care same day genetic testing. Nine (21%) patients were eligible for enhanced breast cancer screening based on an estimated lifetime breast cancer risk of 20% or greater; of these, 100% were referred for high-risk breast screening. There were no patient demographics associated with completion of CARE. Forty-one patients (72%) completed the SUS survey; the median SUS score was 80 (interquartile range, 60-92.5), which corresponded to 85-89th percentile and a letter grade of A-. Conclusions: Implementation of a digital tool for cancer genetic risk assessment identified a quarter of patients to be eligible for genetic testing, among whom 82% completed same-day genetic testing. Additionally, the tool was reviewed favorably by patient users. Large randomized controlled trials of health information technology tools are needed to move closer to the goal of population-based hereditary cancer risk assessment.

Children's and Caregivers' Review of a Guided Imagery Therapy Mobile App Designed to Treat Children With Functional Abdominal Pain Disorders: Leveraging a Mixed Methods Approach With User-Centered Design.

Functional abdominal pain disorders (FAPDs) are highly prevalent and associated with substantial morbidity. Guided imagery therapy (GIT) is efficacious; however, barriers often impede patient access. Therefore, we developed a GIT mobile app as a novel delivery platform. Guided by user-centered design, this study captured the critiques of our GIT app from children with FAPDs and their caregivers. Children aged 7 to 12 years with Rome IV-defined FAPDs and their caregivers were enrolled. The participants completed a software evaluation, which assessed how well they executed specific app tasks: opening the app, logging in, initiating a session, setting the reminder notification time, and exiting the app. Difficulties in completing these tasks were tallied. After this evaluation, the participants independently completed a System Usability Scale survey. Finally, the children and caregivers were separately interviewed to capture their thoughts about the app. Using a hybrid thematic analysis approach, 2 independent coders coded the interview transcripts using a shared codebook. Data integration occurred after the qualitative and quantitative data were analyzed, and the collective results were summarized. We enrolled 16 child-caregiver dyads. The average age of the children was 9.0 (SD 1.6) years, and 69% (11/16) were female. The System Usability Scale average scores were above average at 78.2 (SD 12.6) and 78.0 (SD 13.5) for the children and caregivers, respectively. The software evaluation revealed favorable usability for most tasks, but 75% (12/16) of children and 69% (11/16) of caregivers had difficulty setting the reminder notification. The children's interviews confirmed the app's usability as favorable but noted difficulty in locating the reminder notification. The children recommended adding exciting scenery and animations to the session screen. Their preferred topics were animals, beaches, swimming, and forests. They also recommended adding soft sounds related to the session topic. Finally, they suggested that adding app gamification enhancements using tangible and intangible rewards for listening to the sessions would promote regular use. The caregivers also assessed the app's usability as favorable but verified the difficulty in locating the reminder notification. They preferred a beach setting, and theme-related music and nature sounds were recommended to augment the session narration. App interface suggestions included increasing the font and image sizes. They also thought that the app's ability to relieve gastrointestinal symptoms and gamification enhancements using tangible and intangible incentives would positively influence the children's motivation to use the app regularly. Data integration revealed that the GIT app had above-average usability. Usability challenges included locating the reminder notification feature and esthetics affecting navigation. Children and caregivers rated our GIT app's usability favorably, offered suggestions to improve its appearance and session content, and recommended rewards to promote its regular use. Their feedback will inform future app refinements.

Observer-study-based approaches to quantitatively evaluate the realism of synthetic medical images

Objective. Synthetic images generated by simulation studies have a well-recognized role in developing and evaluating imaging systems and methods. However, for clinically relevant development and evaluation, the synthetic images must be clinically realistic and, ideally, have the same distribution as that of clinical images. Thus, mechanisms that can quantitatively evaluate this clinical realism and, ideally, the similarity in distributions of the real and synthetic images, are much needed. Approach. We investigated two observer-study-based approaches to quantitatively evaluate the clinical realism of synthetic images. In the first approach, we presented a theoretical formalism for the use of an ideal-observer study to quantitatively evaluate the similarity in distributions between the real and synthetic images. This theoretical formalism provides a direct relationship between the area under the receiver operating characteristic curve, AUC, for an ideal observer and the distributions of real and synthetic images. The second approach is based on the use of expert-human-observer studies to quantitatively evaluate the realism of synthetic images. In this approach, we developed a web-based software to conduct two-alternative forced-choice (2-AFC) experiments with expert human observers. The usability of this software was evaluated by conducting a system usability scale (SUS) survey with seven expert human readers and five observer-study designers. Further, we demonstrated the application of this software to evaluate a stochastic and physics-based image-synthesis technique for oncologic positron emission tomography (PET). In this evaluation, the 2-AFC study with our software was performed by six expert human readers, who were highly experienced in reading PET scans, with years of expertise ranging from 7 to 40 years (median: 12 years, average: 20.4 years). Main results. In the ideal-observer-study-based approach, we theoretically demonstrated that the AUC for an ideal observer can be expressed, to an excellent approximation, by the Bhattacharyya distance between the distributions of the real and synthetic images. This relationship shows that a decrease in the ideal-observer AUC indicates a decrease in the distance between the two image distributions. Moreover, a lower bound of ideal-observer AUC = 0.5 implies that the distributions of synthetic and real images exactly match. For the expert-human-observer-study-based approach, our software for performing the 2-AFC experiments is available at https://apps.mir.wustl.edu/twoafc. Results from the SUS survey demonstrate that the web application is very user friendly and accessible. As a secondary finding, evaluation of a stochastic and physics-based PET image-synthesis technique using our software showed that expert human readers had limited ability to distinguish the real images from the synthetic images. Significance. This work addresses the important need for mechanisms to quantitatively evaluate the clinical realism of synthetic images. The mathematical treatment in this paper shows that quantifying the similarity in the distribution of real and synthetic images is theoretically possible by using an ideal-observer-study-based approach. Our developed software provides a platform for designing and performing 2-AFC experiments with human observers in a highly accessible, efficient, and secure manner. Additionally, our results on the evaluation of the stochastic and physics-based image-synthesis technique motivate the application of this technique to develop and evaluate a wide array of PET imaging methods.

Development and Design Needs of Mobile Health (mHealth) Apps for Adolescents.

Context: Adolescent obesity remains a significant public health issue within the United States. Application (app) technology growth and popularity offer new opportunities for research and health improvement. The development of a consolidated mobile health application (mHealth app) for adolescents on these platforms has the potential to improve health outcomes. Most mHealth apps for adolescents, particularly those in the commercial arena, are scaled-down from an adult-targeted app and lack relevant stakeholder feedback. Objectives: To identify adolescent expectations when using an mHealth application and understand visual user-interface needs, and to develop an intuitive and engaging user-interface, we aim to describe the design and functional user requirements for mHealth app. We aim to inform future researchers and app developers about adolescent needs and preferences, as identified by adolescent stakeholders. Study Design: In this mixed method study, we used surveys and interviews/task analysis of adolescents to understand their user requirements and design preferences during the development of a healthy lifestyle app (CommitFit). The survey included the user design industry-standard System Usability Scale (SUS) paired with supplemental questionnaire on the specific features and functionalities of the CommitFit mHealth app. Population Studied: Adolescents ranged from the ages of 13 to 15 years of age, with an average age of 13.6 years old. Results: Ten adolescents were interviewed and surveyed (adapted SUS and supplemental questionnaire) to explore adolescent preferences with visual app design and functionality. Our qualitative results showed that adolescents preferred colorful, user-friendly interfaces paired with gamification to use an mHealth app. Our analysis of SUS survey data validated our user-centered and human-system design and adolescents confirmed their design, feature, and functionality preferences. Adolescent users found CommitFit to be easy to use and provided guidance for visual design needs and preferences. Outcomes: Adolescent stakeholder feedback is crucial in the successful development of an adolescent-targeted mHealth app. Adolescents prefer vibrant colors, modern, easy-to-use interface, gamification and rewards, customizable and personalized, simple, and mature graphics. Adolescents were especially motivated by gamification techniques in maintaining their interest in the application and their health behavior goals.

User experience analysis while visiting selected virtual museums

The paper concerns the study of User Experience by focusing on the usability and user satisfaction aspects. The purpose of this paper was to evaluate the usability of two selected virtual museums conducted with 3 research methods: using an oculograph, a System Usability Scale (SUS) usability survey and Nielsen heuristic. The examination was conducted on the following museums that offer virtual tours: Muzeum Instrumentów Muzycznych in Poznan and Muzeum Zamojskie in Zamosc. The participants of the eye tracking experiment and the SUS survey were 22 students of Computer Science at the Politechnika Lubelska, while the Nielsen heuristics analysis was performed by 3 graduate students with relevant qualifications. The obtained eye tracking data, the results of the SUS questionnaires and the evaluation of the Nielsen heuristics were analyzed quantitatively. In addition, a qualitative analysis of eye tracking results was conducted, which provided heat maps and scanning paths. As a result, it was revealed that in the oculography method, the analyzed websites obtained comparable results. However, in the test performed with the SUS usability survey Muzeum Instrumentów Muzycznych achieved a better result. The expert team reviewing Nielsen heuristics also ranked the museum higher.

Usability and Teachability of Continuous Glucose Monitoring Devices in Older Adults and Diabetes Educators: Task Analysis and Ease-of-Use Survey.

Continuous glucose monitoring (CGM) devices continuously sense and relay glucose concentration data from the interstitial fluid to a mobile phone or receiver. Older adults benefit from this continuous monitoring of glucose levels. Proper deployment of the sensing wire is facilitated by a specialized applicator. Our aim was to assess a new seventh-generation (G7) CGM device (Dexcom, Inc) for use by adults 65 years of age or older and certified diabetes care and education specialists (CDCESs). Ease of use related to intradermal insertion and mobile app setup will be assessed and compared to the fifth- and sixth-generation systems. Formal task analysis was conducted to enumerate the number and complexity of tasks associated with CGM deployment. We recruited 10 older adults with no prior CGM experience and 10 CDCESs to assess ease of use through hands-on insertion and initiation of a G7 system followed by a survey and, for older adults, a system usability scale survey. About half as many tasks are needed to deploy G7 compared to G6. Older adults and CDCESs reported overall high usability of the G7 CGM device. CDCESs noted G7's easier setup compared to previous generations. The system usability scale score for the CGM system was 92.8, which reflects excellent usability. For CDCESs and for older adults using the G7 CGM system, cognitive burden is relatively low and reduced compared to previous CGM systems. Easing of this burden and simplification of the glucose monitoring aspect of proper diabetes management will likely contribute to improved outcomes in this population.

Usability Evaluation of a Noninvasive Neutropenia Screening Device (PointCheck) for Patients Undergoing Cancer Chemotherapy: Mixed Methods Observational Study.

Patients with cancer undergoing cytotoxic chemotherapy face an elevated risk of developing serious infection as a consequence of their treatment, which lowers their white blood cell count and, more specifically, their absolute neutrophil count. This condition is known as neutropenia. Neutropenia accompanied by a fever is referred to as febrile neutropenia, a common side effect of chemotherapy with a high mortality rate. The timely detection of severe neutropenia (<500 absolute neutrophil count/μL) is critical in detecting and managing febrile neutropenia. Current methods rely on blood draws, which limit them to clinical settings and do not allow frequent or portable monitoring. In this study, we demonstrated the usability of PointCheck, a noninvasive device for neutropenia screening, in a simulated home environment without clinical supervision. PointCheck automatically performs microscopy through the skin of the finger to image the blood flowing through superficial microcapillaries and enables the remote monitoring of neutropenia status, without requiring venipuncture. This study aimed to evaluate the usability of PointCheck, a noninvasive optical technology for screening severe neutropenia, with the goal of identifying potential user interface, functionality, and design issues from the perspective of untrained users. We conducted a multicenter study using quantitative and qualitative approaches to evaluate the usability of PointCheck across 154 untrained participants. We used a mixed method approach to gather usability data through user testing observations, a short-answer qualitative questionnaire, and a standardized quantitative System Usability Scale (SUS) survey to assess perceived usability and satisfaction. Of the 154 participants, we found that 108 (70.1%) scored above 80.8 on the SUS across all sites, with a mean SUS score of 86.1 across all sites. Furthermore, the SUS results indicated that, out of the 151 users who completed the SUS survey, 145 (96%) found that they learned how to use PointCheck very quickly, and 141 (93.4%) felt very confident when using the device. We have shown that PointCheck, a novel technology for noninvasive, home-based neutropenia detection, can be safely and effectively operated by first-time users. In a simulated home environment, these users found it easy to use, with a mean SUS score of 86.1, indicating an excellent perception of usability and placing this device within the top tenth percentile of systems evaluated for usability by the SUS. ClinicalTrials.gov NCT04448314; https://clinicaltrials.gov/ct2/show/NCT04448314 (Hospital Universitario 12 de Octubre registration) and NCT04448301; https://clinicaltrials.gov/ct2/show/NCT04448301 (Boston Medical Center registration).

The Roles of a Secondary Data Analytics Tool and Experience in Scientific Hypothesis Generation in Clinical Research: Protocol for a Mixed Methods Study

BackgroundScientific hypothesis generation is a critical step in scientific research that determines the direction and impact of any investigation. Despite its vital role, we have limited knowledge of the process itself, thus hindering our ability to address some critical questions.ObjectiveThis study aims to answer the following questions: To what extent can secondary data analytics tools facilitate the generation of scientific hypotheses during clinical research? Are the processes similar in developing clinical diagnoses during clinical practice and developing scientific hypotheses for clinical research projects? Furthermore, this study explores the process of scientific hypothesis generation in the context of clinical research. It was designed to compare the role of VIADS, a visual interactive analysis tool for filtering and summarizing large data sets coded with hierarchical terminologies, and the experience levels of study participants during the scientific hypothesis generation process.MethodsThis manuscript introduces a study design. Experienced and inexperienced clinical researchers are being recruited since July 2021 to take part in this 2×2 factorial study, in which all participants use the same data sets during scientific hypothesis–generation sessions and follow predetermined scripts. The clinical researchers are separated into experienced or inexperienced groups based on predetermined criteria and are then randomly assigned into groups that use and do not use VIADS via block randomization. The study sessions, screen activities, and audio recordings of participants are captured. Participants use the think-aloud protocol during the study sessions. After each study session, every participant is given a follow-up survey, with participants using VIADS completing an additional modified System Usability Scale survey. A panel of clinical research experts will assess the scientific hypotheses generated by participants based on predeveloped metrics. All data will be anonymized, transcribed, aggregated, and analyzed.ResultsData collection for this study began in July 2021. Recruitment uses a brief online survey. The preliminary results showed that study participants can generate a few to over a dozen scientific hypotheses during a 2-hour study session, regardless of whether they used VIADS or other analytics tools. A metric to more accurately, comprehensively, and consistently assess scientific hypotheses within a clinical research context has been developed.ConclusionsThe scientific hypothesis–generation process is an advanced cognitive activity and a complex process. Our results so far show that clinical researchers can quickly generate initial scientific hypotheses based on data sets and prior experience. However, refining these scientific hypotheses is a much more time-consuming activity. To uncover the fundamental mechanisms underlying the generation of scientific hypotheses, we need breakthroughs that can capture thinking processes more precisely.International Registered Report Identifier (IRRID)DERR1-10.2196/39414

A Pilot Study on Extending the SUS Survey: Early Results

The System Usability Scale (SUS) is a popular method to measure the subjective usability of a system, due largely to the simplicity and rapidity of both collecting and analyzing data. A drawback is that the SUS generates a single unidimensional usability score from 0-100. Several researchers have amassed larger datasets across multiple projects to allow for analysis on additional methods to glean insights from the SUS survey. Along these lines, we investigate the practical value of extending the SUS survey with additional items such as open text responses, and test underlying assumptions of how SUS results are interpreted. We found that while a lower SUS score does generally correlate to a stronger desire to modify the system, people generally want to make modifications to a system regardless of its usability. Further, we found that the amount of user feedback related to modifications to a system provided predicted subjective usability ratings.

Prototype Development and Usability Evaluation of a Clinical Decision Support Tool for Pharmacogenomic Pharmacy in Practice.

Pharmacogenetics, a subset of precision medicine, provides a way to individualize drug dosages and provide tailored drug therapy to patients. This revolution in prescribing techniques has resulted in a knowledge deficit for many healthcare providers on the proper way to use pharmacogenetics in practice. This research study explored the potential adoption of clinical decision support system mobile apps by clinicians through investigating the initial usability of the PGx prototype application in an effort to address the lack of such tools used in practice. The study method included usage of a clinical decision support system programmed within our pharmacogenomics drug dosage application (called PGx) in a simulated environment. Study participants completed the System Usability Scale survey to report on the perceived usefulness and ease of use of the mobile app. The PGx app has a higher perceived usability than 85% of all products tested, considered very good usability for a product. This general usability rating indicates that the nurse practitioner students find the application to be a clinical decision support system that would be helpful to use in practice.

A Quality Improvement Project on Agitation Management in the Emergency Department

IntroductionAgitation is common in the emergency department. When agitation is not detected early, patients can become aggressive and violent, potentially leading to restraint use and subsequent injury. The goals of the project were early detection and management of patient agitation, reduction of restraint use in the emergency department, and determination of the usability of the Behavioral Activity Rating Scale. MethodsThis quality improvement project was assessed using a pre- and posttest single unit design, comparing 4 months of postimplementation data to historic controls at the same time of year in the previous year. The intervention was implementing the Behavioral Activity Rating Scale in the ED electronic medical record. Data were collected through retrospective chart review and nurse survey. From September through December of both 2017 and 2018, data were collected on restraint use. The 4-month 2018 data collection period included measures of Behavioral Activity Rating Scale documentation and the System Usability Scale survey for nurses to measure ease of usability of the Behavioral Activity Rating Scale. ResultsThe Behavioral Activity Rating Scale was documented frequently (n = 4 867 documentations) by emergency nurses to assess patients with behavioral health and medical complaints (n = 780). Nurses identified 18 episodes of violent behavior in behavioral health patients on the Behavioral Activity Rating Scale (2.31%) and applied restraints 18 times. The most common chief complaints for patients who were identified as violent was suicidal ideation (n = 6; 33.33%).In 2017, there were 20 episodes of restraint use during the same time period, a nonsignificant difference (χ2 = 0.72; P = 0.40). However, only 2 patients were kept in restraints longer than 1 day in 2018 compared with 8 in 2017. Emergency nurses found the Behavioral Activity Rating Scale to be usable in the structured usability assessment (μ = 83.46; SD = 11.73). DiscussionThe Behavioral Activity Rating Scale is a usable tool for emergency nurses to assess for patient agitation. With the incorporation of agitation management interventions, the ED team can potentially manage agitation before violence occurs. Further studies are needed on the use of agitation or aggression assessment tools for managing patient behavioral activity such as aggression in the emergency department.

The Usability of the Plan Quality Index to Support Evaluation of Community Health Action Plans.

The U.S. Army requires community health coalitions to develop targeted action plans in order to more effectively address top public health priorities. Reviews of community action plans indicate that they are often poorly developed and not fully implemented. To date, the U.S. Army has not implemented a standardized tool or process to evaluate action plans and provide recommendations for improvement to local installations. The Plan Quality Index (PQI) is an evidence-based, standardized tool that has been used successfully by technical experts to review and improve community action plans related to the prevention of obesity, injury and violence, and cancer. The objective of this project was to determine the usability of the PQI and to provide recommendations for improving community health action plans focused on injury prevention for the U.S. Army. Five Health Promotion Project Officers and one Health Promotion Project Assistant at Army Public Health Center were trained on the PQI and reviewed injury prevention action plans for 17 installations. After using the PQI to assess injury prevention action plans, the individuals were given a System Usability Scale (SUS) survey to assess the usability of the PQI. Results of the SUS survey can range from 0 to 100, but do not represent percentages. A score of 65 indicates "greater than average" usability. Participant scores on the SUS demonstrated that the PQI has a high degree of usability. Mean usability was calculated at 83.3 (range 72.5-97.5, median 85, sd +/- 9.3). The PQI demonstrated a high level of usability by technical experts for providing feedback and recommendations regarding community health action plans for injury prevention in the U.S. Army. As a result, the PQI may be beneficial as a standard tool for community health improvement planning and consultation throughout the Army Public Health Center.

Feasibility of remote digital monitoring using wireless Bluetooth monitors, the Smart Angel\u2122 app and an original web platform for patients following outpatient surgery: a prospective observational pilot study

BackgroundRemote monitoring of mean arterial blood pressure (MAP), heart rate (HR) or oxygen saturation (SpO2) remains a challenge in outpatient surgery. This study evaluates a new digital technology (Smart Angel™) for remotely monitoring hemodynamic data in real time: data transmitted from the patient’s home to a central server, using a dedicated web-based software package.MethodsAdults scheduled for elective outpatient surgery were prospectively enrolled. In the first 5 postoperative days, patients completed a self-report questionnaire (pain, comfort, nausea, vomiting) and recorded SpO2, HR and MAP via two wireless Bluetooth monitors connected to a 4G tablet to transmit the data to a website, in real time, using Smart Angel™ software. Before transmission to the website, these data were also self-reported by the patient on a paper form. The primary outcome was the proportion of variables (self-monitored physiological data + questionnaire scores) correctly transmitted to the hospital via the system compared with the paper version.On Day 5, a system usability scale survey (SUS score 1–100) was also attributed.ResultsFrom May 2018 to September 2018, data were available for 29 out of 30 patients enrolled (1 patient was not discharged from hospital after surgery). The remote monitoring technology recorded 2038 data items (62%) compared with 2656 (82%) items recorded on the paper form (p = 0.001). The most common errors with the remote technology were software malfunctioning when starting the MAP monitor and malfunctioning between the tablet and the Bluetooth monitor. No serious adverse events were noted. The SUS score for the system was 85 (68–93) for 26 patients.ConclusionThis work evaluates the ability of a pilot system for monitoring remote physiological data using digital technology after ambulatory surgery and highlights the digital limitations of this technology. Technological improvements are required to reduce malfunctioning (4G access, transmission between apps).Trial registrationClinicalTrials.gov (NCT03464721) (March 8, 2018).

Automated Tool for Health Utility Assessments: The Gambler II.

Background. The Gambler II is a web-based utility assessment tool supporting visual analogue scale (VAS), standard gamble (SG), and time trade-off (TTO) utility assessments. It contains novel features, including an easy to use project development authoring tool and use of multimedia clips for health state descriptions. Objectives. Evaluate the usability and understandability of the patient-facing side of The Gambler. Investigate the feasibility of using The Gambler and evaluate its impact on patient knowledge regarding the relevant health states. Materials and Methods. We used The Gambler to assess utilities on a convenience sample of 55 users for common long-term complications of type 2 diabetes mellitus: diabetic neuropathy, diabetic retinopathy, and diabetic foot infection requiring transmetatarsal amputation. Using VAS, SG, and TTO, we collected metadata, such as time spent on each assessment and the entire assessment process. We evaluated usability with an adaptation of the System Usability Scale survey and understandability. We evaluated impact on knowledge gained through knowledge assessments about these complications before and after use of The Gambler. Results. Overall satisfaction with The Gambler was high, 4.02 on a 5-point scale. Usability rated highly at 84.93 on a normalized scale between 0 and 100. Knowledge scores increased significantly following use of The Gambler from pretest mean of 68% to posttest mean of 76% (P < 0.01). Average time using the software: ∼7½ minutes. Conclusions. The Gambler is an easy to use and understand computer-based tool for utility assessment. It is feasible to use within clinical encounters to support shared decision making, and it has unique features that make it a powerful tool for investigators interested in research on health utilities.

Association of an Electronic Health Record Add-on App for Neonatal Bilirubin Management With Physician Efficiency and Care Quality

The usefulness of electronic health record (EHR) systems could be significantly enhanced by innovative, third-party EHR add-on apps. To evaluate whether an EHR add-on app for neonatal bilirubin management can save clinicians time and improve patient care. This quality improvement study was conducted at the University of Utah Health Well Baby nursery and outpatient clinics and consisted of 4 substudies: (1) time savings were estimated in an experimental task-timing study comparing the time required for physicians to manage newborns' bilirubin levels with and without the add-on app, (2) app use was estimated from app logs, (3) health care use measures and guideline compliance were compared retrospectively before and after the intervention, and (4) clinician-perceived usability was measured through System Usability Scale surveys. The study took place between April 1, 2016, and September 3, 2019. Data analyses were conducted from October 30, 2018, to September 23, 2019. At baseline, clinicians used a manual approach to ensure compliance with an evidence-based clinical guideline for neonatal bilirubin management. To facilitate guideline compliance, an EHR add-on app that automatically retrieves, organizes, and visualizes relevant patient data was developed. The app provides patient-specific assessments and recommendations, including the risk of rebound hyperbilirubinemia following phototherapy based on a predictive model. The add-on app was integrated with the University of Utah Health EHR on April 12, 2017. Clinician time savings, app use, health care use measures, guideline-compliant phototherapy ordering, and perceived usability as measured by the System Usability Scale survey. The survey is composed of 10 statements with responses ranging from 1 (strongly disagree) to 5 (strongly agree). The survey results in a single score ranging from 0 to 100, with ratings described as worst imaginable (mean System Usability Scale score, 12.5), awful (20.3), poor (35.7), okay (50.9), good (71.4), excellent (85.5), and best imaginable (90.9). In 2018, the application was used 20 516 times by clinicians for 91.84% of eligible newborns. Use of the app saved 66 seconds for bilirubin management tasks compared with a commonly used tool (95% CI, 53-79 seconds; P < .001). Following the intervention, health care use rates remained stable, while orders for clinically appropriate phototherapy during hospitalization increased for newborns with bilirubin levels above the guideline-recommended threshold (odds ratio, 1.84; 95% CI, 1.16-2.90; P = .009). Surveys indicated excellent usability (System Usability Scale score, 83.90; 95% CI, 81.49-86.31). Well-designed EHR add-on apps may save clinicians time and improve patient care. If time-saving apps, such as the bilirubin app, were implemented widely across institutions and care domains, the potential association with improved patient care and clinician efficiency could be significant. The University of Utah Health bilirubin app is being prepared for release into EHR app stores as free-to-use software.