User Performance Evaluation Research Articles

User Performance Evaluation of Four Blood Glucose Monitoring Systems Applying ISO 15197:2013 Accuracy Criteria and Calculation of Insulin Dosing Errors.

IntroductionThe international standard ISO 15197:2013 requires a user performance evaluation to assess if intended users are able to obtain accurate blood glucose measurement results with a self-monitoring of blood glucose (SMBG) system. In this study, user performance was evaluated for four SMBG systems on the basis of ISO 15197:2013, and possibly related insulin dosing errors were calculated. Additionally, accuracy was assessed in the hands of study personnel.MethodsAccu-Chek® Performa Connect (A), Contour® plus ONE (B), FreeStyle Optium Neo (C), and OneTouch Select® Plus (D) were evaluated with one test strip lot. After familiarization with the systems, subjects collected a capillary blood sample and performed an SMBG measurement. Study personnel observed the subjects’ measurement technique. Then, study personnel performed SMBG measurements and comparison measurements. Number and percentage of SMBG measurements within ± 15 mg/dl and ± 15% of the comparison measurements at glucose concentrations < 100 and ≥ 100 mg/dl, respectively, were calculated. In addition, insulin dosing errors were modelled.ResultsIn the hands of lay-users three systems fulfilled ISO 15197:2013 accuracy criteria with the investigated test strip lot showing 96% (A), 100% (B), and 98% (C) of results within the defined limits. All systems fulfilled minimum accuracy criteria in the hands of study personnel [99% (A), 100% (B), 99.5% (C), 96% (D)]. Measurements with all four systems were within zones of the consensus error grid and surveillance error grid associated with no or minimal risk. Regarding calculated insulin dosing errors, all 99% ranges were between dosing errors of − 2.7 and + 1.4 units for measurements in the hands of lay-users and between − 2.5 and + 1.4 units for study personnel. Frequent lay-user errors were not checking the test strips’ expiry date and applying blood incorrectly.ConclusionsData obtained in this study show that not all available SMBG systems complied with ISO 15197:2013 accuracy criteria when measurements were performed by lay-users.Trial RegistrationThe study was registered at ClinicalTrials.gov (NCT02916576).FundingAscensia Diabetes Care Deutschland GmbH.Electronic supplementary materialThe online version of this article (10.1007/s13300-018-0392-6) contains supplementary material, which is available to authorized users.

P134 Home based faecal calprotectin testing: a Canadian user performance evaluation study of IBDoc®

P134 Home based faecal calprotectin testing: a Canadian user performance evaluation study of IBDoc®

System Accuracy and User Performance Evaluation of an Improved System for Self-Monitoring of Blood Glucose.

An improved test cassette for the integrated Accu-Chek® Mobile system (Roche Diabetes Care GmbH, Mannheim, Germany) has been developed. System accuracy of this improved system was evaluated based on ISO 15197:2013, clause 6.3, for three reagent system lots. According to this standard, at least 95% of the system's measurement results shall be within ±15 mg/dL and ±15% of the results of the comparison method at glucose concentrations <100 mg/dL and ≥100 mg/dL (accuracy criterion A), respectively, and at least 99% of results shall be within consensus error grid zones A and B (accuracy criterion B). In addition, accuracy was evaluated in the hands of users based on ISO 15197:2013, clause 8, with one reagent system lot.

User performance evaluation and real-time guidance in cloud-based physical therapy monitoring and guidance system

The effectiveness of traditional physical therapy may be limited by the sparsity of time a patient can spend with the physical therapist (PT) and the inherent difficulty of self-training given the paper/figure/video instructions provided to the patient with no way to monitor and ensure compliance with the instructions. In this paper, we propose a cloud-based physical therapy monitoring and guidance system. It is able to record the actions of the PT as he/she demonstrates a task to the patient in an offline session, and render the PT as an avatar. The patient can later train himself by following the PT avatar and getting real-time guidance on his/her device. Since the PT and user (patient) motion sequences may be misaligned due to human reaction and network delays, we propose a Gesture-Based Dynamic Time Warping algorithm that can segment the user motion sequence into gestures, and align and evaluate the gesture sub-sequences, all in real time. We develop an evaluation model to quantify user performance based on different criteria provided by the PT for a task, trained with offline subjective test data consisting of user performance and physical therapist scores. Moreover, we design three types of guidance which can be provided after each gesture based on user score, and conduct subjective tests to validate their effectiveness. Experiments with multiple subjects show that the proposed system can effectively train patients, give accurate evaluation scores, and provide real-time guidance which helps the patients learn the tasks and reach the satisfactory score with less time.

Analysis of "Accuracy and User Performance Evaluation of a New, Wireless-Enabled Blood Glucose Monitoring System That Links to a Smart Mobile Device".

In the study published in Journal of Diabetes, Science and Technology, Timothy S. Bailey and coauthors evaluated system accuracy and user performance of a new, wireless-enabled blood glucose monitoring system, the Contour Plus ONE blood glucose monitoring system. The authors declare that the study protocol was performed according to ISO 15197:2013 guidelines, in particular to clauses 6.3 and 8. Results show a high level of accuracy of the product in a laboratory setting as well as in the hands of lay users. However, differences between both study parts emerge that stimulate discussion about the used comparison method.

Accuracy and user performance evaluation as key factors in self-monitoring in diabetes patients

Accuracy and user performance evaluation as key factors in self-monitoring in diabetes patients

Intelligent Biohazard Training Based on Real-Time Task Recognition

Virtual environments offer an ideal setting to develop intelligent training applications. Yet, their ability to support complex procedures depends on the appropriate integration of knowledge-based techniques and natural interaction. In this article, we describe the implementation of an intelligent rehearsal system for biohazard laboratory procedures, based on the real-time instantiation of task models from the trainee’s actions. A virtual biohazard laboratory has been recreated using the Unity3D engine, in which users interact with laboratory objects using keyboard/mouse input or hand gestures through a Kinect device. Realistic behavior for objects is supported by the implementation of a relevant subset of common sense and physics knowledge. User interaction with objects leads to the recognition of specific actions, which are used to progressively instantiate a task-based representation of biohazard procedures. The dynamics of this instantiation process supports trainee evaluation as well as real-time assistance. This system is designed primarily as a rehearsal system providing real-time advice and supporting user performance evaluation. We provide detailed examples illustrating error detection and recovery, and results from on-site testing with students from the Faculty of Medical Sciences at Kyushu University. In the study, we investigate the usability aspect by comparing interaction with mouse and Kinect devices and the effect of real-time task recognition on recovery time after user mistakes.

User testing and performance evaluation of the Electronic Quality Improvement Platform for Plans and Pharmacies

ObjectiveTo user-test and evaluate a performance information management platform that makes standardized, benchmarked medication use quality data available to both health plans and community pharmacy organizations. SettingMultiple health/drug plans and multiple chain and independent pharmacies across the United States. EvaluationDuring the first phase of the study, user experience was measured via user satisfaction surveys and interviews with key personnel (pharmacists, pharmacy leaders, and health plan leadership). Improvements were subsequently made to the platform based on these findings. During the second phase of the study, the platform was implemented in a greater number of pharmacies and by a greater number of payers. User experience was then reevaluated to gather information for further improvements. Results: The surveys and interviews revealed that users found the Webbased platform easy to use and beneficial in terms of understanding and comparing performance metrics. Primary concerns included lack of access to real-time data and patient-specific data. Many users also expressed uncertainty as to how they could use the information and data provided by the platform. Conclusion: The study findings indicate that while information management platforms can be used effectively in both pharmacy and health plan settings, future development is needed to ensure that the provided data can be transferred to pharmacy best practices and improved quality care.

Biometric identification with high frequency electrocardiogram: Unregistered user refusal method and performance evaluation.

As a new modality for biometric identification, electrocardiogram-based identification technique has been developed. We proposed a technique with high frequency component of electrocardiogram (HFECG) in QRS segment. In this report, an unregistered user refusal algorithm was combined with the artificial neural network based waveform classifier. The refusal function was realized by simple thresholding technique. HFECGs from twenty collaborators were used for supervised learning. Twenty HFECGs from the same collaborators were tested and false acceptance rate (FAR) and false rejection rate (FRR) were evaluated. Ten HFECGs from other collaborators were also tested to find unregistered user refusal performance. The results show that FAR and FRR in the registrants can be kept within 1%, however, unregistered user refusal performance was not acceptable under the same condition.

Accuracy and user performance evaluation of the Contour® Next Link 2.4 blood glucose monitoring system

BackgroundThe Contour® Next Link 2.4 blood glucose monitoring system (BGMS), which communicates wirelessly exclusively with a Medtronic insulin pump, was assessed in 2 separate studies. MethodsIn the laboratory study (N=100), subject capillary fingertip samples were tested in duplicate using 3 test strip lots; accuracy was assessed based on ISO 15197:2013 section 6.3 accuracy criteria (95% within ±15mg/dl or ±15% of reference for glucose <100 and ≥100mg/dl, respectively). In the clinical trial (N=219), untrained subjects with diabetes tested capillary fingertip and palm blood samples and completed an ease-of-use questionnaire. BGMS and YSI glucose analyzer results were compared based on ISO 15197:2013 section 8 accuracy criteria. ResultsIn the laboratory study, 100% of results met ISO 15197:2013 section 6.3 accuracy criteria. In the clinical trial, 98.6% of capillary fingertip and 97.2% of palm results from subjects met ISO 15197:2013 section 8 accuracy criteria. By Parkes-Consensus Error Grid analysis, 100% of subject capillary fingertip results were within Zone A. Based on questionnaire results, the majority of subjects found the BGMS easy to use. ConclusionsThe BGMS exceeds ISO 15197:2013 section 6.3 and section 8 accuracy criteria in the laboratory and in the hands of untrained users with diabetes.

Experimental evaluation of user performance on two-dimensional and three-dimensional perspective displays in discrete-event simulation

Several experiments were carried out to compare the impacts of using a two dimensional (2D) plan view or a three dimensional (3D) perspective view in discrete event simulation visual displays. The experiments measured the performance of participants in spotting errors, describing the model, and suggesting improvements to the system. The participants using the 3D perspective display performed much better in spotting errors, taking on average about one third of the time of participants observing the 2D display. They also did much better in describing the model. There was no significant difference in suggesting improvements although this may have been because this task was easy. Most participants preferred the 3D perspective view when asked to compare the displays. The experiments indicate that the detailed design of the visual display may have a considerable effect on some of the tasks in a simulation project and hence on whether the overall project is successful.

Performance Analysis of Primary User RSS/DoA Estimation and Localization in Cognitive Radio Networks Using Sectorized Antennas

In this letter we present analytical performance evaluation of primary user (PU) RSS/DoA estimation and localization through cooperating cognitive radios (CRs). We assume that the CRs are equipped with sectorized antennas, an antenna model we use to describe different types of directional antennas with a low hardware overhead (e.g. only a single RF front-end). We first derive the bias and mean square error (MSE) of RSS and DoA estimates based on the simplified least square (SLS) algorithm. We then proceed to derive the theoretical MSE of PU localization using the Stansfield algorithm that fuses SLS estimates from multiple nodes. Simulation results studying the impact of various system parameters on the RSS/DoA estimation and localization accuracy are presented to verify theoretical derivations, as well as provide design guidelines for localization systems based on sectorized antennas.

User performance and satisfaction of tablet physical keyboards

This study presents an evaluation of user performance and satisfaction of three physical keyboards that accompany two popular tablet computers. All keyboards were dual purpose in that they served as tablet covers in addition to typing input devices. The keyboards varied in weight, thickness, and key travel. The thinnest keyboard featured durability and pressure-sensitive keys while the other two keyboards were slightly thicker, but used mechanical keys. Participants unfamiliar with the keyboards were asked to type a series of phrases on each keyboard after a short practice. Typing performance, and perceived usability and workload were assessed. Results show a clear advantage in both performance and satisfaction for the mechanical-key keyboards. Users demonstrated typing speeds approximately 10 words per minute (WPM) slower and were prone to typing errors such as incorrect key substitutions, omissions, and inadvertent insertions with the thinner, pressure-sensitive keyboard. In addition, users reported the flat keyboard was more mentally demanding, more frustrating, and required more effort to use. Results from this study reveal the importance of tactile feedback to user typing performance and satisfaction. Designers of tablet physical keyboards must assess the tradeoff between optimal keyboard size, weight, and thickness, and a design that affords accurate typing.

Syringeless power injector versus dual-syringe power injector: economic evaluation of user performance, the impact on contrast enhanced computed tomography (CECT) workflow exams, and hospital costs

ObjectiveThe utilization of diagnostic imaging has substantially increased over the past decade in Europe and North America and continues to grow worldwide. The purpose of this study was to develop an economic evaluation of a syringeless power injector (PI) versus a dual-syringe PI for contrast enhanced computed tomography (CECT) in a hospital setting.Materials and methodsPatients (n=2379) were enrolled at the Legnano Hospital between November 2012 and January 2013. They had been referred to the hospital for a CECT analysis and were randomized into two groups. The first group was examined with a 256-MDCT (MultiDetector Computed Tomography) scanner using a syringeless power injector, while the other group was examined with a 64-MDCT scanner using a dual-syringe. Data on the operators’ time required in the patient analysis steps as well as on the quantity of consumable materials used were collected. The radiologic technologists’ satisfaction with the use of the PIs was rated on a 10-point scale. A budget impact analysis and sensitivity analysis were performed under the base-case scenario.ResultsA total of 1,040 patients were examined using the syringeless system, and 1,339 with the dual-syringe system; the CECT examination quality was comparable for both PI systems. Equipment preparation time and releasing time per examination for syringeless PIs versus dual-syringe PIs were 100±30 versus 180±30 seconds and 90±30 and 140±20 seconds, respectively. On average, 10±3 mL of contrast media (CM) wastage per examination was observed with the dual-syringe PI and 0±1 mL with the syringeless PI. Technologists had higher satisfaction with the syringeless PI than with the dual-syringe system (8.8 versus 8.0). The syringeless PI allows a saving of about €6.18 per patient, both due to the lower cost of the devices and to the better performance of the syringeless system. The univariate sensitivity analysis carried out on the base-case results within the standard deviation range confirmed the saving generated by using the syringeless device, with saving values between €5.40 and €6.20 per patient.ConclusionThe syringeless PI was found to be more user-friendly and efficient, minimizing contrast wastage and providing similar contrast enhancement quality compared to the dual-syringe injector, with comparable CECT examination quality.

Evaluation of User Interface and Workflow Design of a Bedside Nursing Clinical Decision Support System

BackgroundClinical decision support systems (CDSS) are important tools to improve health care outcomes and reduce preventable medical adverse events. However, the effectiveness and success of CDSS depend on their implementation context and usability in complex health care settings. As a result, usability design and validation, especially in real world clinical settings, are crucial aspects of successful CDSS implementations.ObjectiveOur objective was to develop a novel CDSS to help frontline nurses better manage critical symptom changes in hospitalized patients, hence reducing preventable failure to rescue cases. A robust user interface and implementation strategy that fit into existing workflows was key for the success of the CDSS.MethodsGuided by a formal usability evaluation framework, UFuRT (user, function, representation, and task analysis), we developed a high-level specification of the product that captures key usability requirements and is flexible to implement. We interviewed users of the proposed CDSS to identify requirements, listed functions, and operations the system must perform. We then designed visual and workflow representations of the product to perform the operations. The user interface and workflow design were evaluated via heuristic and end user performance evaluation. The heuristic evaluation was done after the first prototype, and its results were incorporated into the product before the end user evaluation was conducted. First, we recruited 4 evaluators with strong domain expertise to study the initial prototype. Heuristic violations were coded and rated for severity. Second, after development of the system, we assembled a panel of nurses, consisting of 3 licensed vocational nurses and 7 registered nurses, to evaluate the user interface and workflow via simulated use cases. We recorded whether each session was successfully completed and its completion time. Each nurse was asked to use the National Aeronautics and Space Administration (NASA) Task Load Index to self-evaluate the amount of cognitive and physical burden associated with using the device.ResultsA total of 83 heuristic violations were identified in the studies. The distribution of the heuristic violations and their average severity are reported. The nurse evaluators successfully completed all 30 sessions of the performance evaluations. All nurses were able to use the device after a single training session. On average, the nurses took 111 seconds (SD 30 seconds) to complete the simulated task. The NASA Task Load Index results indicated that the work overhead on the nurses was low. In fact, most of the burden measures were consistent with zero. The only potentially significant burden was temporal demand, which was consistent with the primary use case of the tool.ConclusionsThe evaluation has shown that our design was functional and met the requirements demanded by the nurses’ tight schedules and heavy workloads. The user interface embedded in the tool provided compelling utility to the nurse with minimal distraction.

A Model-Based Framework for an Adaptive User Interface in Supervisory Control

Supervisory control of multiple Remotely Piloted Aircrafts (RPAs) involves complex human system coordination. Automation is often utilized to deal with these complexities, and, used properly, is a useful tool for improving operator performance. Using the Vigilant Spirit Control Station simulation, a methodology is proposed for developing a framework for adaptive automation by determining what should be automated for a task, identifying what levels of automation are in the current design for each generic automation tasks, and planning for user performance evaluation testing. This paper introduces a new perspective for integrating adaptive user interface development. Automation in HCI systems is a field of research in aviation, production systems, and other human factors design fields that is complex and not clear on how to use automation to improve performance. A task-by-task evaluation is proposed for recommending a level of automation in a HCI system.

Visualizing Qur'anic Words with Parallel Plot

Parallel plot is a relatively new type of plot in comparison to scatter plot. The aim of this paper is to find out the effectiveness of parallel plot visualization in comparison to the scatter plot for novice users. Comparison through literature shows that parallel and scatter plot visualization techniques offer similar visualization features. A study was conducted based on the parallel plot in Parallel Coordinate for the Qur’an (PaCQ) interface using a relational model, Goals Operators Methods and Selection Rules (GOMS) analysis and user performance evaluation. Based on the GOMS analysis, the subtasks and operation time involved in exploring data in scatter plot exceeds the subtasks and operation time in parallel plot in relation to the PaCQ interface developed for novice user even though, earlier studies found that scatter plot is easier to use. The GOMS analysis shows that the average time taken to complete tasks for the parallel plot is shorter then the scatter plot. A user performance evaluation shows no significant difference in the scores of interpreting the parallel plot within three levels of educational level of novice users. These findings provide evidences that the parallel plot is more effective for PaCQ designed for novice users.

Learning gaze typing: what are the obstacles and what progress to expect?

Gaze interaction is a promising input modality for people who are unable to control their fingers and arms. This paper suggests a number of new metrics that can be applied to the analysis of gaze typing interfaces and to the evaluation of user performance. These metrics are derived from a close examination of eight subjects typing text by gazing on a dwell-time activated onscreen keyboard during a seven-day experiment. One of the metrics, termed “Attended keys per character”, measures the number of keys that are attended for each typed character. This metric turned out to be particularly well correlated to the actual numbers of errors committed (r = 0.915). In addition to introducing metrics specific for gaze typing, the paper discusses how the metrics could make remote progress monitoring possible and provides some general advice on how to introduce gaze typing for novice users.

An evaluation of integrated zooming and scrolling on small screens

Speed-dependent automatic zooming (SDAZ) has been proposed for standard desktop displays as a means of overcoming problems associated with the navigation of large information spaces. SDAZ combines zooming and panning facilities into a single operation, with the magnitude of both factors dependent on simple user interaction. Previous research indicated dramatic user performance improvements when using the technique for document and map navigation tasks. In this paper, we propose algorithmic extensions to the technique for application on small-screen devices and present a comparative experimental evaluation of user performance with the system and a normative scroll–zoom–pan interface. Users responded positively to the system, particularly in relation to reduced physical navigational workload. However, the reduced screen space reduced the impact of SDAZ in comparison to that reported in previous studies. In fact, for one-dimensional navigation (vertical document navigation) the normative interface out-performed SDAZ. For navigation in two dimensions (map browsing) SDAZ supports more accurate target location, and also produces longer task completion times. Some SDAZ users became lost within the information space and were unable to recover navigational context. We discuss the reasons for these observations and suggest ways in which limitations of SDAZ in the small-screen context may be overcome.

Experimental evaluation of user performance in a pursuit tracking task with multimodal feedback

In this paper we describe the results of experimental evaluation of user performance in a pursuit-tracking task with multimodal feedback. Our experimental results indicate that audio can significantly improve the accuracy of pursuit tracking. Experiments with 19 participants have shown that addition of acoustic modalities reduces the error during pursuit tracking for up to 19%. Moreover, experiments indicated the existence of perceptual boundaries of multimodal HCI for different scene complexity and target speeds. We have also shown that the most appealing paradigms are not the most effective ones, which necessitates a careful quantitative analysis of proposed multimodal HCI paradigms.