Traditional Usability Evaluation Research Articles

Agile eHealth Usability Evaluation: A Triangulative Approach to Promoting the Usability of eHealth Systems.

Usability evaluation is difficult to reconcile with agile software development for eHealth systems, because traditional usability evaluation is often complex and cumbersome to implement. However, obtaining prospective users' feedback during agile software development is crucial for improving the usability of eHealth systems, which is why there is an increasing need for agile eHealth usability evaluation. This study investigates whether agile usability evaluations are suitable to evaluate patient-centered eHealth systems being agile developed in health care and are applicable for prospective users, such as older persons suffering from age-related declines. A triangulation study was conducted combining iterative expert interviews with an exploratory case study. The triangulation study revealed that the implementation of an agile eHealth usability evaluation with prospective users such as older persons proved to be possible. Established eHealth usability evaluation methods must be further evolved to address age-related impairments of older persons.

In-vehicle heat-induced impact on the system usability of on-board computer interfaces considering the optimization of scheme presentation order

Extreme heat is now a new normal for many cities due to climate change and urbanisation. The heat challenges have generated significant impacts on people's daily functions, including driving behaviours. In-vehicle temperature condition regulate drivers' thermal comfort, responses, and thereby intervening their cognition of on-board computer interface schemes (OBCIS). However, existing studies on the OBCIS usability evaluation have not well dealt with the impacts of ambient temperature and scheme presentation order on drivers' responses. This study addressed such research gaps by empirically analysing respondents' eye movement, including fixation duration (FD) and fixation times (FT) towards 24 typical interface prototypes under six temperature conditions. Moreover, the genetic algorithm (GA) and simulated annealing algorithm (SAA) were applied to optimise the total distance between the prototypes for optimal OBCIS presentation order. The results indicated that respondents' FD and FT generally increased with temperature. The checkerboard type OBCIS, which got the highest usability score, presented the lowest FT at 23 °C, while it exhibited the highest FT at 26 °C. Moreover, after the two optimisation algorithms had been run for 50 times respectively, paired sample t-test showed that GA performed better than SAA in the optimisation work, and thereby the optimal presentation order was determined. This study provides a reference for understanding heat-induced information interaction and OBCIS design. Compared with traditional usability evaluation methods of human-computer interface, this research properly ensures the reliability of the evaluation results in terms of ambient temperature and order effect bias.

Usability evaluation of mHealth apps for elderly individuals: a scoping review

BackgroundUsability is a key factor affecting the acceptance of mobile health applications (mHealth apps) for elderly individuals, but traditional usability evaluation methods may not be suitable for use in this population because of aging barriers. The objectives of this study were to identify, explore, and summarize the current state of the literature on the usability evaluation of mHealth apps for older adults and to incorporate these methods into the appropriate evaluation stage.MethodsElectronic searches were conducted in 10 databases. Inclusion criteria were articles focused on the usability evaluation of mHealth apps designed for older adults. The included studies were classified according to the mHealth app usability evaluation framework, and the suitability of evaluation methods for use among the elderly was analyzed.ResultsNinety-six articles met the inclusion criteria. Research activity increased steeply after 2013 (n = 92). Satisfaction (n = 74) and learnability (n = 60) were the most frequently evaluated critical measures, while memorability (n = 13) was the least evaluated. The ratios of satisfaction, learnability, operability, and understandability measures were significantly related to the different stages of evaluation (P < 0.05). The methods used for usability evaluation were questionnaire (n = 68), interview (n = 36), concurrent thinking aloud (n = 25), performance metrics (n = 25), behavioral observation log (n = 14), screen recording (n = 3), eye tracking (n = 1), retrospective thinking aloud (n = 1), and feedback log (n = 1). Thirty-two studies developed their own evaluation tool to assess unique design features for elderly individuals.ConclusionIn the past five years, the number of studies in the field of usability evaluation of mHealth apps for the elderly has increased rapidly. The mHealth apps are often used as an auxiliary means of self-management to help the elderly manage their wellness and disease. According to the three stages of the mHealth app usability evaluation framework, the critical measures and evaluation methods are inconsistent. Future research should focus on selecting specific critical measures relevant to aging characteristics and adapting usability evaluation methods to elderly individuals by improving traditional tools, introducing automated evaluation tools and optimizing evaluation processes.

Agile, Easily Applicable, and Useful eHealth Usability Evaluations: Systematic Review and Expert-Validation.

Background Electronic health (eHealth) usability evaluations of rapidly developed eHealth systems are difficult to accomplish because traditional usability evaluation methods require substantial time in preparation and implementation. This illustrates the growing need for fast, flexible, and cost-effective methods to evaluate the usability of eHealth systems. To address this demand, the present study systematically identified and expert-validated rapidly deployable eHealth usability evaluation methods. Objective Identification and prioritization of eHealth usability evaluation methods suitable for agile, easily applicable, and useful eHealth usability evaluations. Methods The study design comprised a systematic iterative approach in which expert knowledge was contrasted with findings from literature. Forty-three eHealth usability evaluation methods were systematically identified and assessed regarding their ease of applicability and usefulness through semi-structured expert interviews with 10 European usability experts and systematic literature research. The most appropriate eHealth usability evaluation methods were selected stepwise based on the experts' judgements of their ease of applicability and usefulness. Results Of these 43 eHealth usability evaluation methods identified as suitable for agile, easily applicable, and useful eHealth usability evaluations, 10 were recommended by the experts based on their usefulness for rapid eHealth usability evaluations. The three most frequently recommended eHealth usability evaluation methods were Remote User Testing, Expert Review, and Rapid Iterative Test and Evaluation Method. Eleven usability evaluation methods, such as Retrospective Testing, were not recommended for use in rapid eHealth usability evaluations. Conclusion We conducted a systematic review and expert-validation to identify rapidly deployable eHealth usability evaluation methods. The comprehensive and evidence-based prioritization of eHealth usability evaluation methods supports faster usability evaluations, and so contributes to the ease-of-use of emerging eHealth systems.

Usability inspection: Novice crowd inspectors versus expert

Objective:This research study aims to investigate the use of novice crowd inspectors for usability inspection with respect to time spent and the cost incurred. This study compares the results of the novice crowd usability inspection guided by a single expert’s heuristic usability inspection (novice crowd usability inspection henceforth) with the expert heuristic usability inspection. Background:Traditional usability evaluation methods are time-consuming and expensive. Crowdsourcing has emerged as a cost-effective and quick means of software usability evaluation. Method:In this regard, we designed an experiment to evaluate the usability of two websites and a web dashboard. Results:The results of the experiment show that novice crowd usability inspection guided by a single expert’s heuristic usability inspection:a) Finds the same usability issues (w.r.t. content & quantity) as expert heuristic usability inspection.b) Is cost-effective than expert heuristic usability inspection employing less time duration. Conclusion:Based on the findings of this research study, we can conclude that the novice crowd usability inspection guided by a single expert’s heuristic usability inspection and expert heuristic usability inspection, on average, gives the same results in terms of issues identified.

Applying Semantic Network Analysis to Develop User Experience Assessment Model for Smart TV

The user experience (UX) concept has widely been adopted in the field of development and evaluation of products and services. One reason behind this emerging trend is the limitation in traditional usability evaluation methods (UEMs), which emphasizes performance measures such as task completion time and error rate. To overcome the limitation, users’ perceived value must be set to reflect users’ various experiences. In order to address this issue, this study proposes a network analysis-based method to investigate a value structure to assess the level of user experience in using a product. Gesture interaction on a smart TV is used as a case study to demonstrate the process of obtaining a quantified value structure from the transcriptions of users’ in-depth interview. A major contribution of this study lies in the transformation from qualitative data to a quantified value structure. Product or service designers, in the industrial field, can develop value structure based on network analysis when evaluating their design alternatives within a properly fitted evaluation framework.

Using a fuzzy comprehensive evaluation method to determine product usability: A proposed theoretical framework.

BACKGROUND: In order to compare existing usability data to ideal goals or to that for other products, usability practitioners have tried to develop a framework for deriving an integrated metric. However, most current usability methods with this aim rely heavily on human judgment about the various attributes of a product, but often fail to take into account of the inherent uncertainties in these judgments in the evaluation process.OBJECTIVE: This paper presents a universal method of usability evaluation by combining the analytic hierarchical process (AHP) and the fuzzy evaluation method. By integrating multiple sources of uncertain information during product usability evaluation, the method proposed here aims to derive an index that is structured hierarchically in terms of the three usability components of effectiveness, efficiency, and user satisfaction of a product.METHODS: With consideration of the theoretical basis of fuzzy evaluation, a two-layer comprehensive evaluation index was first constructed. After the membership functions were determined by an expert panel, the evaluation appraisals were computed by using the fuzzy comprehensive evaluation technique model to characterize fuzzy human judgments. Then with the use of AHP, the weights of usability components were elicited from these experts.RESULTS AND CONCLUSIONS: Compared to traditional usability evaluation methods, the major strength of the fuzzy method is that it captures the fuzziness and uncertainties in human judgments and provides an integrated framework that combines the vague judgments from multiple stages of a product evaluation process.

Pilot evaluation of a method to assess prescribers’ information processing of medication alerts

BackgroundPrescribers commonly receive alerts during medication ordering. Prescribers work in a complex, time-pressured environment; to enhance the effectiveness of safety alerts, the effort needed to cognitively process these alerts should be minimized. Methods to evaluate the extent to which computerized alerts support prescribers’ information processing are lacking. ObjectiveTo develop a methodological protocol to assess the extent to which alerts support prescribers’ information processing at-a-glance; specifically, the incorporation of information into their working memory. We hypothesized that the method would be feasible and that we would be able to detect a significant difference in prescribers’ information processing with a revised alert display that incorporates warning design guidelines compared to the original alert display. MethodsA counterbalanced, within-subject study was conducted with 20 prescribers in a human-computer interaction laboratory. We tested a single alert that was displayed in two different ways. Prescribers were informed that an alert would appear for 10s. After the alert was shown, a white screen was displayed, and prescribers were asked to verbally describe what they saw; indicate how many total warnings; and describe anything else they remembered about the alert. We measured information processing via the accuracy of prescribers’ free recall and their ability to identify that three warning messages were present. Two analysts independently evaluated participants’ responses against a comprehensive catalog of alert elements and then discussed discrepancies until reaching consensus. ResultsThis feasibility study demonstrated that the method seemed to be effective for evaluating prescribers’ information processing of medication alert displays. With this method, we were able to detect significant differences in prescribers’ recall of alert information. The proportion of total data elements that prescribers were able to accurately recall was significantly greater for the revised versus original alert display (p=0.006). With the revised display, more prescribers accurately reported that three warnings were shown (p=0.002). ConclusionsThe methodological protocol was feasible for evaluating the alert display and yielded important findings on prescribers’ information processing. Study methods supplement traditional usability evaluation methods and may be useful for evaluating information processing of other healthcare technologies.

The usefulness of traditional usability evaluation methods

This forum addresses conceptual, methodological, and professional issues that arise in the UX field's continuing effort to contribute robust information about users to product planning and design. ---David Siegel and Susan Dray, Editors

Digital video analysis of health professionals’ interactions with an electronic whiteboard: A longitudinal, naturalistic study of changes to user interactions

As hospital departments continue to introduce electronic whiteboards in real clinical settings a range of human factor issues have emerged and it has become clear that there is a need for improved methods for designing and testing these systems. In this study, we employed a longitudinal and naturalistic method in the usability evaluation of an electronic whiteboard system. The goal of the evaluation was to explore the extent to which usability issues experienced by users change as they gain more experience with the system. In addition, the paper explores the use of a new approach to collection and analysis of continuous digital video recordings of naturalistic “live” user interactions. The method developed and employed in the study included recording the users’ interactions with system during actual use using screen-capturing software and analyzing these recordings for usability issues. In this paper we describe and discuss both the method and the results of the evaluation. We found that the electronic whiteboard system contains system-related usability issues that did not change over time as the clinicians collectively gained more experience with the system. Furthermore, we also found user-related issues that seemed to change as the users gained more experience and we discuss the underlying reasons for these changes. We also found that the method used in the study has certain advantages over traditional usability evaluation methods, including the ability to collect analyze live user data over time. However, challenges and drawbacks to using the method (including the time taken for analysis and logistical issues in doing live recordings) should be considered before utilizing a similar approach. In conclusion we summarize our findings and call for an increased focus on longitudinal and naturalistic evaluations of health information systems and encourage others to apply and refine the method utilized in this study.

Generating a Domain Specific Inspection Evaluation Method through an Adaptive Framework

The electronic information revolution and the use of computers as an essential part of everyday life are now more widespread than ever before, as the Internet is exploited for the speedy transfer of data and business. Social networking sites (SNSs), such as LinkedIn, Ecademy and Google+ are growing in use worldwide, and they present popular business channels on the Internet. However, they need to be continuously evaluated and monitored to measure their levels of efficiency, effectiveness and user satisfaction, ultimately to improve quality. Nearly all previous studies have used Heuristic Evaluation (HE) and User Testing (UT) methodologies, which have become the accepted methods for the usability evaluation of User Interface Design (UID); however, the former is general, and unlikely to encompass all usability attributes for all website domains. The latter is expensive, time consuming and misses consistency problems. To address this need, a new evaluation method is developed using traditional evaluations (HE and UT) in novel ways. The lack of an adaptive methodological framework that can be used to generate a domain- specific evaluation method, which can then be used to improve the usability assessment process for a product in any chosen domain, represents a missing area in usability testing. This paper proposes an adaptive framework that is readily capable of adaptation to any domain, and then evaluates it by generating an evaluation method for assessing and improving the usability of products in a particular domain. The evaluation method is called Domain Specific Inspection (DSI), and it is empirically, analytically and statistically tested by applying it on three websites in the social networks domain. Our experiments show that the adaptive framework is able to build a formative and summative evaluation method that provides optimal results with regard to our newly identified set of comprehensive usability problem areas as well as relevant usability evaluation method (UEM) metrics, with minimum input in terms of the cost and time usually spent on employing traditional usability evaluation methods (UEMs).

이용자 중심 전자책 뷰어 프로그램을 위한 휴리스틱 개발

전자책은 전용 단말기 기술의 향상으로 꾸준히 주목을 받고 있으나, 전자책의 효율성과 인지도에 비하여 상대적으로 이용률이 낮다. 이 연구에서는 전자책 이용 환경을 개선하고 이용자들의 필요와 요구를 만족시키기 위하여 먼저 전자책의 이용과 이용성 연구 설계에서 제안하고 있는 원칙과 기존 휴리스틱 연구 유형에 대한 문헌 조사를 실시하였다. 이에 근거하여 전자책 이용 경험자들을 대상으로 목적성 표집을 하였고 모집된 22명의 실험 참여자들은 휴리스틱 평가를 실시하였다. 본 연구의 실험을 통하여 지적된 이용성 문제와 요구된 이용 편리성을 모두 종합하여 최종 10개의 휴리스틱을 도출하여 제시하였다.

An Empirical Study of Relationships Between Traditional Usability

Task performance data and subjective assessment data are widely used as usability measures in the human-computer interaction (HCI) field. Recently, physiology has also been explored as a metric for evaluating usability. However, it is not clear how physiological measures relate to traditional usability evaluation indexes. In this paper, we investigate the relationships among three kinds of data: task performance, subjective assessment and physiological measures. We found evidence that physiological data correlate with task performance data in a video game: with a decrease of the task performance level, the normalized galvanic skin response (GSR) increases. In addition, physiological data are mirrored in subjective reports assessing stress level. The research provides an initial step toward using physiology as a complementary or an independent usability measure for HCI evaluation.