Usability Evaluation Methods Research Articles

Empirical evaluation of augmented prototyping effectiveness

Augmented reality is a scientific field well known for more than twenty years. Although there is a huge number of projects that present promising results, the real usage of augmented reality applications for fulfilling common tasks is almost negligible. We believe that one of the principal reasons is insufficient usability of these applications. The situation is analogous to the desktop, mobile or cloud application development or even to the web pages design. The first phase of a technology adoption is the exploration of its potential. As soon as the technical problems are overcome and the technology is widely accepted, the usability is a principal issue. The usability is utmost important also from the business point of view. The cost of augmented reality implementation into the production process is substantial, therefore, the usability that is directly responsible for the implemented solution effectiveness must be appropriately tested. Consequently, the benefit of the implemented solution can be measured.This article briefly outlines common techniques used for usability evaluation. Discussed techniques were designed especially for evaluation of desktop applications, mobile solutions and web pages. In spite of this drawback, their application on augmented reality products is usually possible. Further, a review of existing augmented reality project evaluations is presented.Based on this review, a usability evaluation method for our augmented prototyping application is proposed. This method must overcome the fact that the design is a creative process. Therefore, it is not possible to take into account common criteria such as time consumption.

Use of a human factors approach to uncover informatics needs of nurses in documentation of care

PurposeThe success of health information technology implementations is often tied to the impact the technical system will have on the work of the clinicians using them. Considering the role of nurses in healthcare, it is shocking that there is a lack of evaluations of nursing information systems in the literature. Here we report on how a human factors approach can be used to address barriers and facilitators to use of the nursing information system (NIS). Human factors engineering (HFE) approaches provide the theoretical and methodological underpinning to address these socio-technical issues. MethodsThis study investigated the use of an NIS, a module of the electronic health record (EHR) previously implemented throughout the hospital system. The study took place in two hospitals (760 beds and 300 beds) within a three-hospital health system. Earlier in the year, the NIS was implemented throughout the health system. We applied a scenario-based evaluation technique in order to understand the barriers and facilitators to nurse use of the NIS as part of improving the healthcare delivery system. The scenarios were designed to have the nurses interact with the major components of the NIS. The research team developed the standardized scenarios to cover the major functions of the system. ResultsTwelve nurses completed the study and results show that documentation within the NIS was hindered by several aspects of the interface. This paper discusses the themes associated with the usability of the NIS interface analyzing them using usability heuristics. The team also identified facilitators to use and proposed avenues to support or enhance these facilitators. ConclusionsThis study examined the use of an NIS to standardize care and documentation in nursing. It used scenario-based usability testing, applying the “think-aloud” protocol technique to assess the use of the NIS in documenting patient care. This method of usability evaluation exposed an understanding of how nurses use the NIS and their perspective on the system. We hypothesize that this method will offer key insights into how the usability of the NIS not only impacts use but also informs redesign opportunities. In addition, this is one of the few rigorous studies of NIS and provides direction and recommendations for informaticians, developers and nurse decision makers.

Digital video analysis of health professionals’ interactions with an electronic whiteboard: A longitudinal, naturalistic study of changes to user interactions

As hospital departments continue to introduce electronic whiteboards in real clinical settings a range of human factor issues have emerged and it has become clear that there is a need for improved methods for designing and testing these systems. In this study, we employed a longitudinal and naturalistic method in the usability evaluation of an electronic whiteboard system. The goal of the evaluation was to explore the extent to which usability issues experienced by users change as they gain more experience with the system. In addition, the paper explores the use of a new approach to collection and analysis of continuous digital video recordings of naturalistic “live” user interactions. The method developed and employed in the study included recording the users’ interactions with system during actual use using screen-capturing software and analyzing these recordings for usability issues. In this paper we describe and discuss both the method and the results of the evaluation. We found that the electronic whiteboard system contains system-related usability issues that did not change over time as the clinicians collectively gained more experience with the system. Furthermore, we also found user-related issues that seemed to change as the users gained more experience and we discuss the underlying reasons for these changes. We also found that the method used in the study has certain advantages over traditional usability evaluation methods, including the ability to collect analyze live user data over time. However, challenges and drawbacks to using the method (including the time taken for analysis and logistical issues in doing live recordings) should be considered before utilizing a similar approach. In conclusion we summarize our findings and call for an increased focus on longitudinal and naturalistic evaluations of health information systems and encourage others to apply and refine the method utilized in this study.

A thermal preference scale for personalized comfort profile identification via participatory sensing

Thermal comfort is the main driving factor in determining heating, ventilation and air conditioning (HVAC) systems' operational settings. Current building management systems (BMS) operate according to conservatively defined and standardized settings. Many dynamic environmental and human related variables affect occupant thermal comfort in buildings, calling for flexibility in HVAC operations by integrating dynamic user preferences. This paper builds on the vision that a participatory sensing based approach could be used for integrating user thermal comfort preferences into the operational logic of HVAC systems on a real time and continuous basis. The paper emphasizes on participatory sensing requirements for improving human data acquisition by proposing a thermal comfort preference scale. Several guiding features, which could potentially improve the users' expressions of thermal comfort preferences, were evaluated. Interface usability evaluation methods, including think aloud usability assessment studies and task execution surveys, were used in order to assess the proposed alternatives. Moreover, a field experiment was conducted to explore the driving factors, which affect users' thermal preference vote, for determining HVAC control parameters. The results showed that the proposed sensation scale outperforms a five-level ASHRAE like scale and the addition of the guiding features helps increase the consistency between users' expectations and their votes. The results of the field study showed that the ambient temperature is the most effective factor on users' votes. Consequently, HVAC set points for temperature could be used as a control parameter for personalized thermal comfort without any need for retrofitting HVAC system components.

A machine learning-based usability evaluation method for eLearning systems

The research presented in this paper proposes a new machine learning-based evaluation method for assessing the usability of eLearning systems. Three machine learning methods (support vector machines, neural networks and decision trees) along with multiple linear regression are used to develop prediction models in order to discover the underlying relationship between the overall eLearning system usability and its predictor factors. A subsequent sensitivity analysis is conducted to determine the rank-order importance of the predictors. Using both sensitivity values along with the usability scores, a metric (called severity index) is devised. By applying a Pareto-like analysis, the severity index values are ranked and the most important usability characteristics are identified. The case study results show that the proposed methodology enhances the determination of eLearning system problems by identifying the most pertinent usability factors. The proposed method could provide an invaluable guidance to the usability experts as to what measures should be improved in order to maximize the system usability for a targeted group of end-users of an eLearning system.

Characterizing Usability Inspection Methods through the Analysis of a Systematic Mapping Study Extension


 
 
 Usability is one of the most relevant quality aspects in Web applications. A Web application is usable if it provides a friendly, direct and easy to understand interface. Many Usability Inspection Methods (UIMs) have been proposed as a cost effective way to enhance usability. However, many companies are not aware of these UIMs and consequently, are not using them. A secondary study can identify, evaluate and interpret all data that is relevant to the current knowledge available regarding UIMs that have been used to evaluate Web applications in the past few decades. Therefore, we have extended a systematic mapping study about Usability Evaluation Methods by analyzing 26 of its research papers from which we extracted and categorized UIMs. We provide practitioners and researches with the rationale to understand both the strengths and weaknesses of the emerging UIMs for the Web. Furthermore, we have summarized the relevant information of the UIMs, which suggested new ideas or theoretical basis regarding usability inspection in the Web domain. In addition, we present a new UIM and a tool for Web usability inspection starting from the results shown in this paper.
 
 

Improving Usability of Social Networking Systems: A Case Study of LinkedIn

Several different methods are used for assessing the usability of websites. The ‘usability’ of a website describes how competently and easily a user or visitor without any proper training can interact with the site. Nowadays, social networking systems have become a major medium for online communication, marketing, research and collaboration. Users can quickly be in touch with friends and colleagues via a social networking site. The aim of this paper is to investigate the usability of social networking systems, using expert evaluation and user testing, in order to provide recommendations for developers in the improvement of social networking in general and LinkedIn in particular, based on the difficulties (usability problems) experienced by users and experts, and also to measure the users’ performance (both novices and experienced users). In this experiment, the researchers employed two usability evaluation methods: ‘user testing’ and ‘expert evaluation’. The ‘user testing’ includes pre-test, tasks to be performed and post-test usability ratings. The ‘user testing’ is conducted using a thinkaloud protocol and observation to achieve a better understanding of the participants’ behaviours. The ‘expert evaluation’ includes pre-evaluation, actual evaluation (each of the ten Nielson’s heuristics was broken down into a set of principles forming a checklist specifically oriented for the LinkedIn website), followed by post-evaluation.

Building mobile experiences

The mobile device is changing the ways we interact with each other and with the world. The mobile experience is distinct from the desktop or laptop experience; mobile apps require a significantly different design philosophy as well as design methods that reflect the unique experience of computing in the world. This book presents an approach to designing mobile media that takes advantage of the Internet-connected, context-aware, and media-sharing capabilities of mobile devices. It introduces tools that can be used at every stage of building a mobile application, from concept creation to commercialization, as well as real-world examples from industry and academia. The methods outlined apply user-centered design processes to mobile devices in a way that makes these methods relevant to the mobile experience--which involves the use of systems in the complex spatial and social world rather than at a desk. The book shows how each project begins with generative research into the practices and desires of a diverse set of potential users, which grounds research and design in the real world. It then describes methods for rapid prototyping, usability evaluation, field testing, and scaling up solutions in order to bring a product to market. Building Mobile Experiences grew out of an MIT course in communicating with mobile technology; it is appropriate for classroom use and as a reference for mobile app designers.

Must evaluation methods be about usability? Devising and assessing the utility inspection method

Whereas research in usability evaluation abounds, few evaluation approaches focus on utility. We present the utility inspection method (UIM), which prompts evaluators about the utility of the system they evaluate. The UIM asks whether a system uses global platforms, provides support infrastructure, is robust, gives access to rich content, allows customisation, offers symbolic value and supports companionship among users and between users and developers. We compare 47 participants’ use of UIM and heuristic evaluation (HE). The UIM helps identify more than three times as many problems as HE about the context of activities; HE helps identify 2.5 times as many problems as UIM about the interface. Usability experts consider the problems found with UIM more severe and more complex to solve compared to those found with HE. We argue that UIM complements existing usability evaluation methods and discuss future research on utility inspection.

Generating a Domain Specific Inspection Evaluation Method through an Adaptive Framework

The electronic information revolution and the use of computers as an essential part of everyday life are now more widespread than ever before, as the Internet is exploited for the speedy transfer of data and business. Social networking sites (SNSs), such as LinkedIn, Ecademy and Google+ are growing in use worldwide, and they present popular business channels on the Internet. However, they need to be continuously evaluated and monitored to measure their levels of efficiency, effectiveness and user satisfaction, ultimately to improve quality. Nearly all previous studies have used Heuristic Evaluation (HE) and User Testing (UT) methodologies, which have become the accepted methods for the usability evaluation of User Interface Design (UID); however, the former is general, and unlikely to encompass all usability attributes for all website domains. The latter is expensive, time consuming and misses consistency problems. To address this need, a new evaluation method is developed using traditional evaluations (HE and UT) in novel ways. The lack of an adaptive methodological framework that can be used to generate a domain- specific evaluation method, which can then be used to improve the usability assessment process for a product in any chosen domain, represents a missing area in usability testing. This paper proposes an adaptive framework that is readily capable of adaptation to any domain, and then evaluates it by generating an evaluation method for assessing and improving the usability of products in a particular domain. The evaluation method is called Domain Specific Inspection (DSI), and it is empirically, analytically and statistically tested by applying it on three websites in the social networks domain. Our experiments show that the adaptive framework is able to build a formative and summative evaluation method that provides optimal results with regard to our newly identified set of comprehensive usability problem areas as well as relevant usability evaluation method (UEM) metrics, with minimum input in terms of the cost and time usually spent on employing traditional usability evaluation methods (UEMs).

The Evaluation Method of Software Usability based on UI

Viewpoint about the quality of software is various. But, from the end-user point of view, user interface of software may be all to express the quality of software. But the detailed evaluation criteria were not established about usability evaluation method based on user interface so far, though user interface of software is the main object of usability quality characteristics. In this paper, we established a system about the method extracting and evaluating some quality characteristics elements measurable form user interface of software by analyzing the relationship between the elements constructing quality characteristics and user interface related to quality evaluation of software. We expect that this result of study will be a fundamental study adapting and evaluating to various types of user interface.

A Framework for Generating Domain-Specific Heuristics for Evaluating Online Educational Websites

Websites are growing in use worldwide but need to bcontinuously evaluated and monitored to measure their efficiency, effectiveness and user sa tisfaction, and ultimately to improve quality. For this purpose, heuristic evaluation methodologies, such as Nielsen 's Heuristics, have become the accepted means for t he usability evaluation of user interface designs; how ever, they are general, and unlikely to encompass a ll usability attributes for all website domains. The aim of this paper is to enhance one of the most-used usability evaluation methods by generating specific heuristics for the e ducational domain, and then to compare and contrast them against Nielsen's ten heuristics (as first validation stage for proposed framework) in terms of the nu mber and severity of problems found, and of a number of usab ility measurements. The result show that the propos ed framework succeeded in building a new set of heuris tics for online educational websites, which managed to discover uniquely 55 (69%) of the usability problem s in all chosen websites (80 problems in total), in comparison with Nielsen's heuristics, which discove red only 6 (8%). 19 problems (24%) were commonly discovered (overlapping or sharing) by both sets of heuristics. The time taken using Nielsen's heurist ics was less than the time taken using the newly developed Educa tional Heuristics but this is because Nielsen's heu ristics do not cover all the issues related to educational web sites. It appears that the framework for generating context- specific heuristics did in fact produce an efficaci ous set of Educational Heuristics that covered the issues well in this domain.

A Review : Healthcare Usability Evaluation Methods

<i>A Review</i> : Healthcare Usability Evaluation Methods

Usability Study of a Computer-Based Self-Management System for Older Adults with Chronic Diseases

BackgroundUsability can influence patients’ acceptance and adoption of a health information technology. However, little research has been conducted to study the usability of a self-management health care system, especially one geared toward elderly patients.ObjectiveThis usability study evaluated a new computer-based self-management system interface for older adults with chronic diseases, using a paper prototype approach.MethodsFifty older adults with different chronic diseases participated. Two usability evaluation methods were involved: (1) a heuristics evaluation and (2) end-user testing with a think-aloud testing method, audio recording, videotaping, and interviewing. A set of usability metrics was employed to determine the overall system usability, including task incompletion rate, task completion time, frequency of error, frequency of help, satisfaction, perceived usefulness, and perceived ease of use. Interviews were used to elicit participants’ comments on the system design. The quantitative data were analyzed using descriptive statistics and the qualitative data were analyzed for content.ResultsThe participants were able to perform the predesigned self-management tasks with the current system design and they expressed mostly positive responses about the perceived usability measures regarding the system interface. However, the heuristics evaluation, performance measures, and interviews revealed a number of usability problems related to system navigation, information search and interpretation, information presentation, and readability. Design recommendations for further system interface modifications were discussed.ConclusionsThis study verified the usability of the self-management system developed for older adults with chronic diseases. Also, we demonstrated that our usability evaluation approach could be used to quickly and effectively identify usability problems in a health care information system at an early stage of the system development process using a paper prototype. Conducting a usability evaluation is an essential step in system development to ensure that the system features match the users’ true needs, expectations, and characteristics, and also to minimize the likelihood of the users committing user errors and having difficulties using the system.

Sirius: A heuristic-based framework for measuring web usability adapted to the type of website

The unquestionable relevance of the web in our society has led to an enormous growth of websites offering all kinds of services to users. In this context, while usability is crucial in the development of successful websites, many barely consider the recommendations of experts in order to build usable designs. Including the measurement of usability as part of the development process stands out among these recommendations. One of the most accepted methods for usability evaluation by experts is heuristic evaluation. There is abundant literature on this method. However, there is a lack of clear and specific guidelines to be used in the development and evaluation process. This is probably an important factor contributing to the aforementioned generalized deficiency in web usability.We miss an evaluation method based on heuristics whose measure is adapted to the type of evaluated website. In this paper we define Sirius, an evaluation framework based on heuristics to perform expert evaluations that takes into account different types of websites. We also provide a specific set of evaluation criteria, and a usability metric that quantifies the usability level achieved by a website depending on its type.

A Glowworm Optimization Method for the Design of Web Services

A method for adaptive usability evaluation of B2C eCommerce web services is proposed. For measuring eCommerce usability a checklist integrating eCommerce quality and usability is developed. By a Glowworm swarm optimization (GSO) neural networks-based model the usability dimensions and their checklist items are adaptively selected. A case study for usability evaluation of an eCommerce anthurium retail website is carried out. The experimental results show that GSO with neural networks supports the allocation of usability problems and the defining of relevant improvement measures. The main advantage of the approach is the adaptive selection of most significant checklist dimensions and items and thus significant reduction of the time for usability evaluation and design.

Empirical validation of a usability inspection method for model-driven Web development

Web applications should be usable in order to be accepted by users and to improve their success probability. Despite the fact that this requirement has promoted the emergence of several usability evaluation methods, there is a need for empirically validated methods that provide evidence about their effectiveness and that can be properly integrated into early stages of Web development processes. Model-driven Web development processes have grown in popularity over the last few years, and offer a suitable context in which to perform early usability evaluations due to their intrinsic traceability mechanisms. These issues have motivated us to propose a Web Usability Evaluation Process (WUEP) which can be integrated into model-driven Web development processes. This paper presents a family of experiments that we have carried out to empirically validate WUEP. The family of experiments was carried out by 64 participants, including PhD and Master's computer science students. The objective of the experiments was to evaluate the participants’ effectiveness, efficiency, perceived ease of use and perceived satisfaction when using WUEP in comparison to an industrial widely used inspection method: Heuristic Evaluation (HE). The statistical analysis and meta-analysis of the data obtained separately from each experiment indicated that WUEP is more effective and efficient than HE in the detection of usability problems. The evaluators were also more satisfied when applying WUEP, and found it easier to use than HE. Although further experiments must be carried out to strengthen these results, WUEP has proved to be a promising usability inspection method for Web applications which have been developed by using model-driven development processes.

A comparison of usability evaluation methods for evaluating e-commerce websites

The importance of evaluating the usability of e-commerce websites is well recognised. User testing and heuristic evaluation methods are commonly used to evaluate the usability of such sites, but just how effective are these for identifying specific problems? This article describes an evaluation of these methods by comparing the number, severity and type of usability problems identified by each one. The cost of employing these methods is also considered. The findings highlight the number and severity level of 44 specific usability problem areas which were uniquely identified by either user testing or heuristic evaluation methods, common problems that were identified by both methods, and problems that were missed by each method. The results show that user testing uniquely identified major problems related to four specific areas and minor problems related to one area. Conversely, the heuristic evaluation uniquely identified minor problems in eight specific areas and major problems in three areas.

Dimensions of Concern

Producing useful and usable software often requires continuous and iterative evaluation. This paper introduces a novel usability evaluation method based on the Cognitive Dimensions of Notations framework. The target of our evaluation is Herbal a suite of tools designed to simplify agent development by providing a high-level language and maintenance-oriented development environment. The method introduced here uncovers dimensions of concern, which are used to measure the usability of Herbal and to identify areas for improvement in the design. In this article, we demonstrate how we used dimensions of concern to effectively evaluate and improve usability, and we discuss ways in which our method can be adapted, extended, and applied to improving the usability of other interactive systems.

Backtracking Events as Indicators of Usability Problems in Creation-Oriented Applications

A diversity of user goals and strategies make creation-oriented applications such as word processors or photo-editors difficult to comprehensively test. Evaluating such applications requires testing a large pool of participants to capture the diversity of experience, but traditional usability testing can be prohibitively expensive. To address this problem, this article contributes a new usability evaluation method called backtracking analysis, designed to automate the process of detecting and characterizing usability problems in creation-oriented applications. The key insight is that interaction breakdowns in creation-oriented applications often manifest themselves in backtracking operations that can be automatically logged (e.g., undo and erase operations). Backtracking analysis synchronizes these events to contextual data such as screen capture video, helping the evaluator to characterize specific usability problems. The results from three experiments demonstrate that backtracking events can be effective indicators of usability problems in creation-oriented applications, and can yield a cost-effective alternative to traditional laboratory usability testing.