Identify Usability Problems Research Articles

Development of a Comprehensive Usability Testing and Analysis Framework for the Physical Interface Between Product and User

Objective: The present study developed a comprehensive usability testing and analysis framework based on a physical interface model of product and user and applied the proposed framework to usability testing of canister-type vacuum cleaner. Background: The development of a user-centered product design is important to satisfy customers who want to use the product with ease of use and to keep the manufacturer competitive in the market. Method: The proposed testing and analysis framework consists of (1) characterization of physical product-user interface, (2) preparation and administration of usability testing questionnaire, and (3) analysis and interpretation of usability testing results. A usability evaluation of five vacuum cleaners was planned and administered based on the proposed framework and its analysis produced detailed and overall usability testing results for various aspects such as tasks, usability criteria, and design components. Results: The testing results were further utilized to identify usability problems and preferred design features of the vacuum cleaners. Conclusion: The proposed usability testing and analysis framework was found effective to identify preferred features and problems of a product design in a systematic, holistic manner. Application: The proposed framework can be of effective use for practitioners of product design and development to obtain comprehensive, quantitative usability testing information in a systematic manner.

Determination of the effectiveness of two methods for usability evaluation using a CPOE medication ordering system

ObjectivesTo assess the effectiveness of two usability evaluation methods, cognitive walkthrough (CW) and think aloud (TA), for identifying usability problems and to compare the performance of CW and TA in identifying different types of usability problems. MethodsA CW was performed by two usability evaluators and 10 physicians were recruited to perform a TA usability testing of a CPOE system (Medicator). The severity of identified usability problems was determined and the usability problems were categorized based on the User Action Framework (UAF). The potential of usability problems to cause medication errors was also determined. The thoroughness, validity and effectiveness of the two methods were compared. ResultsFifty seven unique usability problems of different severity, spread over the four phases of interaction as defined by the UAF, were identified. The effectiveness of the TA method for identifying usability problems was 0.08 higher than that of the CW (0.70 vs. 0.62). The thoroughness (the extent to which a method can identify existing usability problems) of the TA was higher for the “Planning” and “Assessment” phases and lower for the “Translation” phase (as defined by UAF). The thoroughness of TA for identifying problems that may potentially result in medication errors was higher than that of CW (0.81 vs. 0.68). The number of usability problems identified by each of the methods was significantly less than the total number of detected real usability problems in Medicator (p<0.001). The observed differences between the number of real usability problems identified by CW and TA (38 vs. 41), the difference between the average severity of the detected problems by CW and TA (2.37 vs. 2.41), and the difference for identifying problems potentially resulting in medication errors (15 vs. 18) were not statistically significant (p>0.4). ConclusionsThis study shows that although TA showed a slightly better effectiveness, there is no significant difference between the performance of the CW and the TA methods in terms of number of usability problems identified and the mean severity of these problems. Since no single evaluation method will uncover all of the usability problems a combination of methods is advised as the most appropriate approach, especially if usability problems can lead to potentially fatal outcomes.

An Improved Usability Measure Based on Novice and Expert Performance

The novice–expert ratio method (NEM) pinpoints user interface design problems by identifying the steps in a task that have a high ratio of novice to expert completion time. This study tested the construct validity of NEM's ratio measure against common alternatives. Data were collected from 337 participants who separately performed 10 word-completion tasks on a cellular phone interface. The logarithm, ratio, Cohen's d, and Hedges's ĝ measures had similar construct validity, but Hedges's ĝ provided the most accurate measure of effect size. All these measures correlated more strongly with self-reported interface usability and interface knowledge when applied to the number of actions required to complete a task than when applied to task completion time. A weighted average of both measures had the highest correlation. The relatively high correlation between self-reported interface usability and a weighted Hedges's ĝ measure as compared to the correlations found in the literature indicates the usefulness of the weighted Hedges's ĝ measure in identifying usability problems.

Heuristic Evaluation and Usability Testing of a Computerized Patient-Reported Outcomes Survey for Headache Sufferers

The aim of this study was to evaluate usability of a prototype tablet PC-administered computerized adaptive test (CAT) of headache impact and patient feedback report, referred to as HEADACHE-CAT. Heuristic evaluation specialists (n = 2) formed a consensus opinion on the application's strengths and areas for improvement based on general usability principles and human factors research. Usability testing involved structured interviews with headache sufferers (n = 9) to assess how they interacted with and navigated through the application, and to gather input on the survey and report interface, content, visual design, navigation, instructions, and user preferences. Specialists identified the need for improved instructions and text formatting, increased font size, page setup that avoids scrolling, and simplified presentation of feedback reports. Participants found the tool useful, and indicated a willingness to complete it again and recommend it to their healthcare provider. However, some had difficulty using the onscreen keyboard and autoadvance option; understanding the difference between generic and headache-specific questions; and interpreting score reports. Heuristic evaluation and user testing can help identify usability problems in the early stages of application development, and improve the construct validity of electronic assessments such as the HEADACHE-CAT. An improved computerized HEADACHE-CAT measure can offer headache sufferers an efficient tool to increase patient self-awareness, monitor headaches over time, aid patient-provider communications, and improve quality of life.

Usability Evaluation of An Electronic Medication Administration Record (eMAR) Application

Electronic medication administration records (eMARs) have been widely used in recent years. However, formal usability evaluations are not yet available for these vendor applications, especially from the perspective of nurses, the largest group of eMAR users. To conduct a formal usability evaluation of an implemented eMAR. Four evaluators examined a commercial vendor eMAR using heuristic evaluation techniques. The evaluators defined seven tasks typical of eMAR use and independently evaluated the application. Consensus techniques were used to obtain 100% agreement of identified usability problems and severity ratings. Findings were reviewed with 5 clinical staff nurses and the Director of Clinical Informatics who verified findings with a small group of clinical nurses. Evaluators found 60 usability problems categorized into 233 heuristic violations. Match, Error, and Visibility heuristics were the most frequently violated. Administer Medication and Order and Modify Medications tasks had the highest number of heuristic violations and usability problems rated as major or catastrophic. The high number of usability problems could impact the effectiveness, efficiency and satisfaction of nurses' medication administration activities and may include concerns about patient safety. Usability is a joint responsibility between sites and vendors. We offer a call to action for usability evaluations at all sites and eMAR application redesign as necessary to improve the user experience and promote patient safety.

A tool for rapid assessment of product usability and universal design: Development and preliminary psychometric testing

While there are many available tools and methods to evaluate product usability, few have been tested on user groups with disabilities and even fewer systematically consider universal design principles. This paper describes the development and preliminary psychometric testing of the Rapid Assessment of Product Usability & Universal Design (RAPUUD), a 12-item user-report tool based on the seven principles of universal design. A preliminary set of items was created to elicit ratings of diverse product characteristics (e.g., physical effort, cognitive effort, assistance required, safety). Data were gathered from 61 participants who rated the usability of products they use in their own environments. Each item elicited a full range of responses, with no apparent floor or ceiling effects. Collectively, the 12 items achieved a high internal consistency (Cronbach's α=0.80). The data indicate that the tool was sensitive to differences in functional abilities, as well as differences in product characteristics. The instrument was usable for a range of consumer products, though not all items were appropriate for each and every product. The results suggest that the instrument could become a pragmatic tool for designers to identify usability problems experienced by a diversity of user populations.

Twelve Tips on usability testing to develop effective e-learning in medical education

Usability testing is widely used in the commercial world during the process of developing new products, especially software and websites. However, it appears to be rarely used in the development of e-learning in medical education. The focus of usability testing is the user of the particular product and it informs product development by using a systematic process to identify usability problems at an early stage during product development so that these problems can be rectified. Usability testing of e-learning considers the characteristics of the learner, the technological aspects, the interaction and instructional design and finally the context. Testing under the conditions that the e-learning intervention will typically be used is the preferred method but more extreme situations can provide useful information. Product development should be iterative and rapid cycles of testing and refinement are essential to produce an effective e-learning intervention.

Web usability evaluation with screen reader users: implementation of the partial concurrent thinking aloud technique

A verbal protocol technique, adopted for a web usability evaluation, requires that the users are able to perform a double task: surfing and talking. Nevertheless, when blind users surf by using a screen reader and talk about the way they interact with the computer, the evaluation is influenced by a structural interference: users are forced to think aloud and listen to the screen reader at the same time. The aim of this study is to build up a verbal protocol technique for samples of visual impaired users in order to overcome the limits of concurrent and retrospective protocols. The technique we improved, called partial concurrent thinking aloud (PCTA), integrates a modified set of concurrent verbalization and retrospective analysis. One group of 6 blind users and another group of 6 sighted users evaluated the usability of a website using PCTA. By estimating the number of necessary users by the means of an asymptotic test, it was found out that the two groups had an equivalent ability of identifying usability problems, both over 80%. The result suggests that PCTA, while respecting the properties of classic verbal protocols, also allows to overcome the structural interference and the limits of concurrent and retrospective protocols when used with screen reader users. In this way, PCTA reduces the efficiency difference of usability evaluation between blind and sighted users.

A Conceptual Tool for Usability Problem Identification in Website Development

Techniques and tools that enable website developers without formal training in human-computer interaction to conduct their own usability evaluations would radically advance the integration of usability engineering in website development. This paper presents experiences from usability evaluations conducted by developers in an empirical study of means to support non-experts in identifying usability problems. A group of software developers who were novices in usability engineering analyzed a usability test session with the task of identifying usability problems experienced by the user. When doing this, they employed a simple one-page conceptual tool that supports identification of usability problems. The non-experts were able to conduct a well-organized usability evaluation and identify a reasonable amount of usability problems with a performance that was comparable to usability experts.

Usability Problem Reports for Comparative Studies: Consistency and Inspectability

ABSTRACT This study explores issues of consistency and inspectability in usability test data analysis processes and reports. Problem reports resulting from usability tests performed by three professional usability labs in three different countries are compared. Each of the labs conducted a usability test on the same product, applying an agreed test protocol that was collaboratively developed by the labs. Each lab first analyzed their own findings as they always do in their regular professional practice. A few weeks later, they again analyzed their findings, but then everyone applied the same method (SlimDEVAN: a simplified version of DEVAN, a method developed for facilitating comparison of findings from usability tests in an academic setting). It was found that levels of agreement between labs did not improve when they all used SlimDEVAN, suggesting that there was inherent subjectivity in their analyses. It was found that consistency of single analyst teams varied considerably and that a method like SlimDEVAN can help in making the analysis process and findings more inspectable. Inspectability is helpful in comparative studies based on identified usability problems because it allows for tracing back findings to original observations as well as for laying bare the subjective parts of the data analysis.

Development of a Patient Monitor for Critical Care

Unexpected incidents are common in intensive care medicine. One of the means to detect, diagnose and treat these events is physiological displays, which show the vital signs of a patient. Current monitors in the Intensive Care Unit (ICU) provide numerical and waveform data. The present project focused on determining the problems and needs of ICU nurses for patient monitoring and on developing a display that addresses some of the problems. Semi-structured interviews with 26 ICU nurses identified usability problems of current patient monitors. Most nurses mentioned that current monitors make it difficult to access trends of vital signs and do not allow marking of events or interventions on the trend display. The results of the interviews resulted in a redesign of the patient display that is currently being evaluated. The results of the evaluation study will also be presented.

RWELS: A Remote Web Event Logging System

Event logs are an important data source for identifying usability problems in websites. We present a web-based client-server application, Remote Web Event Logging System (RWELS), for logging user-interface events generated in the Microsoft Internet Explorer during a user's interaction with the pages of a website. RWELS logs events without interfering with the user's interaction — no additional interaction is required on part of a user to enable logging. RWELS is configurable and allows user-centric event logging. A usability analyst can choose the set of events to be captured and the pages of the website to be logged for a particular user. The event logs are dispatched through HTTP to the server where they are stored as text files. Users are identified uniquely and the event logs are associated with the user sessions.

Event Patterns as Indicators of Usability Problems

Event logs collected automatically contain a wealth of information about the usability of an application's user interface. Manual analysis of the event logs to identify usability problems is, however, a time consuming process. There are certain generic event patterns that can be associated with usability problems and can act as indicators of usability problems. These patterns can be automatically detected in event logs. We survey the event pattern indicators that have been identified and reported earlier in the literature. We identify new event patterns that get generated when users face usability problems during web browsing. We report on an experimental study that establishes the connection between these patterns and usability problems. The ultimate goal of this research is to develop techniques that can discover these patterns in event logs and reduce the time and cost of usability evaluation of the user interface.

Teaching Usability Methods to Undergraduate Students: Can We Measure Changes in Experience?

Instructors at the Air Force Academy recently developed an undergraduate Human-Computer Interaction (HCI) laboratory to teach basic components of HCI concepts to undergraduate students. The objective was to develop a teaching laboratory so that students would gain the necessary experience to conduct usability evaluation on local projects. The purpose of this study was to examine the changes in a student's technique of identifying usability problems while using the HCI laboratory. Thirteen students in an undergraduate HCI course participated in the study. During a pre-and post-assessment, we collected several measures in order to quantify any changes experienced by the students as they logged usability problems. These measures included attention focus, number of problems identified, word count, and use of HCI terms in describing usability problems. Results showed that the metrics of number of usability problems identified and the use of HCI technical terms were particularly sensitive to changes over the semester.

User Centered Design and Testing Methods for the Development of the Menu System for Mobile Phones

User centered design and usability evaluation of mobile devices are becoming important activities in the development process of mobile products. This paper presents a case study and introduces different techniques to support the process of designing a menu system embedded into a mobile phone. These techniques facilitated a systematic user needs analysis in the early stages of the development life-cycle, to formulate a navigational structure for designing prototypes, to identify usability problems for iterative prototype design, and to develop design guidelines. The introduced techniques involve laddered grid methods to elicit users' mental models, focus groups, usability testing, and expert reviews. The advantages and disadvantages of each method used during the design project were summarized, and usability practitioners in the real environment of the mobile devices industry are advised to be selective the combination of items they incorporate to suit their purposes.

Applying Heuristic Evaluation to Improve the Usability of a Telemedicine System

The development of a telemedicine system should not only take advantage of technological advances but also pay close attention to users and the human issues involved. In this paper we examine the utility of heuristic evaluation in improving the usability of a digital emergency medical services (EMS) system equipped on an ambulance. The digital EMS system used advanced communication technologies to help remotely located trauma specialists gain access to patient data in real-time and direct life-saving measures in a timely fashion. To improve its usability, three experts inspected prototypes of the system according to 14 software usability heuristics. The analyses revealed information on the prevalence, severity, and nature of heuristic violations in the user interface design. The results were subsequently utilized to guide the iterative software design process. A comparison between two consecutive prototypes showed that the second design had only half as many usability violations as the first prototype and had considerable improvement in a number of usability heuristic categories. The validity of heuristic evaluation was examined in an ethnographic study of paramedics using a prototype of the system in their work environment. Users' task performances partially verified heuristic evaluation results. However, they also revealed problems that were not identified in heuristic evaluation but only became prominent during field observation. In conclusion, we argue that usability should be given high priority in the development of a telemedicine system, and that heuristic evaluation can be an effective and efficient way to identify usability problems in the early stage of software development.

Usability study of travel websites

A usability study was conducted to identify usability problems as well as recommendations for improvement for three travel sales websites. The study performed testing on twenty participants, between the ages of 19 and 65, recruited from the university campus consisting of students, faculty, and staff. The three websites tested were Expedia.com, Orbitz.com, and Travelocity.com. Each participant was given general instructions and a pre-survey to determine their demographics and level of Internet experience. The usability study tested participants on the task of finding the same itinerary on each travel website. The participant during testing was under observation of the experimenter that maintained an observation log. A post-survey along with a debriefing session was conducted to gather additional feedback. The average testing time for participants was 30 minutes. The results of this study are presented as well as a future research discussion consisting of the development of usability guidelines for designers of travel websites.

Validation of an Objective Behaviorally-Based Usability Method for Real-Time Assessments

Rapid increases in computer and data-link technology have resulted in revolutionary but complex displays that comprise integrated cockpits. Consequently, usability techniques must evolve to ensure that advances in information quantity do not compromise information quality, thereby compromising safety. The UserCHAT is a usability method designed to maximize efficient diagnosis of usability problems identified from a behaviorally-based perspective while minimizing time and resource limitations associated with typical assessments. The present study sought to validate the UserCHAT as a suitable methodology that provides accurate real-time identification of usability problems by evaluators with little usability training. Expert and Intermediate evaluators accurately recorded over 80% of first inefficient actions performed by users on benchmark tasks using a relatively complex MFD while Novices with minimal training with UserCHAT captured over 65%. These results suggest that the UserCHAT does not require extensive evaluator training to effectively record user performance and thereby identify usability problems.

Better discount evaluation: illustrating how critical parameters support heuristic creation

This paper describes a heuristic creation process based on the notion of critical parameters, and a comparison experiment that demonstrates the utility of heuristics created for a specific system class. We focus on two examples of using the newly created heuristics to illustrate the utility of the usability evaluation method, as well as to provide support for the creation process, and we report on successes and frustrations of two classes of users, novice evaluators and domain experts, who identified usability problems with the new heuristics. We argue that establishing critical parameters for other domains will support efforts in creating tailored evaluation tools.

The system reliability analyser tool

This paper describes the design and evaluation of a socio-technical design support system, the system reliability analyser (SRA). The tool is used to validate non-functional system requirements, such as system reliability. It employs a Bayesian belief network (BBN) model to assess system reliability (Pearl in Probabilistic reasoning in intelligent systems: networks of plausible information, 1988) based on a variety of high-level operational scenarios. The tool diagnoses problematic areas in future system models and assists in the identification of their causes. The evaluation of the tool demonstrated that it supported the task it was intended to do. The evaluation also identified usability problems in the current visualisations and illustrated their resolution.