Usability Heuristic Evaluation Research Articles

Online Focus Groups for the Development of a Usability Evaluation Tool: Lessons Learned

Introduction: Focus group discussions are a well-established method for acquiring insights from experts in different fields. This method requires special amendments when it is used for the development of different tools in the healthcare domain. The objective of this paper is to present the lessons learned from online focus group sessions held for the development of a heuristic usability evaluation tool for mobile health applications.Material and Methods: Two online focus group sessions were conducted with the participation of ten medical informatics experts to develop the tool. The sessions were recorded using screen recording software. The comments provided by the experts were categorized, and the lessons learned from these sessions were identified and reported.Results: The experiences achieved from the online focus group sessions were categorized into the following ten lessons. 1. Engage the participants fully in online session discussions 2. Use an appropriate and interesting format. 3. Select an appropriate number of people for online sessions 4. Invite people having the closest expertise related to the research topic 5. Employ a technical support technician in addition to the coordinator. 6. Prevent the emergence of a new topic in sessions. 7. Arrange the required hardware and software facilities before the session. 8. Prepare the content in an appropriate language. 9. Use online tools to schedule sessions. 10. Use screen-recording software.Conclusion: This paper reports the lessons learned from holding online focus group sessions in the process of developing a heuristic usability evaluation tool for mobile health applications. Although these lessons were learned in a study focusing on the development of a usability tool, they can also be used to improve the results of focus group sessions in other fields of medical informatics.

Enhancing Ada Compliance for Websites and Online Applications: A Heuristic Evaluation Approach

The guidance provided by the ADA regarding online applications can be unclear and has led to the adoption of Web Content Accessibility Guidelines (WCAG) 2.0 and Section 508 of the Rehabilitation Act of 1973 as the standards for assessing online accessibility and achieving ADA compliance. While these standards are helpful, they are not perfect and have faced criticism due to usability concerns. As a result, our objective is to create review heuristics that combine the guidance of WCAG 2.0 and Section 508, which can serve as a practical and effective tool for assessing web ADA compliance. The heuristics function similar to a usability heuristic evaluation, which has been proven successful for decades. This article will first focus on developing the ADA compliance heuristics, then it demonstrates the use of the heuristics.

User experience evaluation of Edurell interface for video augmentation

AbstractThis paper deals with the use of videos in education and describes an approach to improve the learners’ experience in the first-watch and rewatch learning contexts. Actually, learning through videos raises challenging issues like non-linear navigation and poor structure of the video content. Hence, a novel application is presented, designed to address such issues. To this aim, the proposed Edurell environment enriches standard video lessons with new services based on the contents’ structure of the video, resulting in a new hypervideo educational application. The system extracts the concepts explained in a given video lesson, their prerequisite relations, and the segments of the video where they occur, together with other pieces of knowledge. The resulting knowledge graph allows creating augmented video lessons and enables learners to perform non-linear navigation and in-depth exploration of their learning contents. This idea was implemented in a custom web-based video player with interactive visual tools. Then, a heuristic usability evaluation was performed with HCI experts. Also, a user study with university students was done, aimed to evaluate the perceived usability and the student’s experience in the two mentioned learning contexts. The first results achieved show the effectiveness of the proposed approach.

Preliminary Usability Evaluation of Personal Medical Devices towards Better Home Healthcare

Portable medical devices have been playing an important role in improving health outcomes across the socio-economic spectrum. In the recent decades, substantial population have started using such devices to monitor important vital signs. Significant proportion of these users are older patients with chronic diseases, declined physical and/or cognitive capabilities. Furthermore, these portable devices are often used in unguided environments, that can present unpredictable use contexts. In this preliminary study, we conducted usability inspection on five portable medical devices. Cognitive walkthrough followed by Nielson’s ten Usability Heuristic evaluation method were utilized for the purpose of usability inspection. Sixteen usability concerns were uncovered in such evaluation study. We also recorded the elapsed time of measuring vital signs with all five devices on a daily basis in a month. The average total completion time after familiarization (after the first week) was 5.17 minutes. Results suggest that ‘non-expert’, independent users may face challenges when using portable medical devices in home care settings. Further, the study results also indicate that unsatisfactory usability of portable medical devices could pose hinderances in technology adoption among low eHealth literate users.

Research on the Usability Evaluation Model of Mobile Application Interface Design for Active Senior

Advances in information technology and the COVID-19 that has been sweeping the world since 2019 have affected people’s lives and changed their consumption patterns in various ways, and the non-face-to-face consumption culture that has emerged from them has led to an increasing national usage rate of mobile applications. However, due to the initial aging of their bodies, the active elderly encounter difficulties in using mobile applications, such as the inability to use them smoothly due to the complicated design process or the difficulty in recognizing the font size, especially since the current mobile application interface is mainly designed with the younger generation in mind. Therefore, it is necessary to improve the usability of their mobile applications in order to make them use mobile shopping applications more easily and efficiently. Accordingly, the purpose of this thesis is to compile and summarize the characteristics of the active elderly and their mobile application status through the literature research method through relevant data and prior studies, and to grasp the main barriers to their use of mobile shopping applications. It also summarizes the visual elements of mobile shopping application interface into five elements: color, layout, icon, typography, and navigation. Secondly, based on the expert heuristic usability evaluation principles, specific usability evaluation principles are determined for the usability evaluation of mobile interfaces by combining the characteristics of active seniors, so as to establish a usability evaluation model of interface design that is relevant to active senior.

Empirical Research on the Metaverse User Experience of Digital Natives

The metaverse has been settled as a platform that is widely beloved by digital natives that are familiar with mobile devices and immersive contents. Thanks to the protocol enabling hedonic interaction, the user experience provides significant value from its communication, enabling learning experiences anytime and anywhere. However, the research topics are focused on the promotions of technology development, marketing effects, and relevant investment consensus. Surprisingly, the biggest problem was the lack of research from the perspective of the young generation, who mainly use the metaverse. This paper intends to examine the usability of digital native participants in detail and suggest how immersive contents, usage environment, and interface aspects should be designed from their point of view. As a result, the significant engagement factors and improvements, through heuristic usability evaluation considering content and user control, were discovered from individual interviews. Conversely, the elements to be supplemented in user experience were derived from information architecture and usage environment categories. In conclusion, the theoretical basis of the empirical usability evaluation on metaverse platforms and following recommendations with practical implications could gain more importance from this research.

Towards a Set of Heuristics for Evaluating Chatbots

Chatbots are artificial intelligence tools that interact with people in different contexts. A chatbot can be useful to streamline daily processes, serve customers 24 hours a day, provide information about classes, among other things. The appearance of new development technologies has made creating a chatbot an increasingly fast and straightforward process, bringing this kind of applications to people who had never considered using them before. However, this speed in development can lead to specific problems, many of them caused by the lack of usability evaluations. Heuristic usability evaluations are user interface review processes carried out by experts and are an essential part of any assessment process. To date, there are no heuristics to evaluate the usability of chatbots. Therefore, this work proposes five usability heuristics in chatbots that come from the experience developing this type of applications, as well as from a broad review of state of the art. The set of heuristics was tested using a case study with the help of five experts, who evaluated an education-oriented chatbot. The results revealed that, although the proposed heuristics need refinement, they are an excellent first step in broadening the horizon of usability evaluations in chatbots.

Evaluating Aspects of Usability in Video Game-Based Programming Learning Platforms

Abstract Teaching computer programming is an important topic. Due to Science and Technology initiatives, these topics are considered in different training cycles. For higher education, students must cultivate fundamental concepts for the development of software applications, which not only contribute to the knowledge of programming languages but also to opening guidelines for computational thinking. However, selecting a proper tool can be complex. Especially for the diversity of alternatives on the web. Further, not all of them meet basic usability requirements. In this study, we present a set of platforms that seek to develop programming skills based on video games. The search consisted of 4 stages: (i) definition of the research questions, (ii) scope review, (iii) execution of search and (iv) platform selection. Finally, we employ a usability heuristic evaluation for a novice programming system to determine best practices.

A Heuristic Evaluation of Usability for Environmental Control Units’ Eye-Tracking Interfaces at Veterans Affairs Spinal Cord Injuries and Disorders Centers

Heuristic evaluation is a valid method for assessing interface usability. Results derived from these methods are useful in identifying the most impactful design elements influencing usability and user experience of a system. By establishing the features in need of most improvement, designers can more effortful consider their approaches for iterative efforts to improve usability and user experience. This work used established design principles (i.e., Shneiderman’s Golden Rules and Nielsen’s Heuristics) to analyze the inpatient environmental control units deployed at VA Spinal Cord Injuries and Disorders (SCI/D) Centers to assist with task completion, foster independence, and improve the quality of life in veterans with spinal cord injuries and disorders. Analyses revealed many usability issues in the current interface, varying in severity from minor to catastrophic. Specific design recommendations to address these issues are provided.

Heuristic Evaluation of the User Interface for a Semi-Autonomous Agricultural Robot Sprayer

This study presents the heuristic evaluation, as a usability inspection method for Human-Robot Interaction (HRI) systems. First, the methodology to engineer a semi-autonomous agricultural robot sprayer is presented, and then findings from heuristic usability evaluation studies that were carried out on a human-robot interface for a semi-autonomous agricultural vineyard robot sprayer. The following research-based heuristics for the design of robot teleoperation were used: Platform architecture and scalability, Error prevention and recovery, Visual design, Information presentation, Robot state awareness, Interaction effectiveness and efficiency, Robot environment/surroundings awareness, and Cognitive factors. In each evaluation study, usability problems were identified, and specific suggestions were documented for HRI usability improvement. In each design iteration, a smaller number of usability issues were identified.. Results of the final heuristic evaluation showed that the system is at a good level of usability and is expected to provide satisfactory services to its typical users.

Heuristic Evaluation of the User Interface for a Semi-Autonomous Agricultural Robot Sprayer

This study presents the heuristic evaluation, as a usability inspection method for Human-Robot Interaction (HRI) systems. First, the methodology to engineer a semi-autonomous agricultural robot sprayer is presented, and then findings from heuristic usability evaluation studies that were carried out on a human-robot interface for a semi-autonomous agricultural vineyard robot sprayer. The following research-based heuristics for the design of robot teleoperation were used: Platform architecture and scalability, Error prevention and recovery, Visual design, Information presentation, Robot state awareness, Interaction effectiveness and efficiency, Robot environment/surroundings awareness, and Cognitive factors. In each evaluation study, usability problems were identified, and specific suggestions were documented for HRI usability improvement. In each design iteration, a smaller number of usability issues were identified.. Results of the final heuristic evaluation showed that the system is at a good level of usability and is expected to provide satisfactory services to its typical users.

Web Service to Retrieve and Semantically Enrich Datasets for Theses From Open Educational Repositories

The paper describes the design and implementation of a semantic web service that retrieves theses and extends the keyword based-search of a DSpace repository taking into account the roles of advisors and steering committee members formally represented into a custom-made ontology. The service uses SPARQL queries and the serialization module of RDF DSpace, this links the item submission process and the ontology, thus the more theses are added into a repository, the more instances are inserted into the ontology. The paper provides empirical insights about how to reuse theses metadata and includes the results of an exploratory and self-management survey of usability heuristic evaluation of a web site that enables to access the proposed service. Heuristics were estimated with a purposive sample of students, teachers, and managers, the results indicated a high satisfaction level and showed that the service increased theses accessibility in the web environment. The service also generates semantically enriched datasets that coexist with the repository, they are of utility and value to educational organizations as they give institutional visibility.

Data Discovery Paradigms: User Requirements and Recommendations for Data Repositories

As data repositories make more data openly available it becomes challenging for researchers to find what they need either from a repository or through web search engines. This study attempts to investigate data users’ requirements and the role that data repositories can play in supporting data discoverability by meeting those requirements. We collected 79 data discovery use cases (or data search scenarios), from which we derived nine functional requirements for data repositories through qualitative analysis. We then applied usability heuristic evaluation and expert review methods to identify best practices that data repositories can implement to meet each functional requirement. We propose the following ten recommendations for data repository operators to consider for improving data discoverability and user’s data search experience: 1. Provide a range of query interfaces to accommodate various data search behaviours. 2. Provide multiple access points to find data. 3. Make it easier for researchers to judge relevance, accessibility and reusability of a data collection from a search summary. 4. Make individual metadata records readable and analysable. 5. Enable sharing and downloading of bibliographic references. 6. Expose data usage statistics. 7. Strive for consistency with other repositories. 8. Identify and aggregate metadata records that describe the same data object. 9. Make metadata records easily indexed and searchable by major web search engines. 10. Follow API search standards and community adopted vocabularies for interoperability.

Heuristic usability evaluation on games: a modular approach

Heuristic evaluation is the preferred method to assess usability in games when experts conduct this evaluation. Many heuristics guidelines have been proposed attending to specificities of games but they only focus on specific subsets of games or platforms. In fact, to date the most used guideline to evaluate games usability is still Nielsen’s proposal, which is focused on generic software. As a result, most evaluations do not cover important aspects in games such as mobility, multiplayer interactions, enjoyability and playability, etc. To promote the usage of new heuristics adapted to different game and platform aspects we propose a modular approach based on the classification of existing game heuristics using metadata and a tool, MUSE (Meta-heUristics uSability Evaluation tool) for games, which allows a rebuild of heuristic guidelines based on metadata selection in order to obtain a customized list for every real evaluation case. The usage of these new rebuilt heuristic guidelines allows an explicit attendance to a wide range of usability aspects in games and a better detection of usability issues. We preliminarily evaluate MUSE with an analysis of two different games, using both the Nielsen’s heuristics and the customized heuristic lists generated by our tool.

Focus Section Health IT Usability: Applying a Task-Technology Fit Model to Adapt an Electronic Patient Portal for Patient Work.

Although electronic patient portals are offered by most health care organizations, poor usability and poor fit to patient needs may pose barriers to adoption. We collaborated with an academic hospital to conduct iterative user evaluation of a newly deployed portal designed to deliver inpatient data upon hospital discharge. Three evaluators applied heuristic usability evaluation and conducted 23 individual user testing sessions with patients with chronic disease or managing the care of family members with chronic disease. Evaluation and development/improvement were conducted iteratively. User testing and analysis of qualitative data were both conducted from the perspective of a task-technology fit framework, to assess the degree of fit between the portal and patient work. Ability to complete health information management tasks, perceived usability, and positive comments from users improved over the course of the iterative development. However, patients still encountered significant difficulties accomplishing certain tasks such as setting up proxy accounts. The problems were most severe when patients did not start with a clear understanding of tasks that they could accomplish. In exploring the portal, novice users frequently described anecdotes from their own medical history or constructed fictional narratives about a hypothetical patient. Chronic illness imposes a significant workload on patients, and applying a task-technology framework for evaluation of a patient portal helped improve the portal's fit to patient needs. However, it also revealed that patients often lack a clear understanding of tasks that would help them accomplish personal health information management. Portal developers may need to educate patients about types of patient work involving medical centers, in a way that developers of clinical information systems do not need to do. An approach to doing this might be to provide narratives about hypothetical patients.

Using scenarios and personas to enhance the effectiveness of heuristic usability evaluations for older adults and their care team

ObjectiveUsing heuristics to evaluate user experience is a common methodology for human-computer interaction studies. One challenge of this method is the inability to tailor results towards specific end-user needs. This manuscript reports on a method that uses validated scenarios and personas of older adults and care team members to enhance heuristics evaluations of the usability of commercially available personal health records for homebound older adults. Materials and MethodsOur work extends the Chisnell and Redish heuristic evaluation methodology by using a protocol that relies on multiple expert reviews of each system. It further standardizes the heuristic evaluation process through the incorporation of task-based scenarios. ResultsWe were able to use the modified version of the Chisnell and Redish heuristic evaluation methodology to identify potential usability challenges of two commercially available personal health record systems. This allowed us to: (1) identify potential usability challenges for specific types of users, (2) describe improvements that would be valuable to all end-users of the system, and (3) better understand how the interactions of different users may vary within a single personal health record. ConclusionsThe methodology described in this paper may help designers of consumer health information technology tools, such as personal health records, understand the needs of diverse end-user populations. Such methods may be particularly helpful when designing systems for populations that are difficult to recruit for end-user evaluations through traditional methods.

Towards an online video platform for MOOCs

This paper presents the OpenDelos platform implemented in the context of the project “Central Repository of Greek Open Courses”. OpenDelos is an online video editing and publishing platfοrm for online courses, e.g., Massive Open Online Courses (MOOCs). OpenDelos is a unique open source platform that supports not only video on demand functionality but also live streaming functionality along with the related scheduling functionality. OpenDelos is suitable for Higher Education Institutions (HEIs) offering (a) academic roles structure, (b) unique data presentation of the (c) searching and browsing features, (d) hosting of open multimedia and educational content. The main goal of this paper is to present the technical aspects of OpenDelos platform. Τhus, this paper describes OpenDelos’ architecture, modules, and implementation issues. Furthermore, the heuristic usability evaluation conducted by experts during the implementation of OpenDelos platform is presented. The heuristic evaluation showed that OpenDelos did not present major usability problems. Experts’ comments have been taken into account to improve the platform. The improved version is ready and it has been distributed to the end users in order to measure end users’ acceptance towards OpenDelos.

Heuristic Evaluation of Usability for Gamified Co-Curricular Lesson: A Study Case of Learn to Scout

Heuristic Evaluation of Usability for Gamified Co-Curricular Lesson: A Study Case of Learn to Scout

Heuristic Usability Evaluation of Wearable Mental State Monitoring Sensors for Healthcare Environments

In recent years, there has been tremendous growth in both the number and diversity of wearable sensors, including sensors for monitoring mental state. Understanding physiological metrics such as fatigue and stress is an important aspect of human factors research, yet collecting and analyzing these measures can be resource intensive. This paper explores the usability and applicability of four off-the-shelf wearable sensors (Emotiv Epoc, Melon Headband, Spire Stone, and Muse™ Headband) for applied healthcare research. We perform a heuristic usability evaluation of the four sensors and discuss the extent to which each device can be applied to human factors healthcare research in clinical settings.

Heuristic Usability Evaluation of University of Hong Kong Libraries' Mobile Website

As mobile technology grows rapidly, more and more students use mobile devices for educational purposes. Under this circumstance libraries, especially academic libraries, should try to expand their services and design mobile websites to meet users' needs. The history of mobile library websites is relatively short, especially in Asia. Usability evaluation of mobile library websites is a new issue for study. As such, this paper evaluates the usability of the University of Hong Kong Library (HKUL) mobile website as a case study by benchmarking against two other universities (the Harvard University and the Chinese University of Hong Kong), according to the ten usability heuristics developed by Nielsen (1994). The evaluation result shows that the mobile website contains some usability issues in 5 heuristics, such as unable to inform users the waiting time, some information not provided in a logical way, some consistency problems in displaying contents, lack of advanced searching for expert users, and inadequate helpful error message. The weakness found in this way provides useful hints in making improvement to mobile website. The design of the HKUL mobile website is quite similar to other mobile library websites, and thus, the results are quite useful for many other academic libraries.