Heuristic Evaluation Research Articles

Potential effectiveness and efficiency issues in usability evaluation within digital health: A systematic literature review

Context:Digital Health (DH) is widely considered essential for sustainable future healthcare systems. Software quality, particularly usability, is crucial for the success and adoption of most DH products. However, concerns about the effectiveness and efficiency of usability evaluation of DH products have been raised. Objective:This article aims to analyse the prevalence and application contexts of usability evaluation methods in DH and to highlight potential issues related to their effectiveness and efficiency. Method:A systematic literature review of usability evaluation studies, published by (academic) practitioners between 2016 and April 2023, was conducted. 610 primary articles were identified and analysed, utilising five major scientific databases. Results:Our findings show a preference for inquiry (85%) and testing (63%) methods, with inspection used less frequently (17%). The published studies employed methods like questionnaires (75%); notably the SUS (49%), semi-structured interviews (25%), and heuristic evaluations (73%), with percentages based on their group. Data collection mainly involved the use of participant feedback (45%), audio/video recordings (44%), and system logs (20%), with both qualitative and quantitative data analyses prevalent in studies. However, several usability characteristics such as accessibility, memorability, and operability were found to be largely overlooked, and automation tools or platforms were not widely used. Among the systems evaluated were mHealth applications (70%), telehealth platforms (36%), health information technology (HIT) solutions (29%), personalized medicine (Per. Med.) (17%), wearable devices (12%), and digital therapeutics (DTx) interventions (6%), with the participation of general users, patients, healthcare providers, and informal caregivers varying based on the health condition studied. Furthermore, insights and experiences gathered from 24 articles underscored the importance of a mixed-method approach in usability evaluations, the limitations of traditional methods, the necessity for sector-specific customisation, and the potential benefits of remote usability studies. Moreover, while eye-tracking emerged as a promising evaluation technique, careful execution and interpretation are crucial to avoid data misinterpretation. Conclusion:The study’s findings showed that employing a combination of inquiry and testing-based methods is prevalent for evaluating DH platforms. Despite an array of DH systems, method distribution remained consistent across platforms and targeted user groups. The study also underlines the importance of involving target user groups in the process. Potentially affected cognitive abilities of participants and potential user groups of interest have to be taken into account when choosing evaluation methods, and methods might therefore need to be tailored. Complementary inspection methods might be particularly useful when recruiting representative participants is difficult. Several potential paths for future research are outlined, such as exploring novel technologies like artificial intelligence, for improved automation tool support in the usability evaluation process.

Development of a Competency-Based Triage Education Application and Usability Testing for Triage Training Based on the Korean Triage and Acuity Scale.

Improving the quality of emergency nursing care requires enhancing triage competency through education programs that integrate the content of the Korean Triage and Acuity Scale, a triage system specifically developed for the Korean context. Thus, this study developed a competency-based triage education application (CTEA), based on the Korean Triage and Acuity Scale, and evaluated its effectiveness through usability testing. The CTEA used various teaching methods and game mechanisms, including lectures, case studies, and video-based scenarios, to improve triage competency. In the usability testing phase, the CTEA was refined through expert heuristic evaluation and user interviews conducted with the think-aloud method. Six themes were derived from the interviews, highlighting the need for a well-structured program with realistic scenarios, easy accessibility, gamification, functional improvements, and future triage educational applications. These findings suggest that the CTEA is effective and satisfactory for users and can contribute to improving the triage competency of emergency nurses. [J Contin Educ Nurs. 2024;55(1):33-41.].

User interface analysis on shopee website using heuiristic method

Human and computer interaction is a science that focuses on planning and designing a system so that users can interact and work together according to the needs of the users themselves in the most effective way at every step of their work. The role of the user interface greatly influences the usability of an application or system. Heuristic Evaluation (HE) is considered the best usability evaluation method compared to others because it can find severe problems that other methods cannot detect. Therefore researchers will analyze the user interface on the Shopee website using the HE method. User Interface Analysis on Websites with the HE Method through online questionnaire distribution, from the results of online questionnaire distribution, it can be concluded that the website has deficiencies in terms of auxiliary features and documentation to assist users in using the system. Declared less because it has the lowest rcount (Total) value of 0.587 compared to other counts. Suggestions for research on Shopee user satisfaction analysis using the HE method, the authors convey suggestions that need to be improved in future research, namely that it is necessary to conduct research on Shopee user satisfaction using the HE method in a broader scope.

Designing and Evaluating a Nutrition Recommender System for Improving Food Security in a Developing Country.

Due to the increased price of foods in recent years and the diminished food security in Iran, nutrition recommender systems can suggest the most suitable and affordable foods and diets to users based on their health status and food preferences. Objective: The present study aimed to design and evaluate a recommender system to suggest healthy and affordable meals and provide a tele-nutrition consulting service. This applied three-phase study was conducted in 2020. In the first stage, the food items' daily prices were extracted from credible sources, and accordingly, meals were placed in three price categories. After conducting a systematic review of similar systems, the requirements and data elements were specified and confirmed by 10 nutritionists and 10 health information management and medical informatics experts. In the second phase, the software was designed and developed based on the findings. In the third phase, system usability was evaluated by four experts based on Nielsen's heuristic evaluation. Initially, 72 meals complying with nutritional principles were placed in three price categories. Following a literature review and expert survey, 31 data elements were specified for the system, and the experts confirmed system requirements. Based on the information collected in the previous stage, the Web-based software TanSa in the Persian language was designed, developed, and presented on a unique domain. During the evaluation, the mean severity of the problems associated with Nielsen's 10 principles was 1.2, which is regarded as minor. To promote food security, the designed system recommends healthy, nutritional, and affordable meals to individuals and households based on user characteristics.

Reflections on Surrogate-Assisted Search-Based Testing: A Taxonomy and Two Replication Studies based on Industrial ADAS and Simulink Models

Surrogate-assisted search-based testing (SA-SBT) aims to reduce the computational time for testing compute-intensive systems. Surrogates enhance testing techniques by improving test case generation focusing the testing budget on the most critical portions of the input domain. In addition, they can serve as approximations of the system under test (SUT) to predict test results instead of executing the tests on compute-intensive SUTs. This article reflects on the existing SA-SBT techniques, particularly those applied to system-level testing and often facilitated using simulators or complex test beds. Recognizing the diversity of heuristic algorithms and evaluation methods employed in existing SA-SBT techniques, our objective is to synthesize these differences and present a comprehensive view of SA-SBT solutions. In addition, by critically reviewing our previous work on SA-SBT, we aim to identify the limitations in our proposed algorithms and evaluation methods and to propose potential improvements. We present a taxonomy that categorizes and contrasts existing SA-SBT solutions and highlights key research gaps. To identify the evaluation challenges, we conduct two replication studies of our past SA-SBT solutions: One study uses industrial advanced driver assistance system (ADAS) and the other relies on a Simulink model benchmark. We compare our results with those of the original studies and identify the difficulties in evaluating SA-SBT techniques, including the impact of different contextual factors on results generalization and the validity of our evaluation metrics. Based on our taxonomy and replication studies, we propose future research directions, including re-considerations in the current evaluation metrics used for SA-SBT solutions, utilizing surrogates for fault localization and repair in addition to testing, and creating frameworks for large-scale experiments by applying SA-SBT to multiple SUTs and simulators.

User-centered design in mobile human-robot cooperation: consideration of usability and situation awareness in GUI design for mobile robots at assembly workplaces

Abstract Mobile cobots can increase the potential for assembly work in industry. For human-friendly automation of cooperative assembly work, user-centered interfaces are necessary. The design process regarding user interfaces for mobile human-robot cooperation (HRC) shows large research gaps. In this article an exemplary approach is shown to design a graphical user interface (GUI) for mobile HRC at assembly workplaces. The design is based on a wireframe developed to support situation awareness. An interactive mockup is designed and evaluated. This is done in two iterations. In the first iteration, a user analysis is carried out using a quantitative survey with n = 31 participants to identify preferred input modalities and a qualitative survey with n = 11 participants that addresses touch interfaces. The interactive mockup is developed by implementing design recommendations of the usability standards ISO 9241 – 110, 112 and 13. A heuristic evaluation is conducted with n = 5 usability experts and the measurement of situation awareness with n = 30 end users. In the second iteration, findings from the preceding iteration are implemented in the GUI and a usability test with n = 20 end users is conducted. The process demonstrates a combination of methods that leads to high usability and situation awareness in mobile HRC.

Prototipe Rekam Kesehatan Personal pada Pasien Diabetes Mellitus (Literature Review)

Personal Health Record (PHR) is a collection of health-related data that is archived and managed by an individual or parties associated with the PHR. Its purpose is to provide a comprehensive and accurate summary of a person's health history that can be accessed online. This research is a Literature Review study which aims to determine the features and implementation of Personal Health Records in Patients with Diabetes Mellitus. The method used is the Literature Review method from 7 journals which uses several methods including the prototype method, qualitative methods, heuristic evaluation methods, applied development methods, participatory action research methods, and randomized controlled trial methods with data collection techniques through google scholar, direct science and pubmed. The results of the research are seen in the features and implementation of the Personal Health Record. The Personal Health Record feature for diabetes mellitus with dialysis measures has 25 features on the main page, the PHR feature for diabetes mellitus has 7 main features. PHR implementation allows users to access their personal health records anytime and anywhere. Thus, the PHR system can increase accessibility and allow users to access PHR without time restrictions and also the Personal Health Record can assist in monitoring the health condition of Diabetes Mellitus patients. Research conclusion According to the journals analyzed, PHR is effective because it can encourage patient participation in individual health controls, and can access information about diabetes mellitus through monitoring physical activity, diet, weight, and glucose levels.

Improved multi-search strategy A* algorithm to solve three-dimensional pipe routing design

Pipe routing design (PRD) has a significant impact on the safety and operational efficiency of the system. PRD is involved in multi-disciplinary knowledge such as mechanical engineering, computer graphics and mathematical geometry, and it can be abstracted as a complex combined optimization problem. A* algorithm is one of the most commonly employed approaches to solve two-dimensional (2-D) path planning. However, due to the central idea of A* algorithm is all-traversal search, the search process considers all possibilities and requires a blind search one by one, it is difficult to solve PRD problems in complex three-dimensional (3-D) space environment. In this study, an improved multi-search strategy A* algorithm (MSSA*) is presented to overcome the disadvantages of the traditional A* algorithm in solving 3-D PRD problems. The MSSA* algorithm contains four novel mechanisms which are the node directional discrimination rule, the double-layer domain extension search strategy, the multi-factor heuristic evaluation function, and the dynamic adaptive weighting strategy. Then, a range of experiments are conducted to prove the validity of the presented mechanisms. Subsequently, the MSSA* is compared with several existing algorithms for solving 3-D PRD to further verify the superiority of MSSA* in terms of high-efficiency and practicality. Finally, the MSSA* is applied to solve the 3-D PRD problems for offshore production platforms in oil and gas treatment system.

Website redesign using Lean UX methods and heuristic evaluation (Case Study: SMA Negeri 3 Gorontalo)

Lean UX has grown into a product and service development method commonly used by start-ups around the world. Lean UX is becoming very popular because the product development process can be done with little financial capital, human resources, and time. In addition, heuristic evaluation is one of the testing techniques with a high level of effectiveness. The ten heuristic aspects have been able to overcome most of the errors before a product is launched. In this study, the two methods were combined to obtain optimal product results. This research was conducted on the website of SMA Negeri 3 Gorontalo, which after observing and interviewing, it was found that the current website does not pay much attention to aspects of the user interface and user experience. The results of testing by professionals using the heuristic evaluation method and severity ratings, get an average result of 0.83 where these results are included in the cosmetic problem, which means that the problem does not have to be fixed unless the project time is still available. Furthermore, testing of design recommendations was carried out on students and teachers, and the results of the SEQ assessment were 6.97 out of a scale of 7, from this assessment, it could be concluded that the design recommendations were acceptable to users.

Exploring User Experience and Usability of mHealth applications for people with diabetes: An Evaluation Study Using UEQ and HE4EH Checklist

Diabetes affects 10% of adults aged 20 to 79 globally, caused by insufficient insulin production. Mobile apps are effective in managing this chronic condition. This study aimed to investigate the experience of using mobile applications for healthcare patients with diabetes in their daily activities. Ten participants diagnosed with diabetes were invited to evaluate five Android mobile applications using a predefined protocol. We used the User Experience Questionnaire (UEQ) to assess the usability and user experience of the apps. In addition, the Heuristic Evaluations for mHealth (HE4EH) checklist was used by three domain experts to assess the apps. The results showed that one of the five apps had issues with the stimulus and novelty aspects of user experience. Among all the apps, the novelty had the lowest Likert scale value. While most apps had creative designs and attractive interfaces, the heuristic assessment revealed many violations and concerns about instruction and suitability due to a lack of crucial information for monitoring user routines.

A Conflict Resolution Strategy at a Taxiway Intersection by Combining a Monte Carlo Tree Search with Prior Knowledge

With the escalating complexity of surface operations at large airports, the conflict risk for aircraft taxiing has correspondingly increased. Usually, the Air Traffic Controllers (ATCOs) generate route, speed and holding instructions to resolve conflicts. In this paper, we introduce a conflict resolution framework that incorporates prior knowledge by integrating a Multi-Layer Perceptron (MLP) neural network into the Monte Carlo Tree Search (MCTS) approach. The neural network is trained to learn the allocation strategy for waiting time extracted from actual aircraft taxiing trajectory data. Subsequently, the action probability distribution generated with the neural network is embedded into the MCTS algorithm as a heuristic evaluation function to guide the search process in finding the optimal conflict resolution strategy. Experimental results show that the average conflict resolution rate is 96.8% in different conflict scenarios, and the taxiing time required to resolve conflicts is reduced by an average of 42.77% compared to the taxiing time in actual airport surface operations.

Usability Evaluation of Two National COVID-19 Registration Systems: A Heuristic Evaluation

Background: Data related to COVID-19 are increasing daily; the data analysis of the COVID-19 registry can provide critical information to policymakers and researchers for planning. The poor user interface design of these systems leads to problems in interacting with them. The present study aimed to evaluate the usability of two national systems to record daily data related to COVID-19 patients throughout hospitals in Iran called the "Medical Care Monitoring Center" (MCMC) and the "Electronic System for Syndromic Surveillance of Infectious Diseases" (ISSS). Methods: The interface of the aforementioned systems was assessed using a heuristic evaluation (HE) method in this descriptive, cross-sectional study. Using Nielsen’s 10 usability principles, three trained evaluators identified problems and determined each system’s severity rates independently. Results: This evaluation identified 164 usability problems, including 59 issues in the "ISSS" system and 105 violations in the "MCMC" system. The highest number of ISSS system problems was related to "Consistency and standard" and "Flexibility and efficiency of use". The average severity of problems in this system varied from 2 concerning "Consistency and standards", "Help users recognize, diagnose, and recover from errors", and "Help and documentation" to 3.6 concerning "Visibility of system status". The highest number of MCMC system problems was associated with "Help and documentation" and "Consistency and standards". The average severity of problems in this system varied from 2 to 3.28 regarding "Help and documentation" and "Recognition rather than recall", respectively. Conclusion: Errors may be decreased, data quality can be raised, and interactions can be enhanced by taking into account standards and guidelines for user interface design, such as the HE utilized in this work.

Serious Games Accessibility Design Model for Low-Vision Children

The evolution in information technology has brought worldwide access to information. Information technology has become an approach for effective learning in the education sector. This development provides an opportunity for people with disabilities. However, many educational software and applications in the market, such as serious games, cannot be accessed by low-vision children due to the visual properties in game design. For instance, small text and graphic sizes, lack of colour contrast, complex visual effects, or reliance on visual cues for instructions are among the visual properties that contribute to these accessibility challenges. They can play serious games as long as they are designed based on their accessibility needs. This paper proposed a serious games design model for low-vision children. The three main components are accessibility, game design, and pedagogy. The serious game accessibility design model is implemented using a low-fidelity prototype that takes the form of a serious game mockup. This prototype allows for basic interaction and testing of the game’s features, with a focus on accessibility considerations. The expert validation was conducted on the prototype using heuristic evaluation to assess its usability. Based on the evaluation, the experts agreed on implementing the proposed model to the prototype.

Enhancing UI/UX of The Smart Village System Website Using User-Centered Design Methods

The Smart Village System is a platform specifically designed to help publicize village potential. Of course, it is very important to test UI/UX on sidespin websites which are still very far from perfect, so it becomes a question of whether the UI/UX sidespin is in line with the expectations of the community as the target user. This study intends to design UI/UX on sidespin websites by utilizing the UCD method which aims to improve interface quality, efficiency, and productivity as well as community satisfaction. The UI/UX design on this sidespin website uses the UCD method because several stages involve establishing a relationship and interaction between users. 5 people are willing to become User Personas who can provide input on user satisfaction which will be evaluated using a Heuristic Evaluation which will be combined with Survey Ratings. The results of this study state that all aspects of the satisfaction level of the User Persona are in the satisfied category. Recommendations were found to be developed by the Smart Village System Team, among others, in terms of the principles of Heuristic Evaluation 4, 5, and 6. Consistency and Standards have restructured sentences and are consistent from one page to another. Error Prevention has added many features to the sidespin and finally, Recognition rather than recall has implemented local and rural nuances on the page sidespin.

ANALYSIS OF USABILITY USING HEURISTIC EVALUATION METHOD AND MEASUREMENT OF SUS ON PRICILIA APPLICATION

The Presence Digital Application (PRICILIA) is a presence application owned by PT. BGR Logistics Indonesia. However, until now, there has never been an evaluation of usability testing. Complaints from users regarding the PRICILIA application include menu displays that are less interactive, long loading times, and the unavailability of other alternatives besides GPS. Of course, this affects the level of user satisfaction with the application. Therefore, usability testing is needed to be able to measure the level of user comfort, application feasibility, and the application interface. In this study, the system evaluation method used is Heuristic Evaluation with measurement using the System Usability Scale (SUS). The results of this study indicate that aspects that need to be improved with high priority are Error Prevention (H5) and Recognition rather than Recall (H6) because they have a seriousness rating on a scale of 3, while the average score of the final seriousness rating obtained from a total of 10 heuristic aspects is 1, 72 which is then rounded off to a scale of 2. The SUS test results obtained an average final SUS score of 55.13. The results of the calculation of the SUS method are that the Acceptance Ranges have low marginal status, the Grade Scale is on a D scale, and the Adjective Twigs are at the OK level. This shows that the PRICILIA application still needs improvement. Therefore, 30 recommendations for improvement are proposed for future application development.

A Scoping Review of Heuristics in Videos Games Research: Definitions, Development, Application, and Operationalisation

Heuristics present a cheap and effective way of evaluating usability. However, in video games, evaluating unique player experiences that are dependent on individual preferences and abilities presents a challenge that goes beyond usability. Video games are more than just functional software, so games heuristics have been adapted to help examine functionality and experience. This paper reports on how papers published in the ACM Digital Library between 2012 and 2022 develop and apply heuristics in video games research. We found that heuristics are often used outside their intended purpose of being used in an expert evaluation. Instead, they are used as survey instruments, interview guides, codes for thematic analysis, and as design guidelines. This research contributes to HCI and video games research by distinguishing the terms design guidelines and design principles from heuristics. We make recommendations for researchers around developing heuristics and conducting video game heuristic evaluations. We propose a method for operationalising heuristics and make recommendations for the implementation of heuristics to improve the quality of video game heuristic reviews.

PENILAIAN HEURISTIK TERHADAP TAHAP PERSUASIF SISTEM E-PEMBELAJARAN BERASASKAN MOOCs

The learning concept of Massive Open Online Courses (MOOCs) is capable of receiving a large number of users and it requires little basic knowledge related to the subject taken. This technology will create dynamism and personalization of a wider user learning experience. In order to ensure that the MOOCs e-learning system can benefit users, researchers need to understand and consider the dynamic exchange between user cognition and motivation. Persuasive technology can be applied and integrated to facilitate student adaptation while accelerating their attitude change without using coercion. This technology will create relationships between individuals through computer-mediated interactions or human-computer-human interactions, as well as interactions between individuals and computers. Since the success in the use of e-learning systems depends on the readiness and acceptance of students, the low level of use of e-learning systems previously caused the benefits of online learning to not be optimally received. Research on this topic is still at an early stage where students' opinions or perceptions have not been fully studied. Thus, this study focuses on the heuristic evaluation of the persuasiveness of the Open Learning (PT) system used in Malaysian higher education institutions. The heuristic evaluation of the Open Learning system involves two expert evaluators and this evaluation is done in the form of a face-to-face discussion. Both evaluators are content creators for the PT system who are also experts in the field of e-learning and have been involved from the initial phase of using the PT system in Malaysia. The evaluation done on this PT system focuses on four persuasive categories namely; 1) primary task support; 2) dialogue support; 3) credibility support; and 4) social support. These four persuasive categories are outlined in the Persuasive Design Development Model (PRP). The conclusion from this study is that both experts agree that the dialogue support category needs to be improved by focusing on improving the effective reminder features of the PT system. The reminder principle is one of the persuasive principles outlined in the Persuasive Design Development Model (PRP).

Evaluating the Usability of a Medical Care Monitoring Center System Using Heuristic Method

Introduction: Simultaneously with the COVID -19 epidemic, many efforts were made to manage and treat the disease. One of the main activities was the development of health information systems to help monitor this disease, but due to the speed of the disease’s spread, there was not enough time to evaluate the usability of these systems. Therefore, the present research was conducted to evaluate the usability of the Medical Care Monitoring Center (MCMC) system . Method: In this cross -sectional descriptive study (2022), the usability of the MCMS system was evaluated using a heuristic evaluation method. The system’s usability was investigated based on the Nielsen’s usability principles. All the problems identified by evaluators were collected in a single list and the mean severity of each problem was calculated. Results: There were 41 usability problems. The lowest mismatch with usability principles was related to "Help and documentation" (n=1) and the highest one was related to "user control and freedom" (n=8) and "Match with real world" (n=7). In addition, most of the identified problems had a relatively high degree of severity and were placed in the category of major problems. Conclusion: Using the usability evaluation, a significant number of usability problems of the system were identified which, if not resolved, can cause increased errors, dissatisfaction, and decreased quality of information; therefore, it is necessary to solve these problems by the designers and developers of this system.

Usability Evaluation of Tablet-Based Electronic Medical Record Interface in Supporting Elderly Medical Doctors

Recently, tablet-based devices have become significantly more utilized platforms for electronic medical record (EMR) systems. EMR is the digital counterpart of the medical doctor’s office paper charts. EMR systems contain the medical and treatment histories of the patients in a unified practice. Nevertheless, statistics indicate that a considerable percentage of medical doctors are elderly, aged 60 and above. As using mobile handheld devices (including tablets) poses a well-recognized usability challenge for elderly users, the user interface (UI) usability of tablet-based EMR systems must be thoroughly assessed, considering the needs of elderly medical doctors. Accordingly, our objective is to address this need. Three expert evaluators implemented the heuristic evaluation (HE) approach to evaluate the UI usability of a commercial EMR system that is a tablet-based platform. Applying the HE approach helped identify usability problems that elderly medical doctors might encounter when utilizing a tablet-based EMR UI. In total, eight usability problems contributed to the seven heuristic violations discovered.

The Method User Experience Questionnaire Analysis of Identitas Kependudukan Digital Application

User Experience (UX) becomes important to explore and fulfill the needs for developing user-focused applications or products. The problems that exist to measure the level of user experience in the Digital Resident Identity (IKD) application and complaints submitted by users tend to lead to system and service quality problems in the IKD application. Then research purposes are assessed to evaluate the extent to which users can utilize and interact with the application. The method used by researchers is the User Experience Questionnaire (UEQ). UEQ is a widely used scoring methodology used to quantitatively measure user experience by administering a questionnaire to individuals that measures their subjective experiences and perceptions. Variables in UEQ, namely: efficiency, perspicuity, dependability, novelty, stimulation, and interestingness. The evaluation results of user experience in IKD applications using UEQ showed that there were 5 variables that received positive evaluation results, including attractiveness variables (mean 2.14), clarity variables (perspicuity) (mean 1.725), efficiency variables (efficiency) ( mean 1.725), accuracy variable (mean 1.525), and stimulation variable (mean 1.475). Whereas the novelty variable (mean 0.602) gets neutral evaluation results. From these results it can be used by IKD application developers to improve the user experience (user experience) of IKD applications on novelty variables by improving the appearance of applications that are more creative, inventive, and also cutting edge. The developer can pay attention to the recommendations that have been presented in this study. Parties who wish to conduct further research can use other evaluation methods such as Heuristic Evaluation.