Heuristic Evaluation Technique Research Articles

Enhancing Usability Assessment with a Novel Heuristics-Based Approach Validated in an Actual Business Setting

We have long been committed to improving Usability Evaluation and one of the proposals we have been working on the most is the use and improvement of the Heuristic Evaluation (HE) technique. With this in mind, we proposed an improvement which was tested in an experiment. This article describes an experiment carried out in a real business professional context. Fifteen usability experts from a reputable company evaluated eight websites (four supermarket and four bank platforms) using our HE proposal for the first time in real-world scenarios. This experimentation analyzed two main aspects: firstly, whether individual or group evaluations affect the final result, and secondly, whether the heuristic evaluation technique is effective in a real business and professional context. Regarding the Usability Percentage (UP) event, the results indicate that there was little difference between group and individual evaluations. The mean UP for the group was 57.88%, while the mean UP for individuals was 56.66%. The experiment provided sufficient information to suggest a new version of our HE methodology, specifically designed to improve results in real-life contexts. Furthermore, the experiment’s findings support the proposal of this new methodology, which is better suited to the business environment.

IMPROVING MATERIAL REQUIREMENTS PLANNING THROUGH WEB-BASED: A CASE STUDY THAILAND SMEs

In the small business industry, effective cost control is of paramount importance. A well-functioning Material Requirements Planning (MRP) process plays a vital role in enabling businesses to manage their costs efficiently. This research paper explores the improvement of MRP in the context of small businesses in Thailand through the implementation of a web-based application. The objective was to enhance the MRP process by developing an MRP system using ReactJS, NodeJS, and PostgreSQL. The system was evaluated using heuristic evaluation techniques and the results indicated a positive outcome, the mean value is 0.83. The developed web based MRP system proved beneficial for small businesses, as it effectively reduced stocking costs and facilitated efficient raw material procurement. This research provides valuable insights into the implementation of web based MRP systems, enabling small businesses to optimize inventory management and enhance operational efficiency.

Developing a Native Model for Evaluating the Usability of Health Information Systems Through Heuristic Evaluation Technique

Introduction: A quality criterion for evaluating information systems (ISs) is their usability, which can be determined by examining the user interface. User interface issues may generate user errors, dissatisfaction, and inefficiency, making IS adoption difficult. Therefore, the International Standard Organization recommends usability testing to evaluate IS acceptability and deployment. So, this study aimed to develop a native model for the usability testing of hospital information systems (HISs) via heuristic evaluation (HE). Method: This multi -staged qualitative study was conducted in 2020; ScienceDirect, PubMed, Web of Science, Scopus, Civilica, and Magiran databases were searched. After meeting the inclusion criteria, the articles were scored. Then, the initial model was created in the following stage by extracting data from library sources and validated using the Delphi technique, and consequently, the final model was presented. Results: : The native model was developed using thirteen principles: 1) visibility of system status, 2) the match between the system and the real world, 3) users’ control and freedom, 4) consistency and standards, 5) error prevention, 6) recognition rather than recall, 7) flexibility and efficiency of use, 8) aesthetic and minimalist design, 9) helping users recognize, diagnose, and recover errors, 10) documentation and help, 11) skill , 12) pleasurable, respectful interaction with users, and 13) privacy protection. Conclusion: This study demonstrated that localizing the usability evaluation of HISs based on trustworthy sources and HE is a wise strategy, because not only it avoids repetitive processes, but also it uses the best up -to -date resources to enable the presentation of the highest evaluation quality.

IoT applied to irrigation systems in agriculture: A usability analysis

The Internet of Things favors using technological tools in rural environments thanks to the ability to connect to the Internet between devices that facilitate daily tasks. The research aims to evaluate the usability of the decision support system for irrigation in agriculture, AgroRIEGO, through the development of an IoT-based device. The sponsors of this project were the Ministry of Information and Communication Technologies and the Center of Excellence in the Internet of Things Appropriation (CEA-IoT) in Colombia. Among the methods used is the use of the heuristic evaluation technique, structured into 15 categories and 62 subcategories of assessment. This analysis was complemented by the contribution of a group of experts in the design and development of IoT applications and devices and agriculture to assess the system's attributes.

User Evaluation of UbiQuitous Access Learning (UQAL) Portal: Measuring User Experience

The goal of user experience (UX) research in human-computer interaction is to understand how humans interact with technology. This paper aimed to evaluate the interface and user experience of UbiQuitous Access Learning Portal (UQAL) and make recommendations for the system interface. UQAL Portal is an e-learning web portal that teaches a targeted group of users how to start a business or an online business using an e-learning portal. The portal will be used to search for business-related information, among other things. The User Experience Questionnaire (UEQ) is used to evaluate user experience. The interface is evaluated using a heuristic evaluation technique based on Nielsen’s ten heuristics. According to the UEQ results, the average score for each aspect in 30 UQAL users is: Attractiveness aspect: 1.77; Perspicuity aspect: 2.20; Efficiency aspect: 2.30; Dependability aspect: 1.73; Stimulation aspect: 0.63; and Novelty aspect: 1.27. A comparison of the average score in the dataset product of UEQ Data Analysis Tool revealed that the Perspicuity, Efficiency, and Dependability aspects of UQAL belonged to the Excellent category. The Attractiveness and Novelty aspects could be categorized as Good, and its stimulation could be categorized as Below Average. Four evaluators participate in the heuristic evaluation, which tests all user categories in UQAL. The findings of this study can be used as a suggestion and reference for UQAL Portal improvement.

Enhancing Usability of E-Learning Platform: A Case Study of Khan Academy

Human-Computer Interaction (HCI) is a subject that measures usability and evaluates the interface of a system, software, or product to see its efficiency, effectiveness, and user satisfaction. On the other hand, at the time, the pandemic spread of Coronavirus (Covid-19) having an immense impact on schools, universities, and other educational sectors. In this current unpredictable situation, the importance of online education is visible, therefore, the usability of e-learning platforms truly plays an essential role in our lives. This study is about to evaluate the usability of a very common e-learning portal i.e., Khan Academy by user testing and heuristic evaluation techniques. Effectiveness, efficiency, and satisfaction of Khan Academy were measured quantitatively, while usability issues were identified qualitatively through heuristic evaluation. The study result shows that Khan Academy suffers from certain usability problems which need to be addressed. The study also presents relevant recommendations which are scrutinized and approved by IT specialists and UI/UX designers. The study concluded that suggested amendments are important to consider enhancing the usability of different e-Learning portals generally and Khan Academy particularly.

Gender-based Game Engagement Model Validation using Low Fidelity Prototype

Lack of female involvement in ICT field calls for new approaches to ICT education. Serious game is an effective tool for learning, so it can be used to motivate and engage female students to study Computing. Engagement elements for serious game were identified from previous researches and a conceptual model for gender-based engagement using serious game was constructed. In this paper, the model validation of a low fidelity prototype using heuristic evaluation technique for educational game is discussed. The heuristic scale was adapted from previous studies, comprise of playability and enjoyment constructs. In addition, educational elements items are also included. The scale was then validated for content validity, and a pilot study with four (5) HCI experts. Finally, five (5) experts from HCI, Game and Computer Networking subject evaluate the model using this scale. Results of the evaluation showed that the model’s constructs have good usability, playability, enjoyment. Future work will be to validate the model using high fidelity working prototype and user testing.

User experience evaluation of e-report application using cognitive walkthrough (cw), heuristic evaluation (he) and user experience questionnaire (ueq)

E-Report is developed based on the web to make it easier in preparing students’ assessment reports in the form of rating reports, competency achievement reports, and ledgers. Even, many high schools/vocational schools in Bali have not implemented the E-Report. Implementation of cognitive walkthrough techniques, heuristic evaluation, and users’ experience questionnaire in this research aimed to find out the results of evaluating user experience from the aspects of effectiveness, efficiency, user satisfaction, and recommendations for improvement. Effectiveness and efficiency were calculated by Cognitive Walkthrough techniques (CW), user satisfaction by user experience questionnaire techniques (UEQ) and recommendations for improvement, obtained from experts who discuss aspects of usability with Heuristic Evaluation techniques and Cognitive Walkthrough data. Based on the results obtained in terms of users, E-Report has been ineffective and inefficient. While in terms of user satisfaction, respondents were satisfied with Applications E-Report. This is because the effectiveness of a task that has a very large error and failure in one of the features as well as the clarity of results that poor user satisfaction can affect processing time. Therefore, evaluation of user experience based on aspects of usability in the E-Report has not met the criteria for products with good usability. Improvements made on the start page, page values, page value results were done through a photo frame. Therefore, identifying the problems from these three sources can further optimize the usability and improve user experience on E-Report.

Safeuristics! Do Heuristic Evaluation Violation Severity Ratings Correlate with Patient Safety Severity Ratings for a Native Electronic Health Record Mobile Application?

Usability of electronic health records (EHRs) remains challenging, and poor EHR design has patient safety implications. Heuristic evaluation detects usability issues that can be classified by severity. The National Institute of Standards and Technology provides a safety scale for EHR usability. Our objectives were to investigate the relationship between heuristic severity ratings and safety scale ratings in an effort to analyze EHR safety. Heuristic evaluation was conducted on seven common mobile EHR tasks, revealing 58 heuristic violations and 28 unique usability issues. Each usability issue was independently scored for severity by trained hospitalists and a Human Factors researcher and for safety severity by two physician informaticists and two clinical safety professionals. Results demonstrated a positive correlation between heuristic severity and safety severity ratings. Regression analysis demonstrated that 49% of safety risk variability by clinical safety professionals (r = 0.70; n = 28) and 42% of safety risk variability by clinical informatics specialists (r = 0.65; n = 28) was explained by usability severity scoring of problems outlined by heuristic evaluation. Higher severity ratings of the usability issues were associated with increased perceptions of patient safety risk. This study demonstrated the use of heuristic evaluation as a technique to quickly identify usability problems in an EHR that could lead to safety issues. Detection of higher severity ratings could help prioritize failures in EHR design that more urgently require design changes. This approach is a cost-effective technique for improving usability while impacting patient safety. Results from this study demonstrate the efficacy of the heuristic evaluation technique to identify usability problems that impact safety of the EHR. Also, the use of interdisciplinary teams for evaluation should be considered for severity assessment.

A Heuristic Evaluation of a Shariah-Compliant Fashion E-Commerce Prototype

Every Muslim should wear shariah-compliant clothes as admonished in the Holy Quran and Hadith. Covering awrah is a command from Allah for all Muslims and they should obey this Islamic Shariah Law. This paper discusses the heuristic evaluation of a shariah-compliant fashion e-commerce prototype for Muslim women. The prototype aims to help Muslim women to find and purchase shariah-compliant clothes easily and furthermore allows more Muslim women to cover their awrah. Besides, the fashion e-commerce also has the capability in giving suggestion to the users based on their body shapes and size.The prototype was evaluated using heuristic evaluation technique with three software ergonomic experts. Although the evaluation indicates that the information provided is useful and the prototype is well-designed, it also revealed few design issues that should be refined in order to improve its usability before testing it on the real user.

UX in Practice: Applying a Heuristic Evaluation Technique to Real World Challenges

Introduction Heuristic Evaluation is a usability method that requires usability experts to review and offer feedback on user interfaces based on a list of heuristics or guidelines. Heuristic Evaluations allow designers to get feedback early and quickly in the design process before a full usability test is done. Unlike many usability evaluation methods, Heuristic Evaluations are performed by usability experts as opposed to target users. That is one reason it is going to make a great challenge activity for the UX Day Challenge session. Heuristic Evaluation is a usability method often used in conjunction with usability testing. During the evaluation, usability experts evaluate an interface based on a list of heuristics or guidelines (Nielsen and Molich, 1990). There are several sets of guidelines and they are used to evaluate a myriad of interfaces from gaming (Pinelle, Wong & Stach, 2008) and virtual reality (Sutcliffe & Gault, 2004) to online shopping (Chen & Macredie, 2005). Some of the most common heuristic guidelines to choose from were created by Nielsen (Nielsen and Molich, 1990) (Nielsen, 1994), Norman (Norman, 2013), Tognazzini (Tognazzini, 1998), and Shneiderman (Shneiderman, Plaisant, Cohen and Elmqvist, 2016). Choosing the best set of guidelines and the most appropriate number of usability professions is important. Nielsen and Molich’s research found that individual evaluators only find 20-51% of the usability problems when evaluating alone. However, when the feedback of three to five evaluators is aggregated together, more usability problems can be uncovered (Nielsen and Molich, 1990). This method can be advantageous because designers can get quick feedback early for iteration before a full round of usability testing is performed. The goal of this session is to introduce this method to some and give others a refresher on how to apply this method in the real world. The Challenge For several years, UX day has offered an alternative session. The most intriguing sessions were interactive and offered hands-on training. For this UX Day Challenge session, teams of at most five participants will perform a Heuristic Evaluation of a sponsor’s website or product. During the session, participants will be introduced to Heuristic Evaluations. Topics such as how to perform one, who should perform one, and when it is appropriate to perform one will be covered. Additionally, the pros and cons of using this method will be discussed. Following the introduction to Heuristic Evaluation, teams will use the updated set of Nielson Heuristics (Nielsen, 1994) for the evaluation exercise. Although there are several sets of heuristics, Nielsen’s is one of the best known and widely accepted sets. The following Updated Nielsen Heuristics will be used: • Visibility of system status • Match between system and the real world • User control and freedom • Consistency and standards • Error prevention • Recognition rather than recall • Flexibility and efficiency of use • Aesthetic and minimalist design • Help users recognize, diagnose, and recover from errors • Help and documentation Following the evaluation period, teams will be asked to report their findings and recommendations to the judges and audience. The judges will deliberate and announce the winner. Conclusion This alternative session will be an opportunity to potentially expose participants to a methodology they may not use often. It will also be an opportunity to have a hands-on learning experience for students who have not formally used this methodology in the real world. Most importantly this session is in continuation of the goal to continue to bring new, interesting, and disruptive sessions to the traditional “conference” format and attract UX practitioners.

Usability Evaluation of the Smartphone User Interface in Supporting Elderly Users From Experts’ Perspective

Despite the necessity for smartphones to be designed to better accommodate the elderly user, currently available smartphone user interfaces (UIs) do not appear to be optimized for that purpose. The objectives of this paper are to identify the potential usability problems among elderly adults when interacting with a smartphone's UI, and recommend improvements to UI designs. The heuristic evaluation technique was used to determine the anticipated usability problems. SMASH, a set of 12 usability heuristics for smartphone and mobile applications was applied within a controlled environment, with five experts with the necessary competence to perform the evaluation. The results of the study indicated that there were 27 usability problems and 27 heuristic violations encountered. “Minimize the user's memory load”and “match between system and real world”were the two most frequently violated heuristics. This study was complemented by testing with elderly people, and the results show that 79.17% of the problems experienced by the elderly were predicted by the experts. Usability problems were classified into four categories: 1) appearance; 2) language; 3) dialogue; and 4) information. Problems categories were further divided into sub-categories, and design solutions were suggested for each sub-category. This study's findings contribute to understanding the problems that hinder elderly users in using smartphones and provide valuable feedback to designers of smartphone technology regarding improvements to the UI to better suit the elderly.

NISHA: Novel Interface for Smart Home Applications for Arabic Regionsubtitle as Needed

Researchers have developed many devices and applications for smart homes to control home’s appliances. The main goal of this research is to propose a touch-based interface (namely, NISHA) for smart homes to meet user needs and requirements and is able to control any appliance in the house. This study is designed for people and circumstances in the Middle East countries (Jordan and West Bank) and therefore, is set out to design a user interface for smart home applications taking into consideration the economic, social, and technological differences. Referring to those differences, NISHA was designed in a classical representational design instead of a modern advanced one, based on virtual images instead of text, full control instead of automatic control, and very restrictive privacy issues for people of these countries still look at smart homes as a technology that threaten their privacy. Moreover, NISHA was tested and evaluated using heuristic and cognitive walk-through evaluation techniques. Evaluation results showed that 80% of users and experts were satisfied with NISHA as a user friendly interface, 90% of users were satisfied that NISHA met their expectations, and finally, 93% of users strongly asked to have NISHA in their daily lives.

Assessing the awareness mechanisms of a collaborative programming support system

The learning and teaching of Programming can benefit from the principles of Computer Supported Collaborative Learning (CSCL). With that purpose in mind, the COLLECE system was created to support synchronous collaborative programming in learning settings. Unlike other systems with similar objectives, COLLECE incorporates many elements to support group awareness. This article presents an empirical study in which the usefulness of some of the awareness mechanisms included in this system is evaluated. One of the main contributions of this work is the combination of different techniques to evaluate interactive systems, such as questionnaires, laboratory testing, heuristic evaluation, automatic logging and eye tracking techniques. The joint use of these techniques (of both a subjective and objective nature) allows us to carry out a more complete analysis of the system under study and, in particular, about its support of awareness.

Evaluation of a BCMA's Electronic Medication Administration Record.

Barcode medication administration (BCMA) systems can reduce medication errors, but sociotechnical issues are quite common. Although crucial to nurses' work, few usability evaluations are available for electronic medication administration record (eMARs) screens. The purpose of this research was to identify current usability problems in the Veterans Administration's (VA) eMAR/BCMA system and explore how these might affect nurses' situation awareness (SA). Three expert evaluators used 10 tasks/elements, heuristic evaluation techniques, and explored potential impacts using a SA perspective. The results yielded 99 usability problems categorized into 440 heuristic violations with the largest volume in the category of Match With the Real World. Fifteen usability issues were rated as catastrophic with the Administer/Chart medications task having the most. Situational awareness was affected at all levels, especially at Level 2, Comprehension. Usability problems point to important areas for improvement, because these issues have the potential to affect nurses' SA, "at a glance" information, nurse productivity, and patient safety.

An Intelligent Framework for Website Usability

With the major advances of the Internet throughout the past couple of years, websites have come to play a central role in the modern marketing business program. However, simply owning a website is not enough for a business to prosper on the Web. Indeed, it is the level of usability of a website that determines if a user stays or abandons it for another competing one. It is therefore crucial to understand the importance of usability on the web, and consequently the need for its evaluation. Nonetheless, there exist a number of obstacles preventing software organizations from successfully applying sound website usability evaluation strategies in practice. From this point of view automation of the latter is extremely beneficial, which not only assists designers in creating more usable websites, but also enhances the Internet users’ experience on the Web and increases their level of satisfaction. As a means of addressing this problem, an Intelligent Usability Evaluation (IUE) tool is proposed that automates the usability evaluation process by employing a Heuristic Evaluation technique in an intelligent manner through the adoption of several research-based AI methods. Experimental results show there exists a high correlation between the tool and human annotators when identifying the considered usability violations.

Detecting Usability Problems and Offering Lines of Solutions: An Instrument' Proposal for Measuring Usability in Online Services

The growing offer of online public services aims to provide huge benefits to users in terms of time and facilities, and also to governments in terms of cost saving. However, the benefits that both users and providers could extract from the digitalization of transactions are commonly affected by the problems that citizens face while using the facilities. It is at this point where usability appears as a central variable. Although the usability of transactional services is a central variable, there are not clear methodological approaches for measuring the state of the services within Latin-American countries. In order to face this issue, we propose an instrument based upon the heuristic-evaluation technique for assessing the usability of online services in e-government websites. We evaluate the usability of 60 online services offered by the Chilean government in order to validate the instrument, and to show the scope of the instrument and the lines of intervention that such evaluation allows. Results showed three main usability problems of the online services: (1) the system does not provide control to users over their action, (2) the services are structured in a way that produces uncertainty, and (3) there is not guidance to users while they conduct the transactions in the website. We discuss the results appealing to the necessity of having rigorous methodologies for assessing the usability features of transactional services, and showing the main problems reported in the measurement conducted. This demonstrates the relevance of assessing usability in online services when analyzing the slow growing and lack of confidence that online services have among users.

Usability Evaluation of An Electronic Medication Administration Record (eMAR) Application

Electronic medication administration records (eMARs) have been widely used in recent years. However, formal usability evaluations are not yet available for these vendor applications, especially from the perspective of nurses, the largest group of eMAR users. To conduct a formal usability evaluation of an implemented eMAR. Four evaluators examined a commercial vendor eMAR using heuristic evaluation techniques. The evaluators defined seven tasks typical of eMAR use and independently evaluated the application. Consensus techniques were used to obtain 100% agreement of identified usability problems and severity ratings. Findings were reviewed with 5 clinical staff nurses and the Director of Clinical Informatics who verified findings with a small group of clinical nurses. Evaluators found 60 usability problems categorized into 233 heuristic violations. Match, Error, and Visibility heuristics were the most frequently violated. Administer Medication and Order and Modify Medications tasks had the highest number of heuristic violations and usability problems rated as major or catastrophic. The high number of usability problems could impact the effectiveness, efficiency and satisfaction of nurses' medication administration activities and may include concerns about patient safety. Usability is a joint responsibility between sites and vendors. We offer a call to action for usability evaluations at all sites and eMAR application redesign as necessary to improve the user experience and promote patient safety.

Testing Website Usability in Spanish-Speaking Academia through Heuristic Evaluation and Cognitive Walkthroughs

Fil: Gonzalez, Maria Paula. Consejo Nacional de Investigaciones Cientificas y Tecnicas. Centro Cientifico Tecnologico Conicet - Bahia Blanca; Argentina. Universidad Nacional del Sur. Departamento de Ciencias e Ingenieria de la Computacion. Instituto de Ciencias e Ingenieria de la Computacion; Argentina

Heuristic Evaluation of E-Learning Products Extended Garrett's Model of User Experience

The Azim Premji Foundation has created a repository of educational CDs based on various state curricula for bringing technology-based learning to schools, specifically schools in rural areas. This initiative is expected to meet several objectives, primarily: (a) to attract children to schools; (b) once attracted to school, sustain their interest in attending school; and (c) ensure effective learning by providing relevant and compelling content. Evaluating the first two objectives is an ongoing process, made possible through direct measures— tracking enrolments and dropouts. However, to ascertain learning effectiveness of the CDs, an objective analysis and evaluation of the product was needed. In this article we share our experience of using the heuristic evaluation technique to evaluate the CDs and the extended Garrett's model of user experience for organizing the findings of each CD into clear action lists for redesigning existing CDs and designing others in the future.