Usability Evaluation Approach Research Articles

An eye-tracking-based approach to evaluate the usability of government portal websites in pilot smart cities

PurposeThe study aims to pioneer an innovative approach for the evaluation of government portal websites (GPWs) by introducing an eye-tracking-based method. The research meticulously pinpoints and analyses the distinct usability issues and challenges that users encounter while navigating and interacting with GPWs.Design/methodology/approachThis study devises an eye-tracking-based GPW usability evaluation approach, which focuses on the major functions (i.e. government information disclosure, government services and interactive responses) of GPWs. An Entropy Weighted Technique for Order Preference by Similarity to an Ideal Solution (EW-TOPSIS) method is employed to process eye-tracking indicator results for deriving GPW usability results.FindingsThe proposed approach is demonstrated to assess the usability of 12 GPWs in pilot smart cities in China, and it is found that most GPWs have lower-than-average usability. GPWs with low usability require more cognitive load that exhibit increased fixation and saccade. The comparisons among the GPW usability results from (1) the eye-tracking experiment, (2) questionnaire surveys and (3) the ready-made performance evaluation report validate the effectiveness of eye-tracking-based GPW usability evaluation.Originality/valueThe work contributes to shifting the GPW usability evaluation approach from a subjective judgment paradigm to an objective paradigm, as well as provides implications for enhancing GPW usability, including improving search function, reducing website complexity and prioritizing user needs.

Usability Evaluation Methods Used in Electronic Discharge Summaries: Literature Review.

With the widespread adoption of digital health records, including electronic discharge summaries (eDS), it is important to assess their usability in order to understand whether they meet the needs of the end users. While there are established approaches for evaluating the usability of electronic health records, there is a lack of knowledge regarding suitable evaluation methods specifically for eDS. This literature review aims to identify the usability evaluation approaches used in eDS. We conducted a comprehensive search of PubMed, CINAHL, Web of Science, ACM Digital Library, MEDLINE, and ProQuest databases from their inception until July 2023. The study information was extracted and reported in accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses). We included studies that assessed the usability of eDS, and the systems used to display eDS. A total of 12 records, including 11 studies and 1 thesis, met the inclusion criteria. The included studies used qualitative, quantitative, or mixed methods approaches and reported the use of various usability evaluation methods. Heuristic evaluation was the most used method to assess the usability of eDS systems (n=7), followed by the think-aloud approach (n=5) and laboratory testing (n=3). These methods were used either individually or in combination with usability questionnaires (n=3) and qualitative semistructured interviews (n=4) for evaluating eDS usability issues. The evaluation processes incorporated usability metrics such as user performance, satisfaction, efficiency, and impact rating. There are a limited number of studies focusing on usability evaluations of eDS. The identified studies used expert-based and user-centered approaches, which can be used either individually or in combination to identify usability issues. However, further research is needed to determine the most appropriate evaluation method which can assess the fitness for purpose of discharge summaries.

Evaluating the usability of a semantic environmental health data framework: Approach and study

Environmental exposures transported across air, land and water can affect our health making us more susceptible to developing a disease. Therefore, researchers need to face the complex task of integrating environmental exposures and linking them to health events with the relevant spatiotemporal and health context for individuals or populations. We present a usability evaluation approach and study of a semantic framework (i.e. Knowledge Graph, Methodology and User Interface) to enable Health Data Researchers (HDR) to link particular health events with environmental data for rare disease research. The usability study includes 17 HDRs with expertise in health data related to Anti-Neutrophil Cytoplasmic Antibody (ANCA)-associated vasculitis (AAV) in Ireland and Kawasaki Disease in Japan, and with no previous practical experience in using Semantic Web (SW) technologies. The evaluation results are promising in that they indicate that the framework is useful in allowing researchers themselves to link health and environmental data whilst hiding the complexities of SW technologies. As a result of this work, we also discuss the limitations of the approach together with the applicability to other domains. Beyond the direct impact on environmental health studies, the description of the evaluation approach can guide researchers in making SW technologies more accessible to domain experts through usability studies.

The Taxonomy Mobile Learning Applications in Higher Institutions of Learning in Ugandan Universities: A Case of Kabale University, Uganda

Since the use of mobile devices outpaces that of laptops and desktop computers today, the usability of these mobile devices is an important consideration. When mobile learning (a new kind of electronic learning) takes shape, bringing an important feature of mobility, the trend expands deeper into teaching and learning. Usability describes the quality characteristics of software product usage; hence, usability testing is a crucial concern in developing companies for the success of product deployment and use. The vast majority of existing usability evaluation approaches were created for desktop software development. As a result, currently existing models do not specifically address mobile learning, presenting a gap that we aimed to remedy. The study developed a model that estimates usability as a function of aggregated usability influencing factors. To provide a more comprehensive model, the proposed model includes essential features from other accessible models and incorporates the majority of those that assist mobile learning. A mobile learning prototype application was designed, tested, and installed to evaluate the efficiency of the developed model, coupled with a task list for objective research. Using a sophisticated statistical technique, the feedback from the experiment and survey was then utilized to assess and validate the prototype application in terms of high, average, or low usability. The findings act as guides for eLearning developing businesses to create more relevant mobile learning applications with high levels of usability.

A Suggested Integrated Approach for the Evaluation of Remote Intranet Usability

As the societal impacts of the Covid-19 progress, so do the workplace and business challenges the pandemic has brought on. It is now crucial to conduct user testing to measure and optimize the usability of corporate Intranets under pandemic conditions. This paper suggests and validates an integrated approach for Intranet remote usability evaluation validated by user testing in a work from home context during a pandemic. The main contribution of the current research is the discovery of some users preferences specific to work from home workers during a pandemic not previously addressed in the literature on intranets.

An Audio Personal Health Library of Clinic Visit Recordings for Patients and Their Caregivers (HealthPAL): User-Centered Design Approach.

BackgroundProviding digital recordings of clinic visits to patients has emerged as a strategy to promote patient and family engagement in care. With advances in natural language processing, an opportunity exists to maximize the value of visit recordings for patients by automatically tagging key visit information (eg, medications, tests, and imaging) and linkages to trustworthy web-based resources curated in an audio-based personal health library.ObjectiveThis study aims to report on the user-centered development of HealthPAL, an audio personal health library.MethodsOur user-centered design and usability evaluation approach incorporated iterative rounds of video-recorded sessions from 2016 to 2019. We recruited participants from a range of community settings to represent older patient and caregiver perspectives. In the first round, we used paper prototypes and focused on feature envisionment. We moved to low-fidelity and high-fidelity versions of the HealthPAL in later rounds, which focused on functionality and use; all sessions included a debriefing interview. Participants listened to a deidentified, standardized primary care visit recording before completing a series of tasks (eg, finding where a medication was discussed in the recording). In the final round, we recorded the patients’ primary care clinic visits for use in the session. Findings from each round informed the agile software development process. Task completion and critical incidents were recorded in each round, and the System Usability Scale was completed by participants using the digital prototype in later rounds.ResultsWe completed 5 rounds of usability sessions with 40 participants, of whom 25 (63%) were women with a median age of 68 years (range 23-89). Feedback from sessions resulted in color-coding and highlighting of information tags, a more prominent play button, clearer structure to move between one’s own recordings and others’ recordings, the ability to filter recording content by the topic discussed and descriptions, 10-second forward and rewind controls, and a help link and search bar. Perceived usability increased over the rounds, with a median System Usability Scale of 78.2 (range 20-100) in the final round. Participants were overwhelmingly positive about the concept of accessing a curated audio recording of a clinic visit. Some participants reported concerns about privacy and the computer-based skills necessary to access recordings.ConclusionsTo our knowledge, HealthPAL is the first patient-centered app designed to allow patients and their caregivers to access easy-to-navigate recordings of clinic visits, with key concepts tagged and hyperlinks to further information provided. The HealthPAL user interface has been rigorously co-designed with older adult patients and their caregivers and is now ready for further field testing. The successful development and use of HealthPAL may help improve the ability of patients to manage their own care, especially older adult patients who have to navigate complex treatment plans.

A novel approach for usability evaluation of mobile applications

Mobile applications (apps) have been massively developed and released in recent years. In this competitive market, the users’ adoption of the app or satisfaction greatly depends on its usability. Therefore, tech companies and app developers are looking for quick and easy approaches to evaluate the usability of their products early in the design cycle. The objective of this study was to introduce a novel approach combining the capabilities of the gesture-level model (GLM) and heuristic evaluation to assess the usability of mobile apps in early stages of the design cycle and without the need for costly and time-consuming user testing. The approach can be run on a graphical user interface and includes uncertainties regarding user skill level and distraction. Preliminary results suggested that this approach can be used for usability evaluation of mobile apps; however, its validity and generalizability to other domains need to be validated in the future with human-subject experiments.

Towards a prior validation of a model-based approach for mobile usability evaluation

Towards a prior validation of a model-based approach for mobile usability evaluation

A Heuristic Based Approach for Usability Evaluation of Academic Portals

A Heuristic Based Approach for Usability Evaluation of Academic Portals

A systematic review of software usability studies

The aim of this review is to summarize, analyze various research studies and identify different research gaps regarding usability standards and models, usability evaluation methods, usability metric, usability at different phases of software development life cycle and application domains of usability. This systematic review of usability studies between 1990 and 2016 has been conducted and 150 studies are identified. We conclude that researchers have not reached at consensus w.r.t. software usability models. We identify that Efficiency, Effectiveness, Satisfaction, and Learnability are commonly addressed attributes in various existing software usability models and standards. Further, developers do not have sufficient knowledge to decide the appropriate usability evaluation method to use in given domain. On the contrary, Usability Testing, Heuristic Evaluation and Questionnaire are identified frequently used methods for usability evaluation. Our findings investigate different metrics and measurement approaches used for usability estimation. But, current methods for usability measurement in practice do not include all ISO and ANSI defined aspects of usability into a single metric. Although, we identify studies concerning the integration of usability and software engineering into a single framework with generalizable results, their practical implementation is still missing and significantly needed. Conversely, this study highlights the fact that around 71% of studies address usability related issues during Design-Phase of software development life cycle. At present, usability issues have been identified in various domains but around 33.82% of studies identify that usability evaluation approach is widely used in Web-Domain.

Improving Online Teaching in a Required Geriatrics Clerkship Using Heuristic Evaluation

We demonstrate conducting heuristic evaluation to improve the usability of online medical education using an instrument specifically designed for e-learning. Specifically, we describe the process and outcomes of implementing heuristic evaluation to enhance a newly developed online geriatric module. The result of this implementation indicated the feasibility and usefulness of incorporating HE, as a relatively low-cost usability evaluation approach, into the refinement of e-learning systems in medical education. Lessons learned from this implementation and future steps are presented at the end.

A Heuristic Based Approach for Usability Evaluation of Academic Portals

A Heuristic Based Approach for Usability Evaluation of Academic Portals

Usability Evaluation Approach of Educational Resources Software Using Mixed Intelligent Optimization

Aiming at the problems of strong subjectivity and uncertain fuzziness of attribute weights in the software usability evaluation approach, an evaluation approach based on mixed intelligent optimization was proposed, which combines subjective and objective methods to measure software usability for educational resources software. Firstly, the usability evaluation index system of educational resources software was established, and the basic probability assignment was generated by the interval method from the historical sample data. Then the weight optimization problem was adapted to the smooth optimization problem by the maximum entropy function method, and the hybrid social cognitive optimization (HSCO) algorithm was introduced to solve the optimal weights of evidence. Finally, the software usability level was fused by DS evidence theory. The experimental results show that the educational resources software usability evaluation approach can objectively and truly reflect the usability of the software. It provides an efficient way to evaluate the usability of the software.

An Enhanced Web Mining Approach for Product Usability Evaluation in Feature Fatigue Analysis using LDA Model and Association Rule Mining with Fruit Fly Algorithm

Objectives: The rapid growth of E-commerce emphasizes the customers' attention on product purchases through large number of websites. The customers tend to buy the products with more and best features, but those features fails to satisfy their expectations. This limitation is termed as Feature Fatigue (FF) in product usability analysis in web mining. The objective of this novel approach is to evaluate the product usability effectively and supports the designers to make decisions in future. Methods: The consumers' reviews are collected from various E-commerce websites using web crawler. These review sentences are preprocessed by removing the stop words and stemming. The synonym dictionary is created from the preprocessed sentences. Findings: In order to attain usability evaluation frequent item set is identified by improved Apriori algorithm and the association rule is generated. Finally the product capability is evaluated effectively in the FF analysis using analyzed features. Improvements: The product reviews are gathered from the E-commerce websites to analyze the feature usability of that product and obtain 95% accuracy. The feature analysis report helps the manufacture to alleviate the FF by balancing the capability and usability of the product.

A new usability evaluation approach for touch screen mobile devices

Satisfaction of new expectations such as product usability and customer feelings are increasing the value of mobile electronic devices. In this study, cell phones, which are widely used among all touch screen mobile devices, are selected and various features which affect the usability of these products are researched in analytical network process (ANP)-technique for order preference by similarity to ideal solution-based (TOPSIS) Taguchi method. With the participation of fifteen subjects, usability tests are performed for four equivalent brands of touch screen phones. The relationships and interactions between the criteria affecting the usability of the touch screen phones and the results of the questionnaires are evaluated by ANP. Also, ANP is combined into TOPSIS-based Taguchi method and the most usable touch screen mobile phones with respect to optimal criteria levels are determined with a proposed integrated approach. This study helps the touch screen mobile phone designers focus on the usability that contribute most to the desirable product options. The proposed ANP-TOPSIS-based Taguchi method provides an effective way for facilitating the consumer oriented touch screen mobile phone design process.

Usability Evaluation Approach for VLS: A Conceptual Framework

E-learning software systems have been proposed by many researchers and academics as alternative platform for pedagogy other than traditional face-to-face. One of such information systems (IS) is Virtual learning Systems (VLS). VLS are becoming an increasingly common form of education due to the need for a platform that provides ability to connect people with required sets of skills, regardless of their location in the world. However, user satisfaction has always been a major factor in the success of software, regardless of whether the software is closed proprietary or open source software (OSS). Although user-centred designs are gaining recognition among virtual learning systems (VLS) community, many design scenarios still do not incorporate usability as one of their primary goals. Accordingly, many individuals believe that if VLS was more usable, its popularity would increase tremendously. Although there are strong usability models for IS, there is still potential to improve the usability of VLS. The usability assessment of VLS projects is an area where relatively little research has been conducted, and, accordingly, the main contribution of this work is a methodology that evaluates the usability maturity of a VLS project. Consequently, the study presents a usability maturity model that is aimed at usability-related issues for virtual learning system projects. The model to be developed is a response to a need for measuring the extent to which virtual learning systems software projects support usability. Specifically, it is intended for assessing and improving the usability aspect in VLS software development. The deductions from the research findings will be especially useful to e-learning practitioners, ICT and Education professionals and policy makers and researchers in Virtual Learning Environments.

E‐commerce websites for developing countries – a usability evaluation framework

PurposeThe purpose of this paper is to develop a methodological usability evaluation approach for e‐commerce websites in developing countries.Design/methodology/approachA multi‐faceted usability evaluation of three Jordanian e‐commerce websites was used, where three usability methods (user testing, heuristic evaluation and web analytics) were applied to the sites.FindingsA four‐step approach was developed to facilitate the evaluation of e‐commerce sites, mindful of the advantages and disadvantages of the methods used in identifying specific usability problems.Research limitations/implicationsThe approach was developed and tested using Jordanian users, experts and e‐commerce sites. The study compared the ability of the methods to detect problems that were present, however, usability issues not present on any of the sites could not be considered when creating the approach.Practical implicationsThe approach helps e‐commerce retailers evaluate the usability of their websites and understand which usability method(s) best matches their need.Originality/valueThis research proposes a new approach for evaluating the usability of e‐commerce sites. A novel aspect is the use of web analytics (Google Analytics software) as a component in the usability evaluation in conjunction with heuristics and user testing.

Usability Study of a Computer-Based Self-Management System for Older Adults with Chronic Diseases

BackgroundUsability can influence patients’ acceptance and adoption of a health information technology. However, little research has been conducted to study the usability of a self-management health care system, especially one geared toward elderly patients.ObjectiveThis usability study evaluated a new computer-based self-management system interface for older adults with chronic diseases, using a paper prototype approach.MethodsFifty older adults with different chronic diseases participated. Two usability evaluation methods were involved: (1) a heuristics evaluation and (2) end-user testing with a think-aloud testing method, audio recording, videotaping, and interviewing. A set of usability metrics was employed to determine the overall system usability, including task incompletion rate, task completion time, frequency of error, frequency of help, satisfaction, perceived usefulness, and perceived ease of use. Interviews were used to elicit participants’ comments on the system design. The quantitative data were analyzed using descriptive statistics and the qualitative data were analyzed for content.ResultsThe participants were able to perform the predesigned self-management tasks with the current system design and they expressed mostly positive responses about the perceived usability measures regarding the system interface. However, the heuristics evaluation, performance measures, and interviews revealed a number of usability problems related to system navigation, information search and interpretation, information presentation, and readability. Design recommendations for further system interface modifications were discussed.ConclusionsThis study verified the usability of the self-management system developed for older adults with chronic diseases. Also, we demonstrated that our usability evaluation approach could be used to quickly and effectively identify usability problems in a health care information system at an early stage of the system development process using a paper prototype. Conducting a usability evaluation is an essential step in system development to ensure that the system features match the users’ true needs, expectations, and characteristics, and also to minimize the likelihood of the users committing user errors and having difficulties using the system.

The effective combination of hybrid usability methods in evaluating educational applications of ICT: Issues and challenges

This paper focuses on usability evaluation of information and communication technologies applications in education (ICTE applications). Various classes of teaching and learning systems are discussed in terms of technologies used and pedagogical approaches. Their usability is analyzed according to various dimensions and the impact of system usability on the learning effectiveness is studied. We argue that various classes of ICTE applications such as multimedia/hypermedia applications, open educational environments and CSCL environments, based on different theoretical perspectives, require fundamentally different approaches in evaluating their usability. The paper is structured as follows: an overview of different usability evaluation approaches is presented first, followed by a discussion on applicability of these techniques in various categories of teaching and learning computer systems. Typical case studies that engage both usability experts and users themselves (students and teachers) are also discussed. The objective is to describe both the methods, and the way to apply them effectively in order to certify the usability of an ICTE application with respect to its teaching and learning objectives.

Usability evaluation of anti-phishing toolbars

Phishing is considered as one of the most serious threats for the Internet and e-commerce. Phishing attacks abuse trust with the help of deceptive e-mails, fraudulent web sites and malware. In order to prevent phishing attacks some organizations have implemented Internet browser toolbars for identifying deceptive activities. However, the levels of usability and user interfaces are varying. Some of the toolbars have obvious usability problems, which can affect the performance of these toolbars ultimately. For the sake of future improvement, usability evaluation is indispensable. We will discuss usability of five typical anti-phishing toolbars: built-in phishing prevention in the Internet Explorer 7.0, Google toolbar, Netcraft Anti-phishing toolbar and SpoofGuard. In addition, we included Internet Explorer plug-in we have developed, Anti-phishing IEPlug. Our hypothesis was that usability of anti-phishing toolbars, and as a consequence also security of the toolbars, could be improved. Indeed, according to the heuristic usability evaluation, a number of usability issues were found. In this article, we will describe the anti-phishing toolbars, we will discuss anti-phishing toolbar usability evaluation approach and we will present our findings. Finally, we will propose advices for improving usability of anti-phishing toolbars, including three key components of anti-phishing client side applications (main user interface, critical warnings and the help system). For example, we found that in the main user interface it is important to keep the user informed and organize settings accordingly to a proper usability design. In addition, all the critical warnings an anti-phishing toolbar shows should be well designed. Furthermore, we found that the help system should be built to assist users to learn about phishing prevention as well as how to identify fraud attempts by themselves. One result of our research is also a classification of anti-phishing toolbar applications.