Usability Evaluation Framework Research Articles

Developing a conceptual framework for usability evaluation of cartographic design principles in Mixed Reality applications

Developing a conceptual framework for usability evaluation of cartographic design principles in Mixed Reality applications

Expert Review on Usefulness of an Integrated Checklist-based Mobile Usability Evaluation Framework

Previous mobile usability studies are only pertinent in the context of ergonomics, physical user interface, and mobility aspects. In addition, much of the previous mobile usability conception was built on desktop computing measurements, such as desktop and web application checklists, or scarcely addressed the mobile user interface. Moreover, the studies focus mainly on interface features for desktop applications and do not reflect comprehensive mobile interface features such as navigation drawers and spinners. Therefore, conducting usability evaluation using conventional usability measurement would result in irrelevant results. In addition, the resulting works are tailored for usability testing, which requires highly skilled evaluators and usability specialists (e.g., usability testers and user experience designers), who are rarely integrated into a development team. The lack of expertise could lead to unreliable usability evaluations. This paper presents a review from industrial experts on a comprehensive and feasible usability evaluation framework developed in our previous work. The framework is dedicated to smartphone apps, which integrate evaluator skills and design concerns. However, there is no evidence of its usefulness in practice. Therefore, the usefulness of the framework measurement for evaluating apps' usability in the eyes of non-usability specialists is empirically assessed in this paper through an expert review. The expert review involved eleven industrial developers and was complemented by a semi-structured interview. The method is replicated in comparison with a framework from another study. The findings show that the formulated framework significantly outperformed the framework (p = 0.0286) from other studies with large effect sizes (r = 1.81) in terms of usefulness.

Designing User Interfaces for Illiterate and Semi-Literate Users: A Systematic Review and Future Research Agenda

Using a systematic literature review, this review examines existing studies on the design, usability, user experience (UX), and adoption of computer applications designed for illiterate and semi-literate users. A total of 45 articles were reviewed from an initial set of 527 articles following an inclusion-exclusion criterion. As outcomes, in addition to synthesizing the present research activities, this review revealed a total of 16 design considerations or guidelines that may improve the usability, user experience, and adoption of software systems to illiterate and semi-literate people. The revealed design considerations were also investigated concerning the country context and application platform. Finally, a set of good rules categorized into five perspectives (uses of text, interaction pattern, uses of pictography, uses of audio-video, and info-architecture) were revealed to design and evaluate software systems to make them more usable to the illiterate and semi-literate users. The review also reveals future research opportunities that include: exploring the design considerations with respect to context-dependent applications; investigating the design requirements for public IT (e.g., vending machines and kiosks) systems; adopting the semiotic concept in UI design; developing a new usability evaluation framework; and exploring the effects of multi-platform and cross-country app design. Thus, the implication of this study is to discover the research gaps and provide future research directions aiming to design applications for illiterate and semi-literate users with enhanced usability and UX.

A framework for usability evaluation of automated driving human-machine interface in context of UAE

As a future mobility enabler, vehicle teleoperation offers a viable option for efficiently leveraging the benefits of driverless cars, whilst fully automated cars (SAE 5) are not completely viable. Despite the huge amount of research on self-driving technology, the notion of riding in a fully automated vehicle is still sceptical. Hence necessitating the presence of a remote operator who tele monitors the vehicle and acquire control in case of failure beyond the capabilities of vehicle automation, such as misuse, disuse or mode confusion and ensuring safety and reliability. Keeping the current inclination towards automated vehicles and the launch of the United Arab Emirates” (UAE) first driverless, autonomous 12-seater shuttle, the researcher has suggested a low-fidelity basic prototype. To connect the vehicle’s automation and human teleoperation, the researcher has proposed a unique and interactive framework of Human-Machine Interface (HMI) to cater teleoperation in the context of automated vehicles (AVs) in the UAE. The design is optimised for the teleoperation of a highly automated shuttle by a public transportation control centre based on a logical and careful inquiry. After deriving a detailed analysis of requirements, a prototype is then created and refined until an interactive user-interface emerges. Fifteen control station experts are presented with a prototype for evaluation in a regular manner and solicited to work out three scenarios with disruptions in the system. The prototype is then evaluated using questionnaires and structured interviews. The results validate the suggested HMI design for the remote operation of a highly automated vehicle. Participants’ feedback shows a particularly high level of satisfaction in terms of perceived capacity, feasibility, usability approval and situational responsiveness. The findings also provide valuable information for an advanced interactive HMI design and pave the way for potential future directions in further research as well.

Evolving a conceptual framework and developing a new questionnaire for usability evaluation of blended learning programs in health professions education

Background: Blended learning programs (BLPs) have been widely adopted across health professions education (HPE). To bolster their impact on learning outcomes, the usability of BLPs should be rigorously evaluated. However, there is a lack of reliable and validated tools to appraise this dimension of BLPs within HPE. The purpose of this investigation was to evolve a conceptual framework for usability evaluation in order to initially develop the Blended Learning Usability Evaluation – Questionnaire (BLUE-Q). Methods: After the completion of a scoping review, we conducted a qualitative descriptive study with seven purposefully selected international experts in usability and learning program evaluation. Individual interviews were conducted via videoconferencing, transcribed verbatim, and analyzed through thematic analysis. Results: Three themes were identified: (1) Consolidation of the multifaceted ISO definition of usability in BLPs within HPE; (2) Different facets of usability can assess different aspects of BLPs; (3) Quantitative and qualitative data are needed to assess the multifaceted nature of usability. The first theme adds nuance to a previously established HPE-focused usability framework, and introduces two new dimensions: ‘pedagogical usability’ and ‘learner motivation.’ The latter two provide guidance on structuring BLP evaluations within HPE. From this followed the development of the BLUE-Q, a new questionnaire that includes 55 Likert scale items and 6 open-ended questions. Conclusions: Usability is an important dimension of BLPs and must be examined to improve the quality of these interventions in HPE. As such, we developed a new questionnaire, solidly grounded in theory and the expertise of international scholars, currently under validation.

Usability evaluation of mHealth apps for elderly individuals: a scoping review

BackgroundUsability is a key factor affecting the acceptance of mobile health applications (mHealth apps) for elderly individuals, but traditional usability evaluation methods may not be suitable for use in this population because of aging barriers. The objectives of this study were to identify, explore, and summarize the current state of the literature on the usability evaluation of mHealth apps for older adults and to incorporate these methods into the appropriate evaluation stage.MethodsElectronic searches were conducted in 10 databases. Inclusion criteria were articles focused on the usability evaluation of mHealth apps designed for older adults. The included studies were classified according to the mHealth app usability evaluation framework, and the suitability of evaluation methods for use among the elderly was analyzed.ResultsNinety-six articles met the inclusion criteria. Research activity increased steeply after 2013 (n = 92). Satisfaction (n = 74) and learnability (n = 60) were the most frequently evaluated critical measures, while memorability (n = 13) was the least evaluated. The ratios of satisfaction, learnability, operability, and understandability measures were significantly related to the different stages of evaluation (P < 0.05). The methods used for usability evaluation were questionnaire (n = 68), interview (n = 36), concurrent thinking aloud (n = 25), performance metrics (n = 25), behavioral observation log (n = 14), screen recording (n = 3), eye tracking (n = 1), retrospective thinking aloud (n = 1), and feedback log (n = 1). Thirty-two studies developed their own evaluation tool to assess unique design features for elderly individuals.ConclusionIn the past five years, the number of studies in the field of usability evaluation of mHealth apps for the elderly has increased rapidly. The mHealth apps are often used as an auxiliary means of self-management to help the elderly manage their wellness and disease. According to the three stages of the mHealth app usability evaluation framework, the critical measures and evaluation methods are inconsistent. Future research should focus on selecting specific critical measures relevant to aging characteristics and adapting usability evaluation methods to elderly individuals by improving traditional tools, introducing automated evaluation tools and optimizing evaluation processes.

A Cognitive Style Based Framework for Usability Evaluation of Online Lecturing Platforms – A Case Study on Zoom and Teams

Recently, Zoom and Microsoft Teams have emerged as the most common tools for lecturing activities in higher education institutions. The fact that these platforms were not originally developed for educational purposes resulted in a significant reduction in learning effectiveness, especially in STEM subjects that require a lot of practical knowledge. Despite the enhancements and updates made to these platforms, they have failed to meet the learning effectiveness criteria. A major hindering to learning effectiveness on such systems can be traced not only to the absence of learning-related features but also to their usability. In this research, we propose a framework for evaluating the usability of online lecturing platforms. The proposed framework introduces a new set of heuristics that are specifically designed for lecturing activities and considers VAK learning styles. Since students with different learning styles learn differently, we developed the first cognitive style-based usability heuristics that can reliably assess the usability of online learning environments for various types of learners. We present a case study that uses the suggested evaluation framework to assess the usability of Zoom and Teams in higher education lecturing activities. The findings of this study, however, can be applied to almost any online lecture platform.

A Coding Framework for Usability Evaluation of Digital Health Technologies.

Several studies have reported low adherence and high resistance from clinicians to adopt digital health technologies into clinical practice, particularly the use of computer-based clinical decision support systems. Poor usability and lack of integration with the clinical workflow have been identified as primary issues. Few guidelines exist on how to analyze the collected data associated with the usability of digital health technologies. In this study, we aimed to develop a coding framework for the systematic evaluation of users' feedback generated during focus groups and interview sessions with clinicians, underpinned by fundamental usability principles and design components. This codebook also included a coding category to capture the user's clinical role associated with each specific piece of feedback, providing a better understanding of role-specific challenges and perspectives, as well as the level of shared understanding across the multiple clinical roles. Furthermore, a voting system was created to quantitatively inform modifications of the digital system based on usability data. As a use case, we applied this method to an electronic cognitive aid designed to improve coordination and communication in the cardiac operating room, showing that this framework is feasible and useful not only to better understand suboptimal usability aspects, but also to recommend relevant modifications in the design and development of the system from different perspectives, including clinical, technical, and usability teams. The framework described herein may be applied in other highly complex clinical settings, in which digital health systems may play an important role in improving patient care and enhancing patient safety.

A novel model usability evaluation framework (MUsE) for explainable artificial intelligence

When it comes to complex machine learning models, commonly referred to as black boxes, understanding the underlying decision making process is crucial for domains such as healthcare and financial services, as well as when they are used in connection with safety critical systems such as autonomous vehicles. As a result, interest in explainable artificial intelligence (xAI) tools and techniques has increased in recent years. However, the user experience (UX) effectiveness of existing xAI frameworks, especially concerning algorithms that work with data as opposed to images, is still an open research question. In order to address this gap, we examine the UX effectiveness of the Local Interpretable Model-Agnostic Explanations (LIME) xAI framework, one of the most popular model agnostic frameworks found in the literature, with a specific focus on its performance in terms of making tabular models more interpretable. In particular, we apply several state of the art machine learning algorithms on a tabular dataset, and demonstrate how LIME can be used to supplement conventional performance assessment methods. Based on this experience, we evaluate the understandability of the output produced by LIME both via a usability study, involving participants who are not familiar with LIME, and its overall usability via a custom made assessment framework, called Model Usability Evaluation (MUsE), which is derived from the International Organisation for Standardisation 9241-11:2018 standard.

Novel Multi-Perspective Usability Evaluation Framework for Selection of Open Source Software Based on BWM and Group VIKOR Techniques

Increasing demand for open-source software (OSS) has raised the value of efficient selection in terms of quality; usability is an essential quality factor that significantly affects system acceptability and sustainability. Most large and complex software packages partitioned across multiple portals and involve many users — each with their role in the software package; those users have different perspectives on the software package, defined by their knowledge, responsibilities, and commitments. Thus, a multi-perspective approach has been used in usability evaluation to overcome the challenge of inconsistency between users’ perspectives; the inconsistency challenge would lead to an ill-advised decision on the selection of a suitable OSS. This study aimed to assist the public and private organizations in evaluating and selecting the most suitable OSS. The evaluation of the OSS software packages to choose the best one is a challenging task owing to (a) multiple evaluation criteria, (b) criteria importance, and (c) data variation; thus, it is considered a sophisticated multi-criteria decision making (MCDM) problem; moreover, the multi-perspective usability evaluation framework for OSS selection lacks in the current literature. Hence, this study proposes a novel multi-perspective usability evaluation framework for the selection of OSS based on the multi-criteria analysis. Integration of best-worst method (BWM) and VIKOR MCDM techniques has been used for weighting and ranking OSS alternatives. BWM is utilized for weighting of evaluation criteria, whereas VIKOR is applied to rank OSS-LMS alternatives. Individual and group decision-making contexts, and the internal and external groups aggregation were used to demonstrate the efficiency of the proposed framework. A well-organized algorithmic procedure is presented in detail, and a case study was examined to illustrate the validity and feasibility of the proposed framework. The results demonstrated that BWM and VIKOR integration works effectively to solve the OSS software package benchmarking/selection problems. Furthermore, the ranks of OSS software packages obtained from the VIKOR internal and external group decision making were similar; the best OSS-LMS based on the two ways was ‘Moodle’ software package. Among the scores of groups in the objective validation, significant differences were identified; this indicated that the ranking results of internal and external VIKOR group decision making were valid, which pointed to the validation of the framework.

Evaluating Usability of Academic Websites through a Fuzzy Analytical Hierarchical Process

In the higher education sector, there is a growing trend to offer academic information to users through websites. Contemporarily, the users (i.e., students/teachers, parents, and administrative staff) greatly rely on these websites to perform various academic tasks, including admission, access to learning management systems (LMS), and links to other relevant resources. These users vary from each other in terms of their technological competence, objectives, and frequency of use. Therefore, academic websites should be designed considering different dimensions, so that everybody can be accommodated. Knowing the different dimensions with respect to the usability of academic websites is a multi-criteria decision-making (MCDM) problem. The fuzzy analytic hierarchy process (FAHP) approach has been considered to be a significant method to deal with the uncertainty that is involved in subjective judgment. Although a wide range of usability factors for academic websites have already been identified, most of them are based on the judgment of experts who have never used these websites. This study identified important factors through a detailed literature review, classified them, and prioritized the most critical among them through the FAHP methodology, involving relevant users to propose a usability evaluation framework for academic websites. To validate the proposed framework, five websites of renowned higher educational institutes (HEIs) were evaluated and ranked according to the usability criteria. As the proposed framework was created methodically, the authors believe that it would be helpful for detecting real usability issues that currently exist in academic websites.

Evaluation of Usability Dimensions of Smartphone Applications

This study analyses different techniques used for evaluation of various usability dimensions of software applications (apps) being used on the smartphones. The scope of this study is to evaluate various aspects of the usability techniques employed in the domain of smartphone apps. Usability assessment methodologies are evaluated for different types of applications running on different operating systems like Android, Blackberry and iOS, etc. Usability evaluation techniques and methodologies with respect to usability heuristics like field experiments, laboratory experiments models and usability standards are discussed in detail. The issues for evaluation of usability of smartphone apps are identified by considering limitations and areas of improvement outlined in the contemporary literature. A conceptual framework for usability evaluation of smartphone apps is also designed which would be validated through experimentation in the thesis work. This study is particularly useful to comprehend usability issues and their likely remedies to produce high quality smartphone apps.

Usability evaluation framework for domain-specific language

Software engineers are increasingly taking advantage of new methods to improve software quality. The use of languages developed for specific domains, which in the literature are known as Domain-Specific Languages (DSLs), has grown in the past years. Although several experimental studies that subjectively evaluate usability of these languages can be found in the literature, few of them have taken advantage of applying Human-Computer Interaction (HCI) techniques in those evaluations. Therefore, the main goals of this paper are to present a usability evaluation framework for DSLs, called Usa-DSL, and to show the evaluation of the framework through a Focus Group method. The evaluation was performed by seven specialists that discussed the framework usability and suggested some modifications of our initial proposal. The specialists recommendations were incorporated in the final framework presented in this paper.

A multi-level usability evaluation of mobile health applications: A case study.

To report a methodological approach for the development of a usable mHealth application (app). This work was guided by a 3-level stratified view of health information technology (IT) usability evaluation framework. We first describe a number of methodologies for operationalizing each level of the framework. Following the description of each methodology, we present a case study which illustrates the use of our preferred methodologies for the development of a mHealth app. At level 1 (user-task), we applied a card sorting technique to guide the information architecture of a mobile HIV symptom self-management app, entitled mVIP. At level 2 (user-task-system), we conducted a usability evaluation of mVIP in a laboratory setting through end-user usability testing and heuristic evaluation with informatics experts. At level 3 (user-task-system-environment), usability of mVIP was evaluated in a real-world setting following the use of the app during a 3-month trial. The 3-level usability evaluation guided our work exploring in-depth interactions between the user, task, system, and environment. Integral to the findings from the 3-level usability evaluation, we iteratively refined the app's content, functionality, and interface to meet the needs of our intended end-users. The stratified view of the health IT usability evaluation framework is a useful methodological approach for the design, development, and evaluation of mHealth apps. The methodological recommendations for using the theoretical framework can inform future usability studies of mHealth apps.

UNSCALE: A Fuzzy-based Multi-criteria Usability Evaluation Framework for Measuring and Evaluating Library Websites

ABSTRACTUsability evaluation of websites has been one of the most important activities to determine user acceptance and to enrich the overall quality of the website. This research, drawing upon above concept, investigates the issues of website usability and proposes a fuzzy-based framework for measuring and evaluating the usability of websites, particularly library websites. The findings of the usability criteria elicitation, expert reviews, survey, and statistical analysis resulted in a comprehensive list of seven usability dimensions along with 20 measuring items which formed the basis of the evaluation framework abbreviate UNSCALE. The inputs of experts in the related field were used to probe weights of key dimensions and measuring items with the motive of examining their level of contribution to the usability index system. The study employed fuzzy theories along with the extent analysis, fuzzy analytic hierarchy process method with the expectation of scrutinizing the relative weightings. The framework was developed including eight cyclical steps for measuring and evaluating website usability by employing the fuzzy comprehensive evaluation method. It can be used to evaluate the overall usability score and also scores regarding each evaluation dimension. The framework was tested for its applicability and practicality on university library websites in Sri Lanka, a developing country with fast-growing internet accessibility. The findings of this research indicate that the proposed framework is useful for librarians in developing more useful library websites. On the other hand, it will provide vital insights to better appraise the outcomes of website usability for the scholars and researchers.

A Framework for Usability Evaluation in EHR Procurement.

Usability should be considered already by the procuring organizations when selecting future systems. In this paper, we present a framework for usability evaluation during electronic health record (EHR) system procurement. We describe the objectives of the evaluation, the procedure, selected usability attributes and the evaluation methods to measure them. We also present the emphasis usability had in the selection process. We do not elaborate on the details of the results, the application of methods or gathering of data. Instead we focus on the components of the framework to inform and give an example to other similar procurement projects.

A model for evaluating the security and usability of e-banking platforms

Convenience and the ability to perform advanced transactions encourage banks clients to use e-banking systems. As security and usability are two growing concerns for e-banking users, banks have invested heavily in improving their web portals security and user experience and trust in them. Despite considerable efforts to evaluate particular security and usability features in e-banking systems, a dedicated security and usability evaluation model that can be used as a guide in the development of e-banking assets remains much less explored. To build a comprehensive security and usability evaluation framework, we first extract security and usability evaluation metrics from the conducted literature review and then include several other evaluation metrics that were not previously identified in the literature. We then propose a structured inspection model for thoroughly evaluating the usability and security of internal and external e-banking assets. We argue that the proposed e-banking security and usability evaluation frameworks in the literature in addition to the existing standards of security best practices (e.g., NIST and ISO) are by no means comprehensive and lack some essential and key evaluation metrics that are of particular interest to e-banking portals. In order to demonstrate the inadequacy of existing models, we use the proposed framework to evaluate five major banks. The evaluation reveals several shortcomings in identifying both missing or incorrectly implemented security and privacy features. Our goal is to encourage other researchers to build upon our work.

Using a fuzzy comprehensive evaluation method to determine product usability: A test case.

BACKGROUND: In order to take into account the inherent uncertainties during product usability evaluation, Zhou and Chan [1] proposed a comprehensive method of usability evaluation for products by combining the analytic hierarchy process (AHP) and fuzzy evaluation methods for synthesizing performance data and subjective response data. This method was designed to provide an integrated framework combining the inevitable vague judgments from the multiple stages of the product evaluation process.OBJECTIVE AND METHODS: In order to illustrate the effectiveness of the model, this study used a summative usability test case to assess the application and strength of the general fuzzy usability framework. To test the proposed fuzzy usability evaluation framework [1], a standard summative usability test was conducted to benchmark the overall usability of a specific network management software. Based on the test data, the fuzzy method was applied to incorporate both the usability scores and uncertainties involved in the multiple components of the evaluation. Then, with Monte Carlo simulation procedures, confidence intervals were used to compare the reliabilities among the fuzzy approach and two typical conventional methods combining metrics based on percentages.RESULTS AND CONCLUSIONS: This case study showed that the fuzzy evaluation technique can be applied successfully for combining summative usability testing data to achieve an overall usability quality for the network software evaluated. Greater differences of confidence interval widths between the method of averaging equally percentage and weighted evaluation method, including the method of weighted percentage averages, verified the strength of the fuzzy method.

A Novel Website Quality and Usability Evaluation Framework for Online Shopping Websites

Objectives: The purpose of this research paper is aimed to determine the effect of subjective measures website quality and usability on its objective measures. A checklist of high level quality factors and sub factors, factors driving the customer to shop online is proposed and a new website quality and usability evaluation framework is developed to evaluate the quality and usability of online shopping websites from the user’s perspective. Methods: Structural equation modeling was applied to determine the subjective measures with the highest and lowest contributing factors for website quality and usability. A survey with 400 respondents was conducted to validate the proposed model. Findings: The Study has identified the important quality factors of the case study website Flipkart which makes it usable and successful. The results of the study can be used for designing websites with improved quality and usability. The study has significance impact on the economic and societal perspectives as the results will help the online shopping sites to improve their websites to improve the degree of user satisfaction and attract more customers to shop online.

Improving web user satisfaction by ensuring usability criteria compliance

Purpose – The purpose of this paper is to improve and facilitate the work of developers and usability evaluators by providing an adaptable and effective support. A well-defined set of criteria and a range of evaluation values for each criterion as well as a complete websites classification, will guide evaluators. A usability percentage and a list of prioritized criteria, adapted to the type of website by a new usability metric, will help developers to improve the website. This improvement will increase the degree of web user satisfaction. Design/methodology/approach – Having established and validated a new usability evaluation framework, several usability tools have been analyzed. None of them totally fulfills the requirements of the evaluation framework. As a result of being unable to customize any of them, a new one has been developed. A study of 42 enterprise websites in an economically depressed region of Europe was performed using the new tool. This study involved 42 evaluators and 118 web users. Users have evaluated the websites before and after the redesign. A end-user computing satisfaction model-based questionary was used to collect data about end-user satisfaction. The results validate the proposal. Findings – The study confirms that the proposed tool provides valuable information during the process of web development, evaluation and redesign. In adittion, it reveals that improving websites usability by ensuring criteria compliance has a positive effect on web users satisfaction. Originality/value – Unlike previous purposes, the proposed tool allows to evaluate any type of website with a well-defined set of evaluation criteria and specific criteria values. As outcomes, the tool provides the website usability degree and a list of criteria ordered by priority repair. These results are adapted to the specific type of website. This makes easier and more effective the redesign of the evaluated website.