Systematic Usability Evaluation Research Articles

Human-centered design of a novel soft exosuit for post-stroke gait rehabilitation

BackgroundStroke remains a major cause of long-term adult disability in the United States, necessitating the need for effective rehabilitation strategies for post-stroke gait impairments. Despite advancements in post-stroke care, existing rehabilitation often falls short, prompting the development of devices like robots and exoskeletons. However, these technologies often lack crucial input from end-users, such as clinicians, patients, and caregivers, hindering their clinical utility. Employing a human-centered design approach can enhance the design process and address user-specific needs.ObjectiveTo establish a proof-of-concept of the human-centered design approach by refining the NewGait® exosuit device for post-stroke gait rehabilitation.MethodsUsing iterative design sprints, the research focused on understanding the perspectives of clinicians, stroke survivors, and caregivers. Two design sprints were conducted, including empathy interviews at the beginning of the design sprint to integrate end-users’ insights. After each design sprint, the NewGait device underwent refinements based on emerging issues and recommendations. The final prototype underwent mechanical testing for durability, biomechanical simulation testing for clinical feasibility, and a system usability evaluation, where the new stroke-specific NewGait device was compared with the original NewGait device and a commercial product, Theratogs®.ResultsAffinity mapping from the design sprints identified crucial categories for stakeholder adoption, including fit for females, ease of donning and doffing, and usability during barefoot walking. To address these issues, a system redesign was implemented within weeks, incorporating features like a loop-backed neoprene, a novel closure mechanism for the shoulder harness, and a hook-and-loop design for the waist belt. Additional improvements included reconstructing anchors with rigid hook materials and replacing latex elastic bands with non-latex silicone-based bands for enhanced durability. Further, changes to the dorsiflexion anchor were made to allow for barefoot walking. Mechanical testing revealed a remarkable 10-fold increase in durability, enduring 500,000 cycles without notable degradation. Biomechanical simulation established the modularity of the NewGait device and indicated that it could be configured to assist or resist different muscles during walking. Usability testing indicated superior performance of the stroke-specific NewGait device, scoring 84.3 on the system usability scale compared to 62.7 for the original NewGait device and 46.9 for Theratogs.ConclusionThis study successfully establishes the proof-of-concept for a human-centered design approach using design sprints to rapidly develop a stroke-specific gait rehabilitation system. Future research should focus on evaluating the clinical efficacy and effectiveness of the NewGait device for post-stroke rehabilitation.

Enhanced System Usability Scale using the Software Quality Standard Approach

The objective of this paper is to improve the current System Usability Scale (SUS) and assess its applicability in the context of Learning Management Systems (LMS). The need to evaluate the usability of systems has become increasingly important in today's market, as it can have a significant impact on the user experience. In light of the COVID-19 pandemic, e-learning has become an essential tool for students, making LMS an appropriate research case study. Through a comprehensive literature review, it was discovered that SUS is the most widely used tool for evaluating system usability. However, SUS fails to satisfy some of the usability criteria outlined in ISO 9126 and ISO 9241-11. Therefore, this paper proposes an enhanced SUS model and its conceptual framework to address these limitations. The proposed model was validated using a case study approach, involving subject matter experts and software testing students, who evaluated the reliability of the enhanced SUS. Additionally, the existing and enhanced SUS models were evaluated based on an LMS case study and the results were used to calculate the enhanced SUS's reliability coefficient using Cronbach's alpha. The validation results show that the enhanced SUS has higher reliability with improved quality coverage compared to the original SUS. The proposed model has the potential to enhance the evaluation of system usability and, consequently, improve user experience.

TrainAR: An Open-Source Visual Scripting-Based Authoring Tool for Procedural Mobile Augmented Reality Trainings

Mobile Augmented Reality (AR) is a promising technology for educational purposes. It allows for interactive, engaging, and spatially independent learning. While the didactic benefits of AR have been well studied in recent years and commodity smartphones already come with AR capabilities, concepts and tools for a scalable deployment of AR are still missing. The proposed solution TrainAR combines an interaction concept, a didactic framework and an authoring tool for procedural AR training applications for smartphones. The contribution of this paper is the open-source visual scripting-based authoring tool of TrainAR in the form of a Unity Editor extension. With this approach, TrainAR allows non-programmer domain experts to create (“author”) their own procedural AR trainings by offering a customized editor, while at any time programmers may decide to utilize Unity’s full capabilities. Furthermore, utility and usability evaluations of several already developed TrainAR trainings (combined n = 317) show that TrainAR trainings provide utility in several contexts and are usable by the target groups. A systematic usability evaluation of the TrainAR Authoring Tool (n = 30) shows that it would be usable by non-programmer domain experts, though the learning curve depends on the media competency of the authors.

Evaluation and Optimization of In-Vehicle HUD Design by Applying an Entropy Weight-VIKOR Hybrid Method

Background: The interface design of in-vehicle head-up display (HUD) is an enlarging research area with interface usability as its core; usability reflects all perspectives of human—machine interaction and thus the evaluation and optimization of usability have multiple objectives. The evaluation and optimization of interface quality involved in usability are subjective and subconscious. Nevertheless, very little attention has been paid to these issues in optimizing usability across multiple objectives. Methods: In this paper, a hybrid scheme evaluation and optimization method based on entropy weight and VIKOR is proposed. First, according to the content of PSSUQ (Post Study System Usability Question), we have established a new usability evaluation system based on the characteristics of HUD. The entropy weight method was used to reduce the subjective factors of the decision-makers and to achieve the objective weight of each indicator. The VIKOR method was used for obtaining the order of alternate schemes and then the optimal interface design scheme was selected. Results: A case study was carried out to illustrate the applicability of the developed model in the usability evaluation of the HUD interface design. The results showed that scheme 1 was the optimized scheme, with minimal value of Si (0.141), Ri (0.119) and Qi (0.000) among the three schemes. When other decision-making methods were applied, the results showed that the optimized scheme was scheme 1, respectively, which verified the feasibility of the proposed method. The entropy—VIKOR model can be used to evaluate and optimize the HUD interface design effectively, which may serve as a reference for designers to achieve insights during the design process and scheme decision-making.

Usability and Vibration Analysis of a Low-Profile Automatic Powered Wheelchair to Motor Vehicle Docking System.

The QLX is a low-profile automatic powered wheelchair docking system (WDS) prototype developed to improve the securement and discomfort of wheelchair users when riding in vehicles. The study evaluates the whole-body vibration effects between the proposed QLX and another WDS (4-point tiedown system) following ISO 2631-1 standards and a systematic usability evaluation. Whole-body vibration analysis was evaluated in wheelchairs using both WDS to dock in a vehicle while riding on real-world surfaces. Also, participants rated the usability of each WDS while driving a wheelchair and while riding in a vehicle in driving tasks. Both WDSs showed similar vibration results within the vibration health-risk margins; but shock values below health-risk margins. Fifteen powered wheelchair users reported low task load demand to operate both WDS; but better performance to dock in vehicles with the QLX (p = 0.03). Also, the QLX showed better usability (p < 0.01), less discomfort (p's < 0.05), and greater security compared to the 4-point tiedown while riding in a vehicle (p's < 0.05). Study findings indicate that both WDS maintain low shock exposure for wheelchair users while riding vehicles, but a better performance overall to operate the QLX compared to the 4-point tiedown system; hence enhancing user's autonomy to dock in vehicles independently.

Mechatronic system usability evaluation

This study aimed to refine and validate a Mechatronic System Usability Evaluation (MSUE) questionnaire. A total of 626 users were selected using random sampling, from the area of West Thessaloniki, Greece. The validity of the questionnaire were tested with the content and construction validity method. The reliability of the MSUE questionnaire instrument were tested using test-retest and internal consistency method. As a result of the factor analysis, the scale remained at 25 items across five axes factors, namely efficiency, effectiveness, Satisfaction, ease of use, and ease of learn. The Cronbach Alpha coefficient for the entire scale was 0.819. The questionnaire was tested and the results was shown a suitable instrument to measure usability on mechatronic systems.

Usability Evaluation of in-Vehicle AR-HUD Interface Applying AHP-GRA

Usability is regarded to be a fundamental requirement for in-vehicle HMIs, and usability evaluation reflects the impact of the interface and the acceptance from the users. This study introduced a usability evaluation model of AR-HUD interface by applying grey analytic hierarchy process (AHP). First, based on the ameliorated PSSUQ (Post-Study System Usability Questionnaire), the usability evaluation system was modified and optimized according to the characteristics of AR-HUD. On this basis, the preference weights for evaluation indexes were calculated by AHP and the idea of group decision. Finally, the criteria of usability were integrated into grey relational degree by applying grey relational analysis (GRA) to obtain optimal design. A case study was conducted to demonstrate the applicability of the developed model to the usability evaluation of AR-HUD interface design. According to the existing AR-HUD interface design, 7 dimensions of design elements (A-G) and 18 interface prototypes (S1-S18) were selected by Taguchi orthogonal array test (TOAT). As the results indicated, the grey relational degree of S5 was 0.923, signifying that it was the optimal sample; and the results were also compared with entropy-TOPSIS to verify the feasibility of the proposed method. The grey-based AHP evaluation model can be used to evaluate the usability level of AR-HUD interface effectively, which may help designers achieve insights for design process and samples decision-making.

Affective Mobile Language Tutoring System for Supporting Language Learning.

Students often face difficulties and experience negative emotions toward second language learning. The affective tutoring system (ATS) is a next-generation learning approach that can detect the affective status of learning to increase performance. Therefore, for the purposes of this study, an innovative affective mobile language tutoring system (AMLTS) was designed to support Japanese language learning. The effects of AMLTS, along with asynchronous discussion, that were intended to improve performance, were examined using a triangulation method. To investigate the effect on emotion, the proposed AMLTS provides a virtual emotion agent that can interact with users and record emotional events, learning assessments, and the results of the interaction into a database. Learning effectiveness evaluations were conducted via two experiments: prototype evaluation and final evaluation. Sixty-three students, all beginners, were invited to use the AMLTS to learn Japanese. The research results show that the proposed AMLTS affective interaction design significantly improves learner engagement and performance. In the emotion feedback analysis and learning process, AMLTS helped students deepen their understanding of the content, enabled them to clearly understand the content, and to engage in peer interaction and experience positive emotions. In the evaluation of system usability, AMLTS reveals good usability for foreign language acquisition.

An augmented reality-based system for improving quality of services operations: a study of educational institutes

PurposeThis study aims to evaluate the usability of the augmented reality-based Evoke Education System (EES) to improve service operations in educational settings. The EES uses an animated character (Moe) to interact with children in a classroom by reproducing their teacher's movements and speech.Design/methodology/approachThis study uses a quantitative approach for the system usability evaluation. The ESS was evaluated by 71 children aged 6–8 years old, from two primary schools. After interacting with the EES, they completed a system usability questionnaire and participated in a knowledge acquisition test.FindingsThe knowledge acquisition test undertaken on the initial day showed statistically significant improvements for children taught with the EES, compared to children taught through traditional teaching approaches. However, the retest nine days later was not statistically significant (as only one school participated) due to low power. This study used confirmatory factor analysis (CFA), resulting in the identification of five essential factors (likeability, interactiveness, retention, effectiveness/attractiveness and satisfaction) that contribute to the EES's usability. The comparison with existing literature shows that these factors are consistent with the definition of system usability provided by the International Organization for Standardization and current academic literature in this field.Research limitations/implicationsThe findings presented in this study are based on the data from only two schools. The research can be extended by involving children from a greater number of schools. Mixed methods and qualitative research approaches can be used for future research in this area to generalise the results.Originality/valueThis study proposes an innovative augmented reality-based education system to help teachers deliver their key messages to the children in a fun way that can potentially increase their knowledge retention.

Systematic usability evaluation on two harnesses for a wearable chairless exoskeleton

Harnesses are used in wearable robots in safety, sports, medical, and industrial environments. Wearable robot harnesses for lower extremities should be designed to prevent musculoskeletal disorders caused by excessive strain on the spine and lower extremities when users wear these robots for prolonged periods. Therefore, this study compares and analyzes the usability of the two harnesses—the former (old design) and the latter (new design)—for the Hyundai chairless exoskeleton (CEX). In total, 27 participants took part in this study. The wearability, stability, and overall wearing satisfaction of the former and the latter harnesses were compared. We conducted an experiment where the participants wore the former and the latter harnesses while completing a series of tasks that resembled the ones performed by the workers on a front-end module assembling line of an automobile factory. After the experiment, the participants took part in a survey and semi-structured interviews. The data was analyses by t-test, correlation, and regression analyses. The survey results demonstrated that for the latter harness the cushioning sensation (thigh r = 0.539; calf r = 0.516) had the strongest effect on the overall wearing satisfaction. In addition, the latter harness scored higher in overall wear satisfaction than the previous harness. This study was able to verify the difference between the new and old designs according to the usability evaluation of the harness. The results of this study are considered to be helpful in design to improve the usability of the harness in the future.

Usability Evaluation of the Website Services Using the WEBUSE Method (A Case Study: covid19.go.id)

The purpose of this study is to analyze the site usability evaluation system using the WEBUSE method on one of the information sites about COVID-19 namely the website covid19.go.id. The basic analysis was carried out on 24 usability criteria classified into 5 usability categories, namely: content, organization, and readability, navigation and links, user interface design, performance and effectiveness, educational purpose. Each category deals with one aspect of usability. Data from respondents’ answers were analyzed using the WEBUSE method to get the usage point and usability level of the website that has been studied. This research produces a usability evaluation information system that provides an online evaluation questionnaire and this analysis is useful to present high-quality system information and user satisfaction aspects of the website covid19.go.id. The case study on the website covid19.go.id, shows that the level of usability and overall level of user satisfaction is good and acceptable for the general public from all walks of life. And in terms of satisfaction in the user interface design category, it is possible that some people are still uncomfortable with the look of the website.

Evaluation of the Usability of Mobile Library Based on Analytic Hierarchy Process: Case of WeChat Library

ABSTRACT This article studies an indicator system to evaluate the usability of WeChat library, which is a special example of a mobile library and communication platform that enables libraries to provide readers various services using the WeChat software. Based on the analysis of WeChat library, the literature survey and Delphi methods were used to determine the usability of evaluation indicators for WeChat library. The analytic hierarchy process was used to calculate the weights of indicators, and an indicator system to evaluate the usability of WeChat libraries was built. This indicator system has three primary indicators and eleven secondary indicators, and passed the consistency test. This usability evaluation system is helpful to construct and develop WeChat library.

Assessing the Relative Importance of an E-learning system’s Usability Design Characteristics Based on Students' Preferences

&lt;p style="text-align:justify"&gt;This study considers the interaction between an e-learning system, the Blackboard system, and the students who use it in Saudi Arabia. While previous work exists, there is limited consideration of the assessment of the preferences of e-learning system usability variables based on students’ perspectives, especially in developing countries such as Saudi Arabia. This paper attempts to fill the gap by investigating the relative importance of the design criteria developed for e-learning system usability evaluation from the students’ perspective in Saudi tertiary education. Based on reviewed literature, a set of usability principles was developed that have had an influence in the students’ learning process and use of the e-learning system. The list includes system navigation, system learnability, visual design, information quality, instructional assessment and system interactivity. An exploratory study was carried out to identify the most important usability design characteristics from a student’s perspective and then evaluate the overall usability of the current e-learning system, based on this subset. A quantitative approach was adopted to weigh usability design characteristics, based on 181 learners’ perceptions. The sample consists of undergraduates who are users of a web-based e-learning system in a university in Saudi Arabia. The research instrument was tested for construct validity and reliability. The analysed results have shown that information quality is the most important dimension followed by the navigation of the e-learning system. The study has also revealed that the system learnability and visual design came third and fourth in order of importance of e-learning system usability assessment. Finally, the least important design categories that influenced the e-learning system usability assessment were instructional assessment and system interactivity. The empirical results of this study may help to provide insights for designers and evaluators leading to a more effective approach to improve the usability and uptake of the e-learning system.&lt;/p&gt;

Performance Evaluation of Alarm Forms under Different Visual Coding on the Digital Interface

To investigate user performance evaluation for different alarm forms on the digital interface under single dimensional information coding, such as color, shape, flash frequency, size and number, this study designed 5 visual alarm modes by following the design specification of visual alarm signal and carried out behavioral experiments for a certain radar interface. Based on the statistical analysis of accuracy rate and reaction time and comprehensive experimental interviews, the results reveal that, visual coding alarm form in the best identification performance is number and the worst is flash frequency. The research conclusions can be applied to the actual alarm interface design. It has a certain reference value for reducing the cognitive friction between the user and the digital interface, establishing a good man-machine cognitive matching relationship and perfecting the usability evaluation system of information coding.

Usability Evaluation of a Novel Robotic Power Wheelchair for Indoor and Outdoor Navigation

ObjectiveTo compare the Mobility Enhancement roBotic (MEBot) wheelchair’s capabilities with commercial electric-powered wheelchairs (EPWs) by performing a systematic usability evaluation. DesignUsability in effectiveness, efficacy, and satisfaction was evaluated using quantitative measures. A semistructured interview was employed to gather feedback about the users’ interaction with MEBot. SettingLaboratory testing of EPW driving performance with 2 devices in a controlled setting simulating common EPW driving tasks. ParticipantsA convenience sample of expert EPW users (N=12; 9 men, 3 women) with an average age of 54.7±10.9 years and 16.3± 8.1 years of EPW driving experience. InterventionsNot applicable. Main Outcome MeasuresPowered mobility clinical driving assessment (PMCDA), Satisfaction Questionnaire, National Aeronautics and Space Administration’s Task Load Index. ResultsParticipants were able to perform significantly higher number of tasks (P=.004), with significantly higher scores in both the adequacy-efficacy (P=.005) and the safety (P=.005) domains of the PMCDA while using MEBot over curbs and cross-slopes. However, participants reported significantly higher mental demand (P=.005) while using MEBot to navigate curbs and cross-slopes due to MEBot’s complexity to perform its mobility applications which increased user’s cognitive demands. ConclusionsOverall, this usability evaluation demonstrated that MEBot is a promising EPW device to use indoors and outdoors with architectural barriers such as curbs and cross-slopes. Current design limitations were highlighted with recommendations for further improvement.

E- learning Translator Prototype for Iraqi Kids

The official language of Iraq is Arabic. Many other languages are spoken by a variety of ethnic groups, most notably are Kurdish and Turkoman. The main aim of this study is to improve the unsatisfactory materials that currently available for teaching English for the Iraqi kids who speak Arabic, Kurdish and Turkoman.&#x0D; This study objective is to develop an interactivity based prototype for children’s who aged from 6-11 years old to educate them some of the English Language vocabulary by introducing word-to-word based online translator. The age 6-11 is an important time of intellectual and emotional development for children, and it is important to remember that even as they gain rudimentary reading skills, they still enjoy being read. The concept of multimedia technology with represented by audio and images used in interactive way.&#x0D; The study includes two types’ interview methods. A semi-structure interview used to gather the prototype requirements. In prototype evaluation a structured interview conducted based on IBM questions form which is designed for system usability evaluation.&#x0D;

User’s Emotions and Usability Study of a Brain-Computer Interface Applied to People with Cerebral Palsy

People with motor and communication disorders face serious challenges in interacting with computers. To enhance this functionality, new human-computer interfaces are being studied. In this work, a brain-computer interface based on the Emotiv Epoc is used to analyze human-computer interactions in cases of cerebral palsy. The Phrase-Composer software was developed to interact with the brain-computer interface. A system usability evaluation was carried out with the participation of three specialists from The Fundação Catarinense de Educação especial (FCEE) and four cerebral palsy volunteers. Even though the System Usability Scale (SUS) score was acceptable, several challenges remain. Raw electroencephalography (EEG) data were also analyzed in order to assess the user’s emotions during their interaction with the communication device. This study brings new evidences about human-computer interaction related to individuals with cerebral palsy.

Developing customizable defence mobile application system usability evaluation scale (Defence-Moasues using Exploratory Factor Analysis (EFA

A Defence Mobile Application System Usability Evaluation Scale (Defence-MoASUES) was designed to support customization of items at the construct level and to examine the construct validity of the item. The defence training system studied was an android-based application system that supported cadet evaluation and reporting. Using Defence-MoASUES, a cross-sectional study was conducted to evaluate users’ perception toward usability evaluation (effectiveness, efficiency and satisfaction measures) of android-based defence training application system. The scale’s psychometric properties of Defence-MoASUES were analysed by conducting an exploratory factor analysis (EFA) test. The sample comprised 384 cadet officers from a defence university. As a result, the customization matches the specific tasks of a defence training system while retaining comparability at the construct level.

Usability Evaluation of Online Courses Service Platform Based on Extenics: Using &amp;quot;Machine Principle&amp;quot; Course as an Example

In this paper, based on the usability evaluation system of online courses service platform, the content experience dimension is quantified by means of professional teachers' subjective scoring on the quality and arrangement of teaching content, and the interface experience dimension is quantified by means of performance measurement, the task completion of the subjects in the learning process is measured by making a series of learning tasks on the completion platform, the subjects who completed the performance measurement experiment were invited to participate in the self-report questionnaire survey, then the emotional experience dimension is quantified by means of self reporting measurement. On this basis, through the establishment of the matter element and the correlation function, an extension evaluation model of the service platform for online courses is developed based on Extenics. Using the quantitative data of each indicator as inputs, the proposed extension evaluation model is used to calculate the correlation degree of each indicator, and the usability of the service platform for online courses is evaluated. The result shows that the usability level of the service platform for online course which is using in Harbin Institute of Technology is in the third level, and it is at a medium level.

Adapting A Unified Electronic Health Record Usability Framework for Evaluation of Connected Health Care Technologies Linking Mobile Data

Background: Evidence-based, objective, and systematic usability evaluation is key to successful connected health care technologies. The increases in patient-facing mobile health technologies not only offer convenience for patients in managing their own health and/or chronic conditions, but also offer the opportunity for health care providers to access patient behaviors and patient-centered outcomes at ease. Thus, it is of significance to link and present patient-facing mobile device data to their health care providers in a secure and uninterrupted way that will facilitate workflow and promote patient provider communication, rather than drawing providers away from patients. This prompts increasing efforts developing connected health care technologies linking mobile data to electronic health record systems guided by user-centered design and redesign principles. However, lack of scientific, objective, and systematic usability evaluation put connected health care technologies at risk for low adoption and eventual failure.