System Usability Scale Ratings Research Articles

Use of the FallAkte Plus System as an IT Infrastructure for the North Rhine-Westphalian General Practice Research Network: Mixed Methods Usability Study.

Primary care research networks can generate important information in the setting where most patients are seen and treated. However, this requires a suitable IT infrastructure (ITI), which the North Rhine-Westphalian general practice research network is looking to implement. This mixed methods research study aims to evaluate (study 1) requirements for an ITI and (study 2) the usability of an IT solution already available on the market, the FallAkte Plus (FA+) system for the North Rhine-Westphalian general practice research network, which comprises 8 primary care university institutes in Germany's largest state. In study 1, a survey was conducted among researchers from the institutes to identify the requirements for a suitable ITI. The questionnaire consisted of standardized questions with open-ended responses. In study 2, a mixed method approach combining a think-aloud approach and a quantitative survey was used to evaluate the usability and acceptance of the FA+ system among 3 user groups: researchers, general practitioners, and practice assistants. Respondents were asked to assess the usability with the validated system usability scale and to test a short questionnaire on vaccination management through FA+. In study 1, five of 8 institutes participated in the requirements survey. A total of 32 user requirements related primarily to study management were identified, including data entry, data storage, and user access management. In study 2, a total of 36 participants (24 researchers and 12 general practitioners or practice assistants) were surveyed in the mixed methods study of an already existing IT solution. The tutorial video and handouts explaining how to use the FA+ system were well received. Researchers, unlike practice personnel, were concerned about data security and data protection regarding the system's emergency feature, which enables access to all patient data. The median overall system usability scale rating was 60 (IQR 33.0-85.0), whereby practice personnel (median 82, IQR 58.0-94.0) assigned higher ratings than researchers (median 44, IQR 14.0-61.5). Users appreciated the option to integrate data from practices and other health care facilities. However, they voted against the use of the FA+ system due to a lack of support for various study formats. Usability assessments vary markedly by professional group and role. In its current stage of development, the FA+ system does not fully meet the requirements for a suitable ITI. Improvements in the user interface, performance, interoperability, security, and advanced features are necessary to make it more effective and user-friendly. Collaborating with end users and incorporating their feedback are crucial for the successful development of any practice network research ITI.

An interactive augmented reality framework to enhance the user experience and operational skills in electronics laboratories

Augmented reality (AR) stands as a widely embraced technology that significantly enhances learning experiences for students. AR offers an instructional approach supported by technological design, thereby fostering enriched learning interactions. This research proposes an interactive AR framework, intended to create an augmented reality learning environment (ARLE) for the specific needs of electronics engineering laboratory hardware operations. The ARLE functions as an active learning system designed using a user-centered methodology. It offers interactive 3D models of laboratory equipment, providing learners with preliminary training in equipment operation. The real-time connection between the laboratory apparatus and the AR environment is established using the Arduino board. This interface empowers users to control the AR simulation through the laboratory equipment seamlessly. An experimental study involving 80 engineering students was conducted to evaluate the impact of AR intervention on user experience, usability, and operational skills. The participants were divided into two groups: the experimental group (N = 40) and the control group (N = 40). The experimental group underwent electronics equipment training using ARLE, while the control group followed instructions from a standard instrument handbook. To assess the usability and user experience of ARLE, the system usability scale (SUS) and the user experience questionnaire (UEQ) were employed (N = 40). The findings revealed an SUS score of 80.9 for ARLE, categorizing it as “good” according to SUS ratings. Additionally, the UEQ results demonstrated significantly favorable scores across the six scales when compared to the benchmark dataset. The study's outcomes demonstrate that AR intervention offers learners significant pedagogical value, resulting in a substantial positive impact on operational skills in electronics laboratories.

User-centered development of an mHealth app for cardiovascular prevention

Background Mobile health (mHealth) apps can be used for cardiovascular disease (CVD) prevention. User-centered design, evidence-based content and user testing can be applied to ensure a high level of usability and adequate app access. Objective To develop and evaluate an mHealth app ( HerzFit) for CVD prevention. Methods HerzFit´s development included a user-centered design approach and guideline-based content creation based on the identified requirements of the target group. Beta testing and a preliminary usability evaluation of the HerzFit prototype were performed. For evaluation, German versions of the System Usability Scale (SUS) and the mHealth App Usability Questionnaire (GER-MAUQ) as well as free text feedback were applied. Results User-centered design thinking led to the definition of four personas. Based on their requirements, HerzFit enables users to individually assess, monitor, and optimize their cardiovascular risk profile. Users are also provided with a variety of evidence-based information on CVD and their risk factors. The user interface and system design followed the identified functional requirements. Beta-testers provided feedback on the structure and functionality and rated the usability of HerzFit´s prototype as slightly above average both in SUS and GER-MAUQ rating. Participants positively noted the variety of functions and information presented in HerzFit, while negative feedback mostly concerned wearable synchronization. Conclusions The present study demonstrates the user-centered development of a guideline-based mHealth app for CVD prevention. Beta-testing and a preliminary usability study were used to further improve the HerzFit app until its official release.

Mixed methods usability evaluation of an assistive wearable robotic hand orthosis for people with spinal cord injury

BackgroundRobotic hand orthoses (RHO) aim to provide grasp assistance for people with sensorimotor hand impairment during daily tasks. Many of such devices have been shown to bring a functional benefit to the user. However, assessing functional benefit is not sufficient to evaluate the usability of such technologies for daily life application. A comprehensive and structured evaluation of device usability not only focusing on effectiveness but also efficiency and satisfaction is required, yet often falls short in existing literature. Mixed methods evaluations, i.e., assessing a combination of quantitative and qualitative measures, allow to obtain a more holistic picture of all relevant aspects of device usability. Considering these aspects already in early development stages allows to identify design issues and generate generalizable benchmarks for future developments.MethodsWe evaluated the short-term usability of the RELab tenoexo, a RHO for hand function assistance, in 15 users with tetraplegia after a spinal cord injury through a comprehensive mixed methods approach. We collected quantitative data using the Action Research Arm Test (ARAT), the System Usability Scale (SUS), and timed tasks such as the donning process. In addition, qualitative data were collected through semi-structured interviews and user observations, and analyzed with a thematic analysis to enhance the usability evaluation. All insights were attributed and discussed in relation to specifically defined usability attributes such as comfort, ease of use, functional benefit, and safety.ResultsThe RELab tenoexo provided an immediate functional benefit to the users, resulting in a mean improvement of the ARAT score by 5.8 points and peaking at 15 points improvement for one user (clinically important difference: 5.7 points). The mean SUS rating of 60.6 represents an adequate usability, however, indicating that especially the RHO donning (average task time = 295 s) was perceived as too long and cumbersome. The participants were generally very satisfied with the ergonomics (size, dimensions, fit) of the RHO. Enhancing the ease of use, specifically in donning, increasing the provided grasping force, as well as the availability of tailoring options and customization were identified as main improvement areas to promote RHO usability.ConclusionThe short-term usability of the RELab tenoexo was thoroughly evaluated with a mixed methods approach, which generated valuable data to improve the RHO in future iterations. In addition, learnings that might be transferable to the evaluation and design of other RHO were generated, which have the potential to increase the daily life applicability and acceptance of similar technologies.

National survey of user-reported usability of electronic medical record software in ophthalmology in Germany.

A nationwide, comparative survey of the physician-reported usability of electronic medical record (EMR) software used by ophthalmologists in Germany using the System Usability Scale (SUS) as a standardized metric. A cross-sectional survey of members of the German Ophthalmological Society (DOG) and professional association of ophthalmologists (BVA) was conducted in May 2022. All 7788 physician members of both societies were invited to participate in an anonymous online-survey by individualized links. User-reported usability of the participants main software used for electronic medical recordkeeping was assessed using the SUS (range 0-100). A total of 881 participants with 51 different EMRs completed the entire questionnaire. Mean EMR-SUS score was 65.7 (SD ± 23.5). Significant differences in mean SUS of several EMR programs were observed with a range of 31.5 to 87.2 in programs with 10 or more responses. 31.8% of all main program SUS ratings were below 50 points. Female gender was associated with 4.02 higher SUS score (95% CI 0.46-7.59). Main program SUS was positively correlated with overall work-related satisfaction and work environment SUS but negatively correlated with the number of programs in the work environment. The SUS of the entire digital work environment including all programs used daily was closely correlated with the main EMR SUS, but not the number of programs used. Our survey revealed a fragmented pattern of EMR use by ophthalmologists in Germany with many competing software products and widely diverging mean System Usability Scale scores. A considerable share of ophthalmologists report EMR usability below what is commonly considered acceptable.

The The Study of UX on Students’ Perception and Attitude of Using Zoom During Covid-19 Pandemic Using User Centered Design Method

The pandemic of COVID-19 resulted in the physical shutdown which has now transformed education into an exclusive "online learning" model. Zoom is being used to evaluate the perceived usability as the reference platform. The students find it less collaborative, less interactive, boring, and less collaborative. From this perspective, the Usability of the current online learning platforms is an important factor, particularly because no physical classes are present. The User-Centered Design (UCD) approach was chosen for this study and using the Usability Scale (SUS) method to evaluate the interface. The objective of this study is to analyze user experience, design solutions and evaluate user interfaces that can meet user needs. A pre-survey to evaluate the difficulties of the Zoom application based on user experience, and a post-survey to see if the upgraded design can help the students using the Zoom application for online learning. Then, use the System Usability Scale (SUS) questionnaire approach to measure the system's usability. After the UCD approach was completed, the researcher did a follow-up survey. The results showed that SUS ratings went up to 85.12. As a result, the previously low acceptability ranges have been raised to acceptable. Additionally, the grade scale has been reclassified B. The Zoom program now has more features and is easier to use, and fulfills the students' needs.

Mobile Access to Medical Records in Heart Transplantation Aftercare: Mixed-Methods Study Assessing Usability, Feasibility and Effects of a Mobile Application.

Background: Patient access to medical records can improve quality of care. The phellow application (app) was developed to provide patients access to selected content of their medical record. It was tested at a heart transplantation (HTx) outpatient clinic. The aims of this study were (1) to assess usability of phellow, (2) to determine feasibility of implementation in routine care, and (3) to study the effects app use had on patients’ self-management. Methods: Usability was measured quantitatively through the System Usability Scale (SUS). Furthermore, usability, feasibility, and effects on self-management were qualitatively assessed through interviews with users, non-users, and health care providers. Results: The SUS rating (n = 31) was 79.9, indicating good usability. Twenty-three interviews were conducted. Although appreciation and willingness-to-use were high, usability problems such as incompleteness of record, technical issues, and complex registration procedures were reported. Improved technical support infrastructure, clearly defined responsibilities, and app-specific trainings were suggested for further implementation. Patients described positive effects on their self-management. Conclusions: To be feasible for implementation in routine care, usability problems should be addressed. Feedback on the effect of app use was encouraging. Accompanying research is crucial to monitor usability improvements and to further assess effects of app use on patients.

Integrating hand exoskeletons into goal-oriented clinic and home stroke and spinal cord injury rehabilitation.

IntroductionRobotic exoskeletons are emerging as rehabilitation and assistive technologies that simultaneously restore function and enable independence for people with disabilities.AimWe investigated the feasibility and orthotic and restorative effects of an exoskeleton-supported goal-directed rehabilitation program for people with hand impairments after stroke or Spinal Cord Injury (SCI).MethodA single-arm case-series feasibility study was conducted using a wearable untethered hand exoskeleton during goal-directed therapy programs with in-clinic and at-home components. Therapists trained stroke and SCI patients to use a hand exoskeleton during rehabilitation exercises, activities of daily living and patient-selected goals. Each patient received a 1-hour in-clinic training session on five consecutive days, then took the exoskeleton home for two consecutive days to perform therapist-recommended tasks. Goal Attainment Scaling (GAS) and the Box and Block Test (BBT) were administered at baseline, after in-clinic therapy and after home use, with and again without wearing the exoskeleton. The System Usability Scale (SUS), Motor Activity Log, and Fugl-Meyer Assessment were also administered to assess the intervention’s acceptability, adherence, usability and effectiveness.ResultsFour stroke patients (Chedoke McMaster Stage of Hand 2–4) and one SCI patient (ASIA C8 Motor Stage 1) 23 ± 19 months post-injury wore the hand exoskeleton to perform 280 ± 23 exercise repetitions in the clinic and additional goal-oriented tasks at home. The patients performed their own goals and the dexterity task with higher performance following the 7-days therapy program in comparison to baseline for both exoskeleton-assisted (ΔGAS: 18 ± 10, ΔBBT: 1 ± 5) and unassisted (ΔGAS: 14 ± 14, ΔBBT: 3 ± 4) assessments. Therapists and patients provided ‘good’ SUS ratings of 78 ± 6 and no harmful events were reported.ConclusionsThe exoskeleton-supported stroke and SCI therapy program with in-clinic and at-home training components was feasible.

Development of a Conversational Agent for Promoting Increased Activity in Users with Traumatic Brain Injury

<h3>Research Objectives</h3> To develop and test a conversational agent-i.e., a chatbot that users communicate via text messaging-designed to augment behavioral activation treatment for reducing depression and increasing participation in individuals with moderate to severe Traumatic Brain Injury (TBI) called RehaBot <h3>Design</h3> A prototype of RehaBot was developed based on feedback from therapist and consumer focus groups and then tested in three separate user demonstration trials. A usability and preliminary efficacy trial is underway. <h3>Setting</h3> Outpatient rehabilitation. <h3>Participants</h3> Rehabilitation providers and persons with moderate to severe TBI. <h3>Interventions</h3> None. <h3>Main Outcome Measures</h3> System Usability Scale (SUS). <h3>Results</h3> Therapist focus group participants indicated that RehaBot could promote engagement with therapeutic plans and provide prompts that cue users to adhere to routines. Respondents with TBI indicated that they would be comfortable interacting with RehaBot, and most had prior experience with chatbots. They expressed a preference for a system accessible via smartphone (vs. computer/voice assistant). Goal reminders, progress updates, and encouragement were identified as beneficial features. Respondents conveyed that misunderstandings between users and chatbots would be frustrating. In demonstration trials, two of the three participants could complete 5 of 5 RehaBot tasks independently. One user completed three independently and a 4th with assistance. The average SUS rating was 4.3 (0.80) on a 5-point scale. <h3>Conclusions</h3> Preliminary evidence suggests that a chatbot augment to behavioral activation treatment-RehaBot-is acceptable and easy to use for persons with moderate to severe TBI. Further research is ongoing to establish usability in a larger sample and evaluate RehaBot's efficacy in improving therapeutic plan adherence. <h3>Author(s) Disclosures</h3> None.

Usability Evaluation of Web Search User Interfaces from the Elderly Perspective

The elderly population is increasing in many countries, often with health and incapacity challenges, largely disengaged them from the world of digital tools like Internet usage. They browse the Internet daily for obtaining needed information through various search engines through the search UI. Earlier technologies were fabricated for improving daily life, but the specific needs of the elderly are neglected often. Currently, available online search UIs are well-developed, but they did not consider usability in their design specifically for the elderly. This research aims to evaluate web search UIs based on the elderly perspectives to identify existing search UIs usability issues and recommend improvements to web search UI designs. The observation technique evaluated two web search UIs (Google interface and Bing interface) with fifteen participants aged 60 years and above. System Usability Scale (SUS) questionnaire was applied to measure the user satisfaction of the current two interfaces. The data collected from the observations were analyzed using content analysis, while the data acquired from the questionnaires were analyzed using the t-test. The results revealed a statistically significant difference in SUS ratings, with Google scoring 73.5 and Bing scoring 66.5, indicating that users prefer the Google interface over the Bing interface. Besides that, the usability issues were identified, and recommendations to improve the design of the search UI were suggested. These findings contribute to a better understanding of the issues that prevent elderly users from using web search UI and valuable feedback to designers on improving the UI to suit the elderly better.

Quantitative usability assessment relying on experiential and specific task based SUS ratings

This study has two main objectives; one is to present the empirical result that shows how the SUS scores vary when end-users perform usability assessment relying on experience and by executing the assigned list of specific tasks on academic websites. The results are obtained by using independent sample t-test that scrutinizes the slight but significant difference in mean SUS scores among two. Results also investigate that 85.29% of websites have shown higher mean SUS scores when usability assessment is conducted with the assigned list of specific tasks whereas only 14.70% of websites have shown higher mean SUS scores when usability assessment is performed based on the experience. Another aim is to determine the relationship which is found to be strong positive correlation between system effectiveness, an ISO metric, measured as task success rate and subjective usability scores, measured through System Usability Scale (SUS) for 50-academic websites. It concludes that successful completion of the tasks results in high usability scores.

Usability Study of Mainstream Wearable Fitness Devices: Feature Analysis and System Usability Scale Evaluation

BackgroundWearable devices have the potential to promote a healthy lifestyle because of their real-time data monitoring capabilities. However, device usability is a critical factor that determines whether they will be adopted on a large scale. Usability studies on wearable devices are still scarce.ObjectiveThis study aims to compare the functions and attributes of seven mainstream wearable devices and to evaluate their usability.MethodsThe wearable devices selected were the Apple Watch, Samsung Gear S, Fitbit Surge, Jawbone Up3, Mi Band, Huawei Honor B2, and Misfit Shine. A mixed method of feature comparison and a System Usability Scale (SUS) evaluation based on 388 participants was applied; the higher the SUS score, the better the usability of the product.ResultsFor features, all devices had step counting, an activity timer, and distance recording functions. The Samsung Gear S had a unique sports track recording feature and the Huawei Honor B2 had a unique wireless earphone. The Apple Watch, Samsung Gear S, Jawbone Up3, and Fitbit Surge could measure heart rate. All the devices were able to monitor sleep, except the Apple Watch. For product characteristics, including attributes such as weight, battery life, price, and 22 functions such as step counting, activity time, activity type identification, sleep monitoring, and expandable new features, we found a very weak negative correlation between the SUS scores and price (r=−.10, P=.03) and devices that support expandable new features (r=−.11, P=.02), and a very weak positive correlation between the SUS scores and devices that support the activity type identification function (r=.11, P=.02). The Huawei Honor B2 received the highest score of mean 67.6 (SD 16.1); the lowest Apple Watch score was only 61.4 (SD 14.7). No significant difference was observed among brands. The SUS score had a moderate positive correlation with the user’s experience (length of time the device was used) (r=.32, P<.001); participants in the medical and health care industries gave a significantly higher score (mean 61.1, SD 17.9 vs mean 68.7, SD 14.5, P=.03).ConclusionsThe functions of wearable devices tend to be homogeneous and usability is similar across various brands. Overall, Mi Band had the lowest price and the lightest weight. Misfit Shine had the longest battery life and most functions, and participants in the medical and health care industries had the best evaluation of wearable devices. The perceived usability of mainstream wearable devices is unsatisfactory and customer loyalty is not high. A consumer’s SUS rating for a wearable device is related to their personal situation instead of the device brand. Device manufacturers should put more effort into developing innovative functions and improving the usability of their products by integrating more cognitive behavior change techniques.

Measuring Perceived Usability: SUS, UMUX, and CSUQ Ratings for Four Everyday Products

ABSTRACTThis research continued previous investigation of the relationships among measures of perceived usability: the System Usability Scale (SUS), three metrics derived from the Usability Metric for User Experience (UMUX), and the Computer System Usability Questionnaire (CSUQ), this time with ratings of four everyday products (Excel, Word, Amazon, and Gmail). SUS ratings of these products were generally consistent with previous reports. Significant differences in SUS means across studies could be due to differences in frequency of use, with implications for using these data as usability benchmarks. Correspondence among the various measures of perceived usability was also consistent with previous research. Considering frequency of use, mean differences ranged from -2.0 to 1.8 (average shift in Sauro-Lewis grade range from -0.6 to 0.8). When SUS scores were above average, the range restriction of the UMUX-LITEr led to relatively large discrepancies with SUS, suggesting it might not always be better than unadjusted UMUXLITE.

Measuring the Usability of Mobile Applications for Phones and Tablets

The use of applications on mobile devices has reached historic levels. Using the System Usability Scale (SUS), data were collected on the usability of applications used on two kinds of mobile platforms—phones and tablets—across two general classes of operating systems, iOS and Android. Over 4 experiments, 3,575 users rated the usability of 10 applications that had been selected based on their popularity, as well as 5 additional applications that users had identified as using frequently. The average SUS rating for the top 10 apps across all platforms was 77.7, with a nearly 20-point spread (67.7–87.4) between the highest and lowest rated apps. Overall, applications on phone platforms were judged to be more usable than applications on the tablet platforms. Practitioners can use the information in this article to make better design decisions and benchmark their progress against a known universe of apps for their specific mobile platform.

Development of a Weighted Heuristic for Website Evaluation for Older Adults

Older adults are the fastest growing population of Internet users. As websites acquire a greater number of older visitors, it is vital that they are designed with this demographic in mind. Older users typically have different user characteristics than younger users; they may have changes in perceptual abilities, motor skills, cognitive abilities, mental models, and confidence in the use of technology. This research documents the development of a new weighted heuristic measure for evaluating the usability of websites for older adults and its validation with performance testing. Results from a repeated measures analysis of variance indicated that websites with different heuristic classifications were significantly different with respect to performance metrics and System Usability Scale ratings. Conclusions point to the need for web design that takes into account preferences and abilities of older web users.

The effect of experience on system usability scale ratings

Longitudinal studies have to do with testing over time and thus take into consideration previous user experience with a product or product versions. However, it is difficult to conduct these types of studies. Therefore the literature is sparse on examples of the explicit effect of user experience on user satisfaction metrics in industry-standard survey instruments. During a development experience in 2009, we used a cross-sectional method to look at the effects of user profiles on ratings for commercial products that use one such instrument, the System Usability Scale or SUS. Recent research has reported finding that differences in user ratings could be based on the extent of a user's prior experience with the computer system, a Web site being visited or a desktop application like Microsoft's Office suite being used. Compared to off-the-shelf office products or personal Web applications, we were curious if we would find the same experience effect for domain specialists using geosciences products in the course of their daily professional job roles. In fact, from data collected with 262 end users across different geographic locations testing two related oilfield product releases, one Web-based and one desktop-based, we found results that were quite close to early assessment studies: Users having a more extensive experience with a product tended to provide higher, more favorable, SUS scores over users with either no or limited experience with a product---and by as much as 15-16%, regardless of the domain product type. This and other observations found during our product testing have led us to offer some practical how-to's to our internal product analysts responsible for managing product test cycles, administering instruments like the SUS to users, and reporting results to development teams.

On the dimensionality of the System Usability Scale: a test of alternative measurement models

The System Usability Scale (SUS), developed by Brooke (Usability evaluation in industry, Taylor & Francis, London, pp 189-194, 1996), had a great success among usability practitioners since it is a quick and easy to use measure for collecting users' usability evaluation of a system. Recently, Lewis and Sauro (Proceedings of the human computer interaction international conference (HCII 2009), San Diego CA, USA, 2009) have proposed a two-factor structure-Usability (8 items) and Learnability (2 items)-suggesting that practitioners might take advantage of these new factors to extract additional information from SUS data. In order to verify the dimensionality in the SUS' two-component structure, we estimated the parameters and tested with a structural equation model the SUS structure on a sample of 196 university users. Our data indicated that both the unidimensional model and the two-factor model with uncorrelated factors proposed by Lewis and Sauro (Proceedings of the human computer interaction international conference (HCII 2009), San Diego CA, USA, 2009) had a not satisfactory fit to the data. We thus released the hypothesis that Usability and Learnability are independent components of SUS ratings and tested a less restrictive model with correlated factors. This model not only yielded a good fit to the data, but it was also significantly more appropriate to represent the structure of SUS ratings.

How Much to Share with Users during Usability Testing

This study evaluated user behaviors during two different testing environments: no intervention (unfacilitated) versus moderate intervention (facilitated). Nineteen participants evaluated a complex 3-D application with about half in each of the two conditions. All subjects carried out a series of tasks and, for each task, we gathered completion times, ease of use and importance ratings, as well as open-ended comments and suggestions for improvement. At the end of the study, participants made System Usability Scale (SUS) and other ratings and gave overall impressions of the application. There were no differences depending on group (facilitated versus unfacilitated) for any of the quantitative task measures, including: percent success, time to complete the tasks, ease of use ratings, and importance ratings. However, we observed differences in suggestions for improving the application; comments were much richer and more useful for the facilitated group. There were reliable group differences in some of the general measures following task completion, including SUS ratings. These findings are discussed in relation to other observations in the literature and pros and cons of each method are described.