Severity of usability problems and system usability scale (Sus) scores on augmented reality (ar) user interfaces

The System Usability Scale (SUS) score survey is a widely respected tool for measuring usability. While there are other surveys available such as the User Experience Questionnaire (UEQ) or the Single Ease Question (SEQ), the SUS is amongst the most popular and widely used instrument. SUS provides an easy-to-understand score with benchmarking. Generally, a SUS score is administered directly after a usability test to assess the user experience and the usability of a product, including websites and smartphone apps and more. However, some researchers have used it as a survey as part of a ‘in the wild’ trial which is often completed after the trial or indeed sometime after the subjects interacted with the technology. With this in mind the aim of this research was to see if a participant’s user experience would change if a SUS score was administered at different times after a test to understand if recalling the usability of technology led to temporal bias for the SUS.

The Pharmaceutical Services Program (PSP) has not yet evaluated the usability of the web application system. The System Usability Scale (SUS) and Net Promoter Score (NPS) assessment tools give you a quick overview of your system's usability and help you improve it further. The goal of this study was to assess the usability of the PRiSMA and My.Pharma-C systems using SUS and NPS. From April 2019 to July 2020, a survey was conducted using a validated self-administered questionnaire that was prompted after each successful log-out from the PRiSMA and My.Pharma-C systems. The questionnaire was divided into three sections: demographic, device type, and SUS and NPS questionnaire. The mean SUS and NPS scores were computed. The survey was completed by 3,959 PRiSMA users and 811 My.Pharma-C users. The SUS score was interpreted as a grade, adjective, and range of acceptability. The PRiSMA mean SUS score was 70.2 (SD 15.44), indicating a 'Grade C' and 'Good' system. The mean SUS score for My.Pharma-C was 58.2 (SD 15.13), indicating a 'Grade D' and 'OK' system. Users accepted both systems on the basis of their usability. Calculated NPS value for PRiSMA was p=42%, n=49% and d=9%, NPS value=+33, whereas My.Pharma-C p=18%, n=46% and d=36%, NPS value=-18. A positive NPS value for PRiSMA represents the user's readiness to recommend the system to others compared to My.Pharma-C. The findings also established that SUS alone is insufficient and needs to be complemented by another method, such as qualitative instruments. It could help to drill down in depth to understand how to address potentially problematic areas for web application systems and assist in the improvement phases. However, both of the tools are applicable to be used to evaluate the usability of the web application systems that managed by the Pharmaceutical Services Program.

BackgroundThe System Usability Scale (SUS) is a widely used scale that has been used to quantify the usability of many software and hardware products. However, the SUS was not specifically designed to evaluate mobile apps, or in particular digital health apps (DHAs).ObjectiveThe aim of this study was to examine whether the widely used SUS distribution for benchmarking (mean 68, SD 12.5) can be used to reliably assess the usability of DHAs.MethodsA search of the literature was performed using the ACM Digital Library, IEEE Xplore, CORE, PubMed, and Google Scholar databases to identify SUS scores related to the usability of DHAs for meta-analysis. This study included papers that published the SUS scores of the evaluated DHAs from 2011 to 2021 to get a 10-year representation. In total, 117 SUS scores for 114 DHAs were identified. R Studio and the R programming language were used to model the DHA SUS distribution, with a 1-sample, 2-tailed t test used to compare this distribution with the standard SUS distribution.ResultsThe mean SUS score when all the collected apps were included was 76.64 (SD 15.12); however, this distribution exhibited asymmetrical skewness (–0.52) and was not normally distributed according to Shapiro-Wilk test (P=.002). The mean SUS score for “physical activity” apps was 83.28 (SD 12.39) and drove the skewness. Hence, the mean SUS score for all collected apps excluding “physical activity” apps was 68.05 (SD 14.05). A 1-sample, 2-tailed t test indicated that this health app SUS distribution was not statistically significantly different from the standard SUS distribution (P=.98).ConclusionsThis study concludes that the SUS and the widely accepted benchmark of a mean SUS score of 68 (SD 12.5) are suitable for evaluating the usability of DHAs. We speculate as to why physical activity apps received higher SUS scores than expected. A template for reporting mean SUS scores to facilitate meta-analysis is proposed, together with future work that could be done to further examine the SUS benchmark scores for DHAs.

This study examined whether the average usability score for a series of tasks was the same as the usability score for the product if usability was measured only after all the tasks had been completed. Fifty participants completed a set of tasks for five websites and fourteen mock voting ballots. Subjective usability assessment was made with the System Usability Scale (SUS). Participants completed the SUS either after each task (five or fourteen SUS administrations, respectively) or after completing the entire set of tasks (one SUS). The results show that the average SUS scores for the task-level assessments were significantly higher than the SUS scores for the test-level assessments. Results were similar for the ballot and website conditions. Task-level SUS scores on the Honda websites ( M = 65.5) were significantly higher than the test-level SUS scores ( M = 42.8), p < 0.0001. Similar results were observed in the ballot condition, where task-level usability assessments were higher ( M = 59.5) than test-level assessments ( M = 38.5), p < 0.0001. Practitioners and those interpreting SUS scores need to be aware of how these experimental differences can lead to different assessment metrics.

Online banking is a critical service offered by financial institutions to their clientele to facilitate easier and faster access to financial services and transactions. Banks currently spend huge amounts of money on development and maintenance of websites and backend systems that offer online banking facilities to clients. Here we address the effect of moderating factors on online banking usability assessment in South Africa. Using statistical analysis techniques that included t-tests, ANOVA and correlation, we investigated whether there are statistically significant mean differences in system usability scale (SUS) scores based on a variety of moderating factors in South Africa. Findings based on a sample of 540 respondents show that SUS scores differ significantly based on factors such as age, experience and income, whereas factors such as gender, use frequency and employment did not affect the mean SUS scores. Given the individual SUS scores for a variety of users based on different demographics, the financial institutions might improve service usability to target specific user groups and realise their return on investment in digital banking channels. Therefore improving service usability might go a long way in encouraging online banking adoption in South Africa. Significance: The overall assessment of online banking service by users based on a SUS measurement tool was investigated. The effect of moderating variables on the mean SUS scores of different user groups was established. An insight into areas of improvement with regard to usability based on demographic information of users is provided.

The System Usability Scale (SUS) is widely used as a quick method for measuring usability; however, past research showed there is only a weak relationship between SUS scores and one behavioral usability measure, and alternatively, SUS corresponds more strongly with user preference. This suggests that the underlying constructs of the SUS may not be well understood. In this study, participants were asked to think aloud while completing a usability test and filling out the SUS. Correlations showed no relationship between behavioral performance and SUS scores. Instead, a relationship was observed between SUS scores and perceived success. Furthermore, participants described a variety of reasons for selecting their SUS responses that were unrelated to the usability of the system, which we have termed rationalizations. This suggests that the SUS is constructed of a combination of experiential components, including attitudinal perceptions. Consequently, SUS scores may be more helpful as a tool for comparison (between competitors, iterations, etc.,) or when used in conjunction with formative usability testing methods to provide a holistic view of real and perceived user experience.

Spatial cues play an important role in navigating people in both physical and virtual spaces. In spatial navigation, visual information with additional cues, such as haptic cues, enables effective guidance. Most haptic devices are applied to various body parts to make mechanical stimuli, while few devices stimulate a head despite the excellent sensitivity. This article presents Virtual Whiskers, a spatial directional guidance technique by cheek haptic stimulation using tiny robot arms attached to a Head-Mounted Display (HMD). The tip of the robotic arm has photo reflective sensors to detect the distance between the tip and the cheek surface. Using the robot arms, we stimulate a point on the cheek obtained by calculating an intersection between the cheek surface and the target direction. In the directional guidance experiment, we investigated how accurately participants identify the target direction provided by our guidance method. We evaluated an error between the actual target direction and the participant's pointed direction. The experimental result shows that our method achieves the average absolute directional error of 2.54° in the azimuthal plane and 6.54° in the elevation plane. We also conducted a spatial guidance experiment to evaluate task performance in a target search task. We compared the condition of visual information, visual and audio information, and visual information and cheek haptics for task completion time, System Usability Scale (SUS) score, NASA-TLX score. The averages of task completion time were M = 6.39 s, SD = 3.34 s, and M = 5.62 s, SD = 3.12 s, and M = 4.35 s, SD = 2.26 s, in visual-only condition, visual+audio condition, and visual+haptic condition, respectively. In terms of the SUS score, visual condition, visual+audio condition, and visual+haptic condition achieved M = 55.83, SD = 20.40, and M = 47.78, SD = 20.09, and M = 80.42, SD = 10.99, respectively. As for NASA-TLX score, visual condition, visual+audio condition, and visual+haptic condition resulted in M = 75.81, SD = 16.89, and M = 67.57, SD = 14.96, and M = 38.83, SD = 18.52, respectively. Statistical tests revealed significant differences in task completion time, SUS score, and NASA-TLX score between the visual and the visual+haptic condition and the visual+audio and the visual+haptic condition.

The aim of this paper is to evaluate user satisfaction based on System Usability Scale (SUS) questionnaire on Augmented Reality (AR) application for productive vocabulary using speech recognition. There is still lack of research focus on user satisfaction towards the use of AR-based app using speech recognition for vocabulary learning in early education. The first objective of this paper is to develop an AR application for children using speech recognition to enhance productive vocabulary learning that integrates visual script (orthography) and audio (phonology). The second objective is to evaluate and identify the user satisfaction in enhancing productive vocabulary methods by combining AR technology and speech recognition to the parents and teachers by doing a mixed method testing such as questionnaire and interview. To achieve this, an interview session was conducted with the experts and SUS questionnaire was given to the teachers and parents of the student to evaluate user satisfaction. The results show that the research hypotheses of this study were successfully achieved. It revealed that the teachers and students were satisfied with the application based on SUS score (SUS score > 68). Since the total SUS score is 80.3, which is above average, it shows that they were satisfied with the overall application. Apart from that, the mean of the usability study based on questionnaire also shows that the users have found that this application is usable in terms of learnability study and that this application is easy to use.

BackgroundRepeated applications of short-term dietary assessment instruments are recommended for estimating usual dietary intake. For this purpose, NutriDiary, a smartphone app for collecting weighed dietary records (WDRs) in the German population, was developed.ObjectiveWe aim to describe NutriDiary and evaluate its usability and acceptability.MethodsNutriDiary was developed as a WDR, allowing users to enter food items via text search, barcode scanning, or free text entry. The sample for the evaluation study included 74 participants (n=51, 69% female, aged 18‐64 years), including 27 (37.5%) experts and 47 (63.5%) laypersons (including n=22, 30%, nutrition students). Participants completed a 1-day WDR and entered a predefined sample meal (n=17 foods) the following day by using NutriDiary. An evaluation questionnaire was answered from which the system usability scale (SUS) score (0‐100) was calculated. A backward selection procedure (PROC REG in SAS; SAS Institute) was used to identify potential predictors for the SUS score (age, sex, status [expert or laypersons], and operating system [iOS or Android]).ResultsThe median SUS score of 75 (IQR 63‐88) indicated good usability. Age was the only characteristic identified as a potential predictor for a lower SUS score (P<.001). The median completion time for an individual WDR was 35 (IQR 19‐52) minutes. Older participants took longer to enter the data than younger ones (18‐30 y: median 1.5, IQR 1.1‐2.0 min/item vs 45‐64 y: median 1.8, IQR 1.3‐2.3 min/item). Most participants expressed a preference for NutriDiary over the traditional paper-based method.ConclusionsGood usability and acceptability make NutriDiary promising for use in epidemiological studies.

BackgroundNeonatal mortality is high in low-resource settings. NeoTree is a digital intervention for neonatal healthcare professionals (HCPs) aiming to achieve data-driven quality improvement and improved neonatal survival in low-resource hospitals. Optimising usability with end-users could help digital health interventions succeed beyond pilot stages in low-resource settings. Usability is the quality of a user's experience when interacting with an intervention, encompassing their effectiveness, efficiency, and overall satisfaction.ObjectiveTo evaluate the usability and usage of NeoTree beta-app and conduct Agile usability-focused intervention development.MethodA real-world pilot of NeoTree beta-app was conducted over 6 months at Kamuzu Central Hospital neonatal unit, Malawi. Prior to deployment, think-aloud interviews were conducted to guide nurses through the app whilst voicing their thoughts aloud (n = 6). System Usability Scale (SUS) scores were collected before the implementation of NeoTree into usual clinical care and 6 months after implementation (n = 8 and 8). During the pilot, real-world user-feedback and user-data were gathered. Feedback notes were subjected to thematic analysis within an Agile “product backlog.” For usage, number of users, user-cadre, proportion of admissions/outcomes recorded digitally, and median app-completion times were calculated.ResultsTwelve overarching usability themes generated 57 app adjustments, 39 (68%) from think aloud analysis and 18 (32%) from the real-world testing. A total of 21 usability themes/issues with corresponding app features were produced and added to the app. Six themes relating to data collection included exhaustiveness of data schema, prevention of errors, ease of progression, efficiency of data entry using shortcuts, navigation of user interface (UI), and relevancy of content. Six themes relating to the clinical care included cohesion with ward process, embedded education, locally coherent language, adaptability of user-interface to available resources, and printout design to facilitate handover. SUS scores were above average (88.1 and 89.4 at 1 and 6 months, respectively). Ninety-three different HCPs of 5 cadres, recorded 1,323 admissions and 1,197 outcomes over 6 months. NeoTree achieved 100% digital coverage of sick neonates admitted. Median completion times were 16 and 8 min for admissions and outcomes, respectively.ConclusionsThis study demonstrates optimisation of a digital health app in a low-resource setting and could inform other similar usability studies apps in similar settings.

To compare the reduction qualities of three-dimensional visible technique without fluoroscopy and two-dimensional fluoroscopy for unstable pelvic fractures during operations. The clinical data of 40 patients with unstable pelvic fractures, who met the selection criteria in three clinical centers between June 2021 and September 2022, were retrospectively analyzed. According to the reduction methods, the patients were divided into two groups. Twenty patients in trial group were treated with unlocking closed reduction system combined with three-dimensional visible technique without fluoroscopy; 20 patients in control group with unlocking closed reduction system under two-dimensional fluoroscopy. There was no significant difference in the gender, age, injury mechanism, Tile type of fracture, Injury Severity Score (ISS), and the time between injury to operation between the two groups ( P>0.05). The qualities of fracture reduction according to the Matta criteria, operative time, intraoperative blood loss, fracture reduction time, times of fluoroscopy, and System Usability Scale (SUS) score were recorded and compared. All operations were successfully completed in both groups. According to the Matta criteria, the qualities of fracture reduction were rated as excellent in 19 patients (95%) in trial group, which was better than that in the control group (13 cases, 65%), with a significant difference ( χ 2=3.906, P=0.048). The operative time and intraoperative blood loss had no significant differences between the two groups ( P>0.05). The fracture reduction time and times of fluoroscopy were significantly less in trial group than in control group ( P<0.05), and SUS score in trial group was significantly higher in trial group than in control group ( P<0.05). Compared to using unlocking closed reduction system under two-dimensional fluoroscopy, three-dimensional visible technique without fluoroscopy can significantly improve the reduction quality of unstable pelvic fractures without prolonging the operative time, and is valuable to reduce iatrogenic radiation exposure for patients and medical workers.

BackgroundSarcopenia, defined as the age-associated loss of muscle mass and strength, can be effectively mitigated through resistance-based physical activity. With compliance at approximately 40% for home-based exercise prescriptions, implementing a remote sensing system would help patients and clinicians to better understand treatment progress and increase compliance. The inclusion of end users in the development of mobile apps for remote-sensing systems can ensure that they are both user friendly and facilitate compliance. With advancements in natural language processing (NLP), there is potential for these methods to be used with data collected through the user-centered design process.ObjectiveThis study aims to develop a mobile app for a novel device through a user-centered design process with both older adults and clinicians while exploring whether data collected through this process can be used in NLP and sentiment analysisMethodsThrough a user-centered design process, we conducted semistructured interviews during the development of a geriatric-friendly Bluetooth-connected resistance exercise band app. We interviewed patients and clinicians at weeks 0, 5, and 10 of the app development. Each semistructured interview consisted of heuristic evaluations, cognitive walkthroughs, and observations. We used the Bing sentiment library for a sentiment analysis of interview transcripts and then applied NLP-based latent Dirichlet allocation (LDA) topic modeling to identify differences and similarities in patient and clinician participant interviews. Sentiment was defined as the sum of positive and negative words (each word with a +1 or −1 value). To assess utility, we used quantitative assessment questionnaires—System Usability Scale (SUS) and Usefulness, Satisfaction, and Ease of use (USE). Finally, we used multivariate linear models—adjusting for age, sex, subject group (clinician vs patient), and development—to explore the association between sentiment analysis and SUS and USE outcomes.ResultsThe mean age of the 22 participants was 68 (SD 14) years, and 17 (77%) were female. The overall mean SUS and USE scores were 66.4 (SD 13.6) and 41.3 (SD 15.2), respectively. Both patients and clinicians provided valuable insights into the needs of older adults when designing and building an app. The mean positive-negative sentiment per sentence was 0.19 (SD 0.21) and 0.47 (SD 0.21) for patient and clinician interviews, respectively. We found a positive association with positive sentiment in an interview and SUS score (ß=1.38; 95% CI 0.37 to 2.39; P=.01). There was no significant association between sentiment and the USE score. The LDA analysis found no overlap between patients and clinicians in the 8 identified topics.ConclusionsInvolving patients and clinicians allowed us to design and build an app that is user friendly for older adults while supporting compliance. This is the first analysis using NLP and usability questionnaires in the quantification of user-centered design of technology for older adults.

Introduction: There are many information systems in Sri Lanka. However, there is no widely accepted tool to measure their usability. System Usability Scale (SUS) is a standardised metric or a quantitative measuring scale for measuring the usability of a website, information system or interactive system. SUS was released to the world by John Brooke in 1986. Currently, SUS has become an industry standard. In SUS, a user responds to 10 short questions on a 5-point Likert scale about the level of agreement. A special methodology is used to prevent unintentional rating responses. A SUS score above 68 would be considered above average. This questionnaire is originally in English language. However, even a native English speaker has implications on reliability and validity of the SUS due to the misunderstanding and misinterpretation of terminology or conflicting language. Thus, translating to native local languages is considered to increase the reliability and validity of SUS. The aim was to develop a Sinhala/Tamil System Usability Scale (SUS) to be used in Sri Lanka for assessing the information system usability. Methods: Translation of SUS questions in English to Sinhala and Tamil Languages after brainstorming sessions. Questions were validated by multi-disciplinary specialists. Translations were aimed at keeping the content simple, clear and concise, achieving a conceptual translation rather than a literal translation and avoid technology jargon. The tool was pretested by using a group of users in Electronic Reproductive Health Information Management System (eRHMIS) of the Family Health Bureau. Results: Sinhala/Tamil questions were finalized, and final SUS questionnaire was developed including a 5-point Likert scale. This tool was selected for measuring the usability of the eRHMIS of Family Health Bureau. Conclusions: There was immediate acceptance as well as plans for future use of translated SUS. This SUS questionnaire can be used to measure the usability of existing information systems in the country.

EVALUATION OF AN ARTIFICIAL INTELLIGENCE HEPARIN DOSING ALGORITHM TO MAINTAIN THERAPEUTIC ANTICOAGULATION DURING ATRIAL FIBRILLATION ABLATION

The goal of the research presented in this paper was to determine if the positively worded System Usability Scale (SUS) can be used in place of the positively and negatively worded standard SUS instrument for the subjective assessment of usability, and whether the results found here replicate those of Sauro and Lewis. Sauro and Lewis' previous study found no evidence that responses to SUS items differed across the standard SUS and the modified, positively worded version of the SUS when participants assessed websites. This study replicates and extends this work by examining a large number of different systems with larger sample sizes to add to the generalizability of previous findings. So that participants could retrospectively assess 20 products, the standard SUS was administered to 268 participants and the positive SUS to 698 participants. SUS scores were computed and the data analyzed using psychometric methods to explore how the two versions of the SUS differed. The standard and positive versions of the SUS yielded similar SUS scores. In addition, both versions of the scale demonstrated evidence in support of reliability and validity. Either version of the SUS can be used with confidence to measure subjective usability. Furthermore, the scores generated from both versions of the SUS can be directly compared. In situations where cognitive load, participants' spoken language, or item consistency with other surveys being given may be a factor, the positive SUS is a viable alternative to the standard SUS.