Usability Measures Research Articles

A Novel Cognitive Behavioral Therapy-Based Generative AI Tool (Socrates 2.0) to Facilitate Socratic Dialogue: Protocol for a Mixed Methods Feasibility Study.

Digital mental health tools, designed to augment traditional mental health treatments, are becoming increasingly important due to a wide range of barriers to accessing mental health care, including a growing shortage of clinicians. Most existing tools use rule-based algorithms, often leading to interactions that feel unnatural compared with human therapists. Large language models (LLMs) offer a solution for the development of more natural, engaging digital tools. In this paper, we detail the development of Socrates 2.0, which was designed to engage users in Socratic dialogue surrounding unrealistic or unhelpful beliefs, a core technique in cognitive behavioral therapies. The multiagent LLM-based tool features an artificial intelligence (AI) therapist, Socrates, which receives automated feedback from an AI supervisor and an AI rater. The combination of multiple agents appeared to help address common LLM issues such as looping, and it improved the overall dialogue experience. Initial user feedback from individuals with lived experiences of mental health problems as well as cognitive behavioral therapists has been positive. Moreover, tests in approximately 500 scenarios showed that Socrates 2.0 engaged in harmful responses in under 1% of cases, with the AI supervisor promptly correcting the dialogue each time. However, formal feasibility studies with potential end users are needed. This mixed methods study examines the feasibility of Socrates 2.0. On the basis of the initial data, we devised a formal feasibility study of Socrates 2.0 to gather qualitative and quantitative data about users' and clinicians' experience of interacting with the tool. Using a mixed method approach, the goal is to gather feasibility and acceptability data from 100 users and 50 clinicians to inform the eventual implementation of generative AI tools, such as Socrates 2.0, in mental health treatment. We designed this study to better understand how users and clinicians interact with the tool, including the frequency, length, and time of interactions, users' satisfaction with the tool overall, quality of each dialogue and individual responses, as well as ways in which the tool should be improved before it is used in efficacy trials. Descriptive and inferential analyses will be performed on data from validated usability measures. Thematic analysis will be performed on the qualitative data. Recruitment will begin in February 2024 and is expected to conclude by February 2025. As of September 25, 2024, overall, 55 participants have been recruited. The development of Socrates 2.0 and the outlined feasibility study are important first steps in applying generative AI to mental health treatment delivery and lay the foundation for formal feasibility studies. DERR1-10.2196/58195.

Social validity of digital social incentives in the treatment of substance use disorders.

Substance use disorders (SUDs) affect millions and have substantial negative consequences for individuals and society. Social incentives that leverage social networks for reinforcement or feedback have been used to improve health behaviors such as physical activity. This study investigated the feasibility, acceptability, and usability of a novel digital social incentive system embedded into a web- and smartphone-based platform for SUD recovery. The system leveraged a preexisting care team to deliver social incentives following notifications on recovery-related goal completion and abstinence to members undergoing SUD treatment. In total, 243 notifications were sent to care-team members, resulting in 117 social incentives, nearly all of which (99.15%) were coded as positive. Treatment members and care-team members provided favorable endorsements on acceptability and usability measures. Some areas of improvement were identified, such as increasing personalization and transparency. This digital social incentive system was feasible, acceptable, and usable as an adjunct treatment component for SUD recovery.

Behaviorally Measuring Usability by Analyzing Users’ Mouse Movement Efficiency

Usability—the extent to which a system can be operated to achieve specified goals with effectiveness, efficiency, and satisfaction—is a hallmark of many successful systems. Identifying usability concerns with existing measures, however, can be problematic. We use biased competition theory (BCT) to explain how interfaces with low usability create attentional interference, which can be measured through mouse movement efficiency (MME). We test our hypotheses in four studies. Study 1 manipulates attentional interference and examines its influence on MME using eye tracking to validate BCT as an appropriate theoretical lens for our research. Study 2 manipulates usability and shows similar markers of attentional interference in MME on a website with two different populations. Study 3 is a field test that compares the reliability of MME in ranking the usability of different components of a commercial web application to rankings produced by a common perceptual measure of usability: perceived ease of use. Collaborating with a Fortune 100 software vendor, Study 4 further demonstrates the efficacy of MME in ranking the usability of various system components compared to a standard industry measure of usability across three different products. The results consistently show that MME can help identify usability differences among parts of systems. The results have implications for facilitating more efficient and temporally precise usability research and mass-deployable usability testing.

Physicians' Views of a Digital Intervention to Support Respiratory Patients: Take-a-Breath Pre-Pilot Usability Study.

Although eHealth interventions are increasingly recognized as a useful tool to support healthcare, relatively few studies focus on the physician-end's usability. This study aims to evaluate the Healthcare Professional's (HCP) platform of the Take-A-Breath project, a Greek initiative for personalized respiratory disease monitoring, training and self-management. The pre-pilot usability study, involving 10 participants, combines qualitative methods, behavioral observations, and standardized measures of user experience and usability. While relatively high scores indicate overall acceptance, concerns are also discussed, particularly related with the volume of information provided and actions available to the users, hindering the usability of the system due to an overload effect. Findings emphasize also the need for more tailored in-app wordings as well as the integration of similar systems with the already set up electronic health record systems. This study contributes to understanding digital intervention success among HCPs in respiratory healthcare.

Information Platform for Antibiotics.

Antibiotic stewardship is continuously evolving to incorporate results from novel research, clinical findings, and specialist recommendations. Numerous dedicated information sources, including web-based solutions, are available to keep medical practitioners informed. However, the provided information is often extensive, requiring users to extract the relevant facts. This study aimed to deliver an information platform that provides references, links, and information in a straightforward and engaging manner. Implementing a high-fidelity prototype prioritized medical and patient-oriented functionalities, structured around questions and quizzes. Additionally, the platform offers access to professional references, such as official healthcare guidelines and scientific articles. The development process adhered to design principles and included user testing with established usability measures (SUS, Nielsen's heuristics), resulting in satisfactory scores from IT experts and somewhat lower scores from users. Although designed to cater to a broader range of users, more work is needed to improve usability for the general public.

Use of Resources in Physical Education

The purpose was to develop an innovation in Physical Education (PE) through a quantitative approach that compared the use of text-based and media-based resources concerning academic performance, usability and satisfaction with the didactic unit, the methodology and the performance of the teacher. Following an experimental design, this research assigned upper-level high school participants (N=28) to text-based and media-based conditions. Measures of academic performance, usability and satisfaction were collected before and after the unit selected according to the correlational format. No significant differences were found regarding satisfaction (p = .65; p = .68) between conditions. Slightly significant differences were found regarding academic performance in both conditions (t = 91, p < .01; t = 105, p < .01). To add, media-based condition showed significant relationships related to teaching work satisfaction (rho = 0.67, p = .01) and usability (rho = 0.56, p = .04). The research concluded the no difference in satisfaction using media resources, but a slight increase in academic performance, an easy perception of media resources besides its high usability and a positive teaching work assessment due to the involvement of the figure.

Expert gaze as a usability indicator of medical AI decision support systems: a preliminary study

Given the current state of medical artificial intelligence (AI) and perceptions towards it, collaborative systems are becoming the preferred choice for clinical workflows. This work aims to address expert interaction with medical AI support systems to gain insight towards how these systems can be better designed with the user in mind. As eye tracking metrics have been shown to be robust indicators of usability, we employ them for evaluating the usability and user interaction with medical AI support systems. We use expert gaze to assess experts’ interaction with an AI software for caries detection in bitewing x-ray images. We compared standard viewing of bitewing images without AI support versus viewing where AI support could be freely toggled on and off. We found that experts turned the AI on for roughly 25% of the total inspection task, and generally turned it on halfway through the course of the inspection. Gaze behavior showed that when supported by AI, more attention was dedicated to user interface elements related to the AI support, with more frequent transitions from the image itself to these elements. When considering that expert visual strategy is already optimized for fast and effective image inspection, such interruptions in attention can lead to increased time needed for the overall assessment. Gaze analysis provided valuable insights into an AI’s usability for medical image inspection. Further analyses of these tools and how to delineate metrical measures of usability should be developed.

Penerapan System Usability Scale dalam Pengukuran Kebergunaan Website SMKN 13 Bandung

SMKN 13 Bandung is a vocational school that prioritizes information technology, most interactions and activities utilize information technology in order to achieve the school's vision and mission. Based on observations and problems found on the SMKN 13 Bandung website, there has been no attempt to evaluate measurements of user usability and satisfaction when visiting this website. This is to ensure that the SMKN 13 Bandung website in the future can be developed and operated with optimal quality according to the experience and needs of its users. In measuring usability, this research uses the System Usability Scale (SUS) method. This is because SUS is a valid and reliable tool for testing usability even on small samples. The SUS score results obtained from measurements on the SMKN 13 Bandung website resulted in an average SUS score of 71.29. These results show that the SMKN 13 Bandung website is quite usable, acceptable but still needs to be evaluated and developed to a better stage. The results of the measurements carried out can be the first step for evaluating the SMKN 13 Bandung’s website.

Effects of training methods on performance of a scaffolding task for workers with different spatial ability

With software developments and advances in display technologies substantially improved, augmented reality (AR) application has gained popularity. In this study, we discuss using classic PowerPoint and AR for two kinds of scaffolding tasks (task-lifeline assembly and hedge assembly) for users with different spatial ability. We considered both objective and subjective measures of performance, i.e., correct rate and system usability and the ITC-sense of presence inventory (ITC-SOPI) scale. The results of the study show that participants using AR achieved higher operating performance than those using PowerPoint. Furthermore, the users’ learning effect was influenced by spatial ability when using PowerPoint. Participants with high spatial ability achieved higher performance than participants with low spatial ability in PowerPoint. However, participants who used AR as a training method did not show significantly different operating performance at different levels of spatial ability. Consequently, AR was believed to be a potential method for enhancing training performance.

"For the first time…I am seriously fighting burnout": clinician experiences with a challenging electronic health record transition.

The Department of Veterans Affairs (VA) is transitioning from its legacy electronic health record (EHR) to a new commercial EHR in a nationwide, rolling-wave transition. We evaluated clinician and staff experiences to identify strategies to improve future EHR rollouts. We completed a convergent mixed-methods formative evaluation collecting survey and interview data to measure and describe clinician and staff experiences. Survey responses were analyzed using descriptive statistics; interview transcripts were coded using a combination of a priori and emergent codes followed by qualitative content analysis. Qualitative and quantitative findings were compared to provide a more comprehensive understanding of participant experience. Employees of specialty and primary care teams at the first nationwide EHR transition site agreed to participate in our study. We distributed surveys at 1-month pre-transition, 2 months post-transition, and 10 months post-transition to each of the 68 identified team members and completed longitudinal interviews with 30 of these individuals totaling 122 semi-structured interviews. Interview participants reported profoundly disruptive experiences during the EHR transition that persisted at 1-year post implementation. Survey responses indicated training difficulties throughout the transition, and sharp declines (P ≤ .05) between pre- and post-go-live measures of EHR usability and increase in EHR burden that were perceived to be due in part to system inefficiencies, discordant positive messaging that initially ignored user challenges, and inadequate support for and attention to ongoing EHR issues. Participants described persistent high levels of stress associated with these disruptions. Our findings highlight strategies to improve employee experiences during EHR transitions: (1) working with Oracle Cerner to resolve known issues and improve usability; (2) role-based training with opportunities for self-directed learning; (3) peer-led support systems and timely feedback on issues; (4) messaging that responds to challenges and successes; and (5) continuous efforts to support staff with issues and address clinician and staff stress and burnout. Our findings provide relevant strategies to navigate future EHR transitions while supporting clinical teams.

Prototype of a web tool for psycho-pedagogical intervention and stimulation of children with ADHD

Digital tools for students with ADHD have been scarcely implemented in current education due to the challenges they pose. Therefore, rigorous research, from the perspective of web application development, will strengthen students' formative process. In the present study, a web platform prototype was developed to automate the process of psycho-pedagogical intervention in children aged 5 and 12 with ADHD, focused on evaluating how this tool contributes to the process of psycho-pedagogical stimulation. The prototype was developed using XP methodology divided into two general phases and semi-structured interview techniques and structured surveys to gather relevant information. The first general phase involved the construction of the prototype, which included applying the interviews to obtain the system requirements and the BPMN process model of the general pedagogical approach. Interface design, architecture design, and prototype development were also carried out. The second general phase evaluated the system through acceptance and load and stress tests. Finally, usability evaluation was conducted following ISO 25010 and ISO 25040 standards. The usability measure obtained was "good," defined according to decision criteria, allowing us to infer the impact of implementing web tools.

A Platform for Emotional Evaluation and Psychological Support: Development and Usability Evaluation using SUMI

Mental illness has been ranked as one of the diseases with the highest number of cases globally. Generally, an individual can only be diagnosed with mental illness by undergoing proper medical procedures by the experts. On the other hand, due to the presence of different beliefs in various cultures and ethnicities, an individual is also claimed to have mental illness by the assumptions made based on the symptoms that the individual is having. This is additionally contributed by feelings of reluctance and being reserved, where the individual would subsequently resort not to go for a proper medical assessment. Therefore, to overcome this problem, this present project was developed as a real-time platform for users with mental health conditions to freely share their problems. Specifically, a rule-based algorithm of Depression Anxiety Stress Scales (DASS) was implemented in the Emotional Evaluation and Psychological Support (PEEPS). Furthermore, a real-time platform for PEEPS was developed and the effectiveness of the whole system was evaluated by using the Software Usability Measurement Inventory (SUMI). The development of the project would benefit the users to evaluate their own level of depression, stress and anxiety through the DASS test and get necessary counselling through the chat feature. According to the results collected, it was shown that the users were satisfied with the PEEPS application. This application is certainly relevant for governmental and non-governmental institutions that are interested in supporting and improving mental health-related issues. Future studies may consider modifying the machine learning technique to anticipate mental health issues as well as adding a few other elements to the application, such as personalized feedback or patient recommendations.

Evolving usability heuristics for visualising Augmented Reality/Mixed Reality applications using cognitive model of information processing and fuzzy analytical hierarchy process

AbstractThe pace of technological advancement is accelerating, and one of the latest developments is the emergence of Augmented Reality (AR) and Mixed Reality (MR) glasses as an extension of smartphones. The key to success lies in innovative research and technology that can reach a wide audience. To ensure a positive user experience, AR/MR glasses must offer interfaces that are easy to use, memorable, and leave a lasting impression. While Nielsen's heuristics are widely accepted as the standard for usability, it is clear that non‐traditional applications require a rethinking of these heuristics to best suit their unique needs. A fresh usability heuristic for augmented and MR applications is designed by combining and modifying the existing models, such as Nielsen's 10 heuristics, Technology Acceptance Model, and Software Usability Measurement Inventory. The resulting framework incorporates 21 main heuristics and 60 sub heuristics. The 21 main heuristics are further grouped into the Norman's cognitive theory model based on the three levels of processing. The industry experts evaluated and validated the usability framework and established a higher level of effectiveness in identifying more usability problems compared with Nielsen's heuristics.

Quality Analysis of SMA Negeri 1 Sokaraja Website using Eye Tracking Method

The role of websites in academic management and enhancing the reputation of a company or institution is very important. The quality of website services also has a crucial role in the world of education. Based on the results of interviews with the person in charge of the SMA Negeri 1 Sokaraja website, it was found that there were problems, namely the satisfaction of website users was not yet known and there were several pages that were not responsive, such as in the homepage/home area, which needed to be improved or optimized. This research involves a number of respondents who will give their views on the SMA Negeri 1 Sokaraja website. The analysis results obtained from this research are expected to provide a clear understanding of the extent to which this website meets the quality standards set by the Eye Tracking method. In addition, the analysis results will also provide a detailed evaluation of each aspect of the website's quality. In this research, the Eye Tracking method is used to measure where users look, how long they look, and the order in which they look. There are aspects of usability measurement, namely efficiency, effectiveness and satisfaction. SMA Negeri 1 Sokaraja website users are the respondents of this research. The results of usability testing with the USE Questionier method with a feasibility percentage of 84.17% with a very feasible category which means the website is very feasible for users. The results of usability testing with the eye-tracking method that each task is well received by users, but the effectiveness of delivering information reaches a score of 73.14, this score is classified as quite effective

Preliminary Virtual Constraint-Based Control Evaluation on a Pediatric Lower-Limb Exoskeleton.

Pediatric gait rehabilitation and guidance strategies using robotic exoskeletons require a controller that encourages user volitional control and participation while guiding the wearer towards a stable gait cycle. Virtual constraint-based controllers have created stable gait cycles in bipedal robotic systems and have seen recent use in assistive exoskeletons. This paper evaluates a virtual constraint-based controller for pediatric gait guidance through comparison with a traditional time-dependent position tracking controller on a newly developed exoskeleton system. Walking experiments were performed with a healthy child subject wearing the exoskeleton under proportional-derivative control, virtual constraint-based control, and while unpowered. The participant questionnaires assessed the perceived exertion and controller usability measures, while sensors provided kinematic, control torque, and muscle activation data. The virtual constraint-based controller resulted in a gait similar to the proportional-derivative controlled gait but reduced the variability in the gait kinematics by 36.72% and 16.28% relative to unassisted gait in the hips and knees, respectively. The virtual constraint-based controller also used 35.89% and 4.44% less rms torque per gait cycle in the hips and knees, respectively. The user feedback indicated that the virtual constraint-based controller was intuitive and easy to utilize relative to the proportional-derivative controller. These results indicate that virtual constraint-based control has favorable characteristics for robot-assisted gait guidance.

Semiquantum Key Distribution Using Initial States in Only One Basis Without the Classical User Measuring

AbstractFrom the perspective of resource theory, it is interesting to achieve the same quantum task using as few quantum resources as possible. Semiquantum key distribution (SQKD), which allows a quantum user to share a confidential key with a classical user who prepares and operates qubits on only one basis, is an important example for studying this issue. To further limit the quantum resources used by users, in this paper, the first SQKD protocol is constructed, which restricts the quantum user to prepare quantum states on only one basis and removes the classical user's measurement capability. Furthermore, it is proven that the constructed protocol is secure against the restricted attack by deriving a key rate expression of the error rate in the asymptotic scenario. The work in this paper provides inspiration for achieving quantum superiority with minimal quantum resources.

Development and Validation of Cloud Applications Usability Development Scale

Cloud computing has revolutionized technology and education by providing scalable virtual resources accessible via the internet. This study emphasizes the importance of usability in cloud services, discussing various scales and metrics while highlighting their strengths and weaknesses. It underscores the necessity for a comprehensive usability scale that integrates both subjective and objective measures to suit diverse user scenarios and populations. The article introduces the Cloud Applications/Services Usability Development Scale, comprising five usability dimensions: Capable, Personal, Reliable, Valuable, and Secure. To validate the instrument, factor analysis was performed on 28 items for each construct, utilizing Principal Component Analysis and Varimax Rotation. The Kaiser-Meyer-Olkin (KMO) and Bartlett’s Test were applied to assess sampling adequacy. Construct validity and reliability were scrutinized using AMOS 21, a statistical analysis software for structural equation modeling (SEM). A confirmation factor analysis was employed to assess both converging and discriminating validity. A survey conducted among actual cloud application users, specifically Google Drive users in Lahore’s public libraries, demonstrated the strong construct validity and reliability of the developed instrument. The Cloud Applications/Services Usability Development Scale proved effective in evaluating the usability of cloud-based applications. This study not only contributes valuable insights to the development and validation of usability measurement tools for cloud applications but also addresses a significant gap in local and international literature. Furthermore, it serves as a model for creating and validating measurement tools for cloud services, offering practical implications for usability testing and methodological advancements in the field of cloud computing.

Co-design and usability of an interactive web-based fertility decision aid for transgender youth and young adults.

To develop a patient- and family-centered Aid For Fertility-Related Medical Decisions (AFFRMED) interactive website targeted for transgender and nonbinary (TNB) youth/young adults and their parents to facilitate shared decision-making about fertility preservation interventions through user-centered participatory design. TNB youth/young adults interested in or currently receiving pubertal suppression or gender-affirming hormone treatment and parents of eligible TNB youth/young adults were recruited to participate in a series of iterative human-centered co-design sessions to develop an initial AFFRMED prototype. Subsequently, TNB youth/young adults and parents of TNB youth/young adults were recruited for usability testing interviews, involving measures of usability (i.e., After Scenario Questionnaire, Net Promotor Score, System Usability Scale). Twenty-seven participants completed 18 iterative co-design sessions and provided feedback on 10 versions of AFFRMED (16 TNB youth/young adults and 11 parents). Nine TNB youth/young adults and six parents completed individual usability testing interviews. Overall, participants rated AFFRMED highly on measures of acceptability, appropriateness, usability, and satisfaction. However, scores varied by treatment cohort, with TNB youth interested in or currently receiving pubertal suppression treatment reporting the lowest usability scores. We co-created a youth- and family-centered fertility decision aid prototype that provides education and decision support in an online, interactive format. Future directions include testing the efficacy of the decision aid in improving fertility and fertility preservation knowledge, decisional self-efficacy, and decision satisfaction.

HOW COVID-19 CHANGED CONSUMER BEHAVIOR TRENDS IN TÜRKİYE?

COVID-19 has altered many aspects of daily life, notably impacting shopping methods. This study examines how these changes affect consumer behavior, focusing on norm sensitivity, risk perception, and digital advertising. The purpose of this research is to understand the role of norm sensitivity, risk perception, and digital advertising as mediators in the relationship between COVID-19 and changes in consumer behavior. Qualitative data is collected from online meetings with 41 participants aged between 19 and 52 and it is analyzed by using MAXQDA. The unique aspect of this study is its qualitative approach and having a relatively large sample size compared to typical qualitative studies. The findings indicate higher sensitivity towards online shopping during COVID-19, perceiving traditional face-to-face shopping as risky mostly due to virus transmission, increased trust in digital channels, the undeniable effects of digital advertisements on purchasing, and the importance of making research before shopping. Furthermore, the findings revealed that shopping methods shifted from traditional face-to-face to online shopping during COVID-19 even in the early times of the pandemic. Based on these findings, businesses should prioritize enhancing their online shopping experiences by improving website usability and security measures. They may also implement safety protocols in physical stores to alleviate consumer concerns. Leveraging targeted digital advertising strategies and investing in consumer research can help businesses adapt to evolving consumer preferences and behaviors.

A Randomized Evaluation of MoodFX, a Patient-Centred e-Health Tool to Support Outcome Measurement for Depression: Une évaluation randomisée de MoodFX, un outil de santé en ligne centré sur le patient pour soutenir la mesure du résultat dans la dépression.

e-Health tools using validated questionnaires to assess outcomes may facilitate measurement-based care for psychiatric disorders. MoodFX was created as a free online symptom tracker to support patients for outcome measurement in their depression treatment. We conducted a pilot randomized evaluation to examine its usability, and clinical utility. Patients presenting with a major depressive episode (within a major depressive or bipolar disorder) were randomly assigned to receive either MoodFX or a health information website as the intervention and control condition, respectively, with follow-up assessment surveys conducted online at baseline, 8 weeks and 6 months. The primary usability outcomes included the percentage of patients with self-reported use of MoodFX 3 or more times during follow up (indicating minimally adequate usage) and usability measures based on the System Usability Scale (SUS). Secondary clinical outcomes included the Quick Inventory of Depressive Symptomatology, Self-Rated (QIDS-SR) and Patient Health Questionnaire (PHQ-9). Forty-nine participants were randomized (24 to MoodFX and 25 to the control condition). Of the 23 participants randomized to MoodFX who completed the user survey, 18 (78%) used MoodFX 3 or more times over the 6 months of the study. The mean SUS score of 72.7 (65th-69th percentile) represents good usability. Compared to the control group, the MoodFX group had significantly better improvement on QIDS-SR and PHQ-9 scores, with large effect sizes and higher response rates at 6 months. There were no differences between conditions on other secondary outcomes such as functioning and quality of life. MoodFX demonstrated good usability and was associated with reduction in depressive symptoms. This pilot study supports the use of digital tools in depression treatment.