User Experience Testing Research Articles

Applying Large Language Model to User Experience Testing

Applying Large Language Model to User Experience Testing

Navigating Inclusivity: Exploring Accessibility in Oslo's Public Transport Mobile App.

This study investigates the accessibility of the Ruter mobile app, a major digital platform provided by the public transport authority for Oslo and Akershus counties in Norway. By employing a combination of case study analysis, accessibility audits, and user experience testing, the research evaluates the app's compliance with WCAG and its overall usability for individuals with disabilities. Despite the inclusion of several accessibility features, the findings reveal significant areas for improvement, particularly in contrast handling, form field support, and logical sequencing. Feedback from a group of six participants highlights the necessity for continuous enhancement of the app design to ensure it is user-friendly and accessible. This research emphasizes the importance of universal design principles and suggests that with targeted enhancements, Ruter AS can significantly improve the inclusivity and accessibility of its digital platforms.

‘We didn't need to know about everything all at once’

Tertiary student engagement with information and academic literacy online resources provided by centralized units is low. Although these resources are designed to prepare students for their assessments, they are often positioned peripherally to students’ learning environments. In attempting to position resources within students’ learning environments, our Library team conduct regular user experience (UX) testing of a course we have designed in our institution’s learning management system. Our two-year UX project has focused on the design and organization of resources that show students examples and guidance with how to do their assessments. Over two rounds of data collection and analysis, UX methods of usability testing, card sorting, low-fidelity wireframing, and high-fidelity prototyping helped identify student preferences for organising and accessing our resources. Findings indicate that students value the provision of relevant examples and guidance that show them what is expected in assessments and that they want direct access to that content as part of their courses. Implications for design include maximizing ease of use by positioning links to relevant examples and guidance in the assessment instructions and/or specific learning materials of courses, with the content being linked to positioned within a centralized repository in the same online learning environment.

PERANCANGAN USER INTERFACE DAN USER EXPERIENCE APLIKASI MONITORING EXPIRED MENGGUNAKAN METODE DOUBLE DIAMOND

Indomaret is one of the franchise business ventures which is a company owned by PT Indomarco Prismatama. At this time the process of monitoring products that are approaching the expiration date is still inefficient, often there are products whose expiration dates are missed and employees have difficulty finding products that will expire while there is no data and reminders about products that will expire. Therefore, a user interface design is needed before designing an application that can facilitate employees in managing expired products to be more efficient. In designing the user interface, researchers used the Double Diamond method which includes 4 stages, namely discover, define, develop, and deliver. For user experience testing, researchers adopted the User Experience Questionnaire (UEQ) method using 6 aspects with 26 items, including Attractiveness, Perpicuity, Effciency, Depandability, Stimulation, Novelty. The test results showed that t5 UEQ scales were on the "Excellent" assessment, namely on the Attractiveness, Efficiency, Dependability, Stimulation, Novelty scales. While one UEQ scale is at "Above Average", namely the Perspicuity scale.

Redesain User Interface pada Website Pudam Banyuwangi Menggunakan Metode User Centered Design

PUDAM Banyuwangi is a regional public company that provides clean water services to the community with branches in several areas. The PUDAM Banyuwangi website was built with the aim of making company information and services more accessible. It is known, during the creation of the website the company did a refresh which was carried out almost every five years to improve the functionality of the website. however, user experience testing has never been carried out measurably on the website user interface. initial testing was carried out using the User Experience Questionnaire (UEQ) as a user experience measurement tolls by getting results from 51 respondents showing the value of user experience on the website on 6 components, namely attractiveness (-0.12), clarity (0.15), efficiency (0.11), accuracy (-0.04), stimulation (-0.07), novelty (-0.29) so that it falls into the "Bad" criteria. To improve this, research was conducted using the User Centered Design (UCD) approach to analyze user needs and provide design update recommendations based on test results. The results showed that the design update increased the user experience evaluation value with an attractiveness value of (1.67), clarity (1.73), efficiency (1.71), accuracy (1.65), stimulation (1.55), novelty (1.51) so that it fell into the "Good" criteria. This can prove that this research provides recommendations for appropriate design updates to improve the functionality of the PUDAM Banyuwangi website in its development.

Evaluation of the Usability and User Experience of the Jaminan Kesehatan Nasional Mobile Application in Indonesia.

The National Health Insurance (Jaminan Kesehatan Nasional or JKN) mobile application represents an administrative transformation of health social insurance in Indonesia. The application's user base has quadrupled in the past 2 years. The study aimed to evaluate the usability and user experience of the JKN mobile application in Indonesia. The study administered an online survey featuring the System Usability Scale (SUS) questionnaire alongside a user experience assessment. The survey was disseminated via social media and garnered responses from 370 participants. The variables examined included usability and user experience, specifically focusing on attractiveness, perspicuity, efficiency, dependability, stimulation, and novelty. Data analysis was performed descriptively, and the results were benchmarked. The JKN mobile application had an average SUS score of 60.9 ± 13.47 (adjective rating "good," Grade "F," acceptability rating of "low marginal," and a percentile range of 34). User experience testing yielded the following results: attractiveness (1.09 ± 1.09), perspicuity (1.05 ± 1.17), efficiency (1.07 ± 1.20), dependability (1.03 ± 1.12), stimulation (0.88 ± 1.20), and novel-ty (0.58 ± 1.09). The benchmark for the JKN mobile application placed the user experience in the "below average" category. The evaluation of the usability and user experience of the JKN mobile application was generally positive. However, there is a need to improve the usability and all parameters of the user experience. Broader-scale research is needed to explore the factors that influence usage and user experience to promote the adoption of the JKN application and increase user satisfaction.

User experience testing methods: Conclusions from the literature

Industry 4.0 focuses on the digitalization of production processes and technological innovation. The concept of Industry 5.0 puts the focus on human-centricity, sustainability, and resilience at the heart of research and development and innovation (R&D&I) processes to allow industry to serve humanity with a long-term vision that considers planetary boundaries. Replacing the technology-driven approach with a fundamentally human-centric approach requires a deep understanding of the working environment and workers interacting with machines to optimize worker well-being, working conditions, and job outcomes. Analyzing computer work User eXperience (UX) in industrial environments is vital. However, user perceptions are usually hidden and a challenge to detect. Therefore, measuring and monitoring perceptions, emotional reactions, subjective elements, preferences, and attitudes in the relationships between usability, work performance, and workload is crucial. This study provides conclusions of a literature review on user experience studies focusing on UX testing methods and the disciplines linked to industrial diversification. Based on literature analysis, it identifies UX testing methods and create own grouping to analyze them. It also examines the disciplinary context of user experience testing.

User Interface and User Experience Design for Vidyanusa on Android Mobile Phones Using an Evaluation Framework

This research will evaluate the User Interface and User Experience (UI/UX) of Vidyanusa for Android mobile phones. User Experience will be measured using sub-variables such as attractiveness, agility, efficiency, dependability, stimulation, and novelty. The object of this research is User Experience. The results of this research will be used to evaluate Vidyanusa's Android mobile product to achieve a product that is suitable for users at the junior high school level. User experience testing on the Vidyanusa mobile application uses evaluation methods with query techniques. Evaluation is a test of the level of system usability and functionality that is carried out in the laboratory, in the field, or in collaboration with users. What is evaluated is the design and implementation, and the technique used is the evaluation technique with the User Experience Questionnaire. The results of the user experience testing of the Vidyanusa mobile application tend to be negative on the points of attractiveness, perspicuity, efficiency, dependability, and novelty based on the User Experience Questionnaire in the benchmark table, but positive on the point of stimulation.

Virtual Coaching for Home Rehabilitation - Evidence of an Empirical Study.

Older individuals often face disabilities or diseases that lower their quality of life (QoL). While inpatient rehabilitation can initially enhance QoL, there is often a lack of continuation at home. Virtual coaches (VCs) as specific embodied conversational agents promise appropriate support for home rehabilitation. They emerge as complementary digital aids to ensure care continuity. This paper presents the results of implementing a full-featured VC for older patients' home rehabilitation in a multi-stage study, summarizing the main results regarding QoL outcomes and user experience tests. The study confirms the intervention as an engaging means for rehabilitation (mostly above user experience thresholds) and improvements of QoL (>10% between experimental and control groups).

Cross-cultural adaptation and user-experience validation of the ACAD Toolkit

Design for learning involves the delicate interweaving of knowledge about learning and knowledge about design. This work is often carried out by heterogenous design teams in which members speak of and value different aspects of design, and different methods for evaluating these designs in use. The challenge of reconciling these often-competing demands is critical to the success of these teams. This short paper outlines work breaking new ground translating an educational design method, developed in English speaking contexts, for use in Spanish speaking contexts. Steeped in socio-cultural and socio[1]material awareness this project explores how the ACAD Toolkit—a set of tangible design related resources embodying networked learning ideals—shapes and is (re)shaped in and through the process of translation. Guided by two questions: (i) How can we explore the process of translating not only language but values and forms of practice? and (ii) How can the ACAD Toolkit be validated in new contexts? This qualitative study involves the thematic analysis of multimodal data including video and audio recordings, and artefacts produced during workshops. Our method builds on traditional cross[1]cultural processes of adaptation that involve adapting, expanding and splitting ideas and concepts in two stages: language translation and user-experience testing. Our analysis is, therefore, reported in two stages. In the first we explore the process of reaching agreement on a test set of translated resources, and in the second we explore how these resources are being enrolled in educational design work in new contexts. The newly translated resources have been tested in three workshops in two Spanish speaking educational settings (Spain and Argentina). After analysing the data from these workshops, the initial translation will be corrected, and instructions will be developed—in both languages—to improve future translations of the ACAD Toolkit and in its ongoing use in English contexts. These instructions and the processes through which they will be developed will produce potential research objects for future educational design research. This method, initially developed with educators in Australia and New Zealand, embodies the very heart of networked learning—the movement of people, objects, and ideas across contexts and time.

User Experience Testing of the Meta Quest 2 for Integration With the Virtual Reality Simulation for Dementia Coaching, Advocacy, Respite, Education, Relationship, and Simulation (VR-SIM CARERS) Program.

Caregivers (CGs) of persons with dementia (PWDs) face numerous challenges, including learning about the condition, managing behavioral symptoms, and prioritizing their own well-being. Virtual reality (VR) technology has emerged as a promising tool to adopt certain elements of existing CG psychoeducation programs, such as the Reitman Centre CARERS (coaching, advocacy, respite, education, relationship, and simulation) program, which has been shown effective in reducing CG burden and stress and building the required skills for caring for PWD. Recently, we have developed a VR prototype utilizing Meta Quest 2 (Meta, Menlo Park, CA, USA), which will be referred to as the (virtual reality simulation for dementia CARERS) VR-SIM CARERS program. This technical report aims to describe the early stages of intervention modeling by testing user experiences related to the hardware used. The Meta Quest 2 VR system is chosen for its accessibility and functionality, aiming to ensure widespread access. Through interviews and observational techniques, we explored CGs age-matched controls' attitudes, comfort, and proficiency with the Meta Quest 2 VR system, which are crucial for informing technological choices. Initial findings revealed mixed attitudes, comfort, and proficiency about the Meta Quest 2 VR system. Although further testing of the Meta Quest 2 VR system within the CG community is warranted, the interpretation of these preliminary results indicates that the VR-SIM CARERS program should have minimal technological skill requirements for user engagement or provide in-depth training resources for the CGs who choose to use the system.

Development of 'OXIS Classification' calibration website for dental researchers.

The variations in interproximal contact areas have been categorised open (O), point (X), straight (I), and curved contact (S). This is based on their shapes and termed as OXIS classification. The interactive OXIS calibration website was developed to assist researchers seeking appropriate knowledge, minimise the overload of material, optimise efficiency in calibration and, to provide repositories for clinicians, healthcare workers, and policymakers. The website was developed in two phases. The first phase of development included expert group discussion, Focus Group Discussion (FGD), and the implementation of FGD recommendations. The second phase emphasised registration and development of the interactive web portal on OXIS classification. The developed website was subjected to user experience testing, functional testing, performance testing, security testing, device, platform testing, and then hosted. Calibrated students and faculties evaluated it with the help of Suitability Assessment of Materials (SAM) and System Usability Scale (SUS). Finally, after amendments, the website was evaluated by non-calibrated researchers and multidisciplinary experts. The total agreement was 74% for overall SAM category. Pooled mean total score of SUS was 52.7 (SD 7.17; range 45-67.5), indicating an average score. The content of the developed website has been evaluated as "satisfactory" and its technical quality as "of higher standards".

From bard to Gemini: An investigative exploration journey through Google’s evolution in conversational AI and generative AI

The advent of artificial intelligence (AI) has significantly transformed various aspects of human life, particularly in information retrieval and assistance. This research presents a comprehensive evaluation of Gemini, previously known as Google Bard, a state-of-the-art AI chatbot developed by Google. Through a meticulous methodology encompassing both qualitative and quantitative approaches, this research aims to assess Gemini’s performance, usability, integration capabilities, ethical implications. Primary data collection methods, including user surveys and interviews, were utilized to gather towards the qualitative feedback on user experiences with Gemini, supplemented by secondary data analysis using tools such as Google Analytics to capture quantitative metrics. Performance evaluation involved benchmarking against other AI chatbots and technical analysis of Gemini’s architecture and training methods. User experience testing examined usability, engagement, and integration with Google Workspace and third-party services. Ethical considerations regarding data privacy, security, and biases in AI-generated content were also addressed, ensuring compliance with major regulations and promoting ethical AI practices. Acknowledging limitations and challenges inherent in the investigative exploration, data analysis was conducted using thematic and statistical methods to derive insights. The results and findings of this research offer valuable insights into the capabilities and limitations of Gemini, providing implications for future AI development, user interaction design, and ethical AI governance. By contributing to the ongoing discourse on AI advancements and their societal impact, this exploration facilitates informed decision-making and lays the groundwork for future research endeavors in the field of AI-driven conversational agents.

Optimisation of an online self-assessment tool for preconception health: user feedback and user-experience testing in people of reproductive age in Australia. (Preprint)

Optimisation of an online self-assessment tool for preconception health: user feedback and user-experience testing in people of reproductive age in Australia. (Preprint)

Designing immersive narratives of Jugra heritage site for cultural edutainment using mixed reality

Technology integration in the tourism industry has introduced new possibilities for enhancing visitor experiences and promoting cultural edutainment in heritage tourism. Mixed Reality (MxR) is a promising technology for achieving these goals. By leveraging MxR, cultural heritage sites can provide tourists with engaging and educational experiences, allowing them to delve into the rich history and cultural significance of these sites. However, the quality of content in MxR applications can significantly impact the overall experience and educational value. This research proposes best practices for MxR development in smart heritage tourism, ensuring the delivery of high-quality, culturally authentic, and educational experiences to visitors. A qualitative approach has been adopted to achieve this aim, utilizing the Speak Aloud method to collect data from participants who interacted with the MxR prototype. The study focuses on a case study of the Jugra heritage urban landscape, using it as a basis for developing an MxR prototype. The research encompasses two stages: the first is the design for prototype development, and the second is to test the application for user feedback. Through discussions on prototype design development and user experience testing, the research proposes a storyboard framework as a guideline for designing the MxR environment. This framework serves as a comprehensive blueprint, ensuring that each phase, from data analysis to the final Mixed Reality application, is meticulously designed and executed. Respondents tested the prototype to gather feedback on the prototype's outcomes, enabling refinement of the blueprint for subsequent levels of testing with heritage visitors. The results of this study contribute to the field of MxR development in smart heritage tourism. The research enables developers to create high-quality, culturally authentic, educational MxR experiences by offering guidelines and best practices. Therefore, the application of MxR in cultural heritage enhances the level of visitor engagement and facilitates a more profound comprehension of cultural heritage.

Detection and Grading of Radiographic Hand Osteoarthritis Using an Automated Machine Learning Platform.

Automated machine learning (autoML) platforms allow health care professionals to play an active role in the development of machine learning (ML) algorithms according to scientific or clinical needs. The aim of this study was to develop and evaluate such a model for automated detection and grading of distal hand osteoarthritis (OA). A total of 13,690 hand radiographs from 2,863 patients within the Swiss Cohort of Quality Management (SCQM) and an external control data set of 346 non-SCQM patients were collected and scored for distal interphalangeal OA (DIP-OA) using the modified Kellgren/Lawrence (K/L) score. Giotto (Learn to Forecast [L2F]) was used as an autoML platform for training two convolutional neural networks for DIP joint extraction and subsequent classification according to the K/L scores. A total of 48,892 DIP joints were extracted and then used to train the classification model. Heatmaps were generated independently of the platform. User experience of a web application as a provisional user interface was investigated by rheumatologists and radiologists. The sensitivity and specificity of this model for detecting DIP-OA were 79% and 86%, respectively. The accuracy for grading the correct K/L score was 75%, with a κ score of 0.76. The accuracy per DIP-OA class differed, with 86% for no OA (defined as K/L scores 0 and 1), 71% for a K/L score of 2, 46% for a K/L score of 3, and 67% for a K/L score of 4. Similar values were obtained in an independent external test set. Qualitative and quantitative user experience testing of the web application revealed a moderate to high demand for automated DIP-OA scoring among rheumatologists. Conversely, radiologists expressed a low demand, except for the use of heatmaps. AutoML platforms are an opportunity to develop clinical end-to-end ML algorithms. Here, automated radiographic DIP-OA detection is both feasible and usable, whereas grading among individual K/L scores (eg, for clinical trials) remains challenging.

Braille Script Translator Website

As technology rapidly evolves, ensuring inclusivity for all members of society, including visually impaired individuals, becomes imperative. This paper introduces Braille Script Translator, a versatile tool designed to address the communication gap between visually impaired individuals and teachers or parents working with them. The system offers a multimodal approach, allowing users to input Braille text via uploaded images, voice, or text. Through the integration of a comprehensive Braille database, the system translates Braille text to English and vice versa, ensuring reliable results. The output is presented through a user-friendly interface, prioritizing ease of use for all users. Performance evaluation focuses on accuracy and efficiency, while user experience testing provides valuable insights for further refinement. Braille Script Translator represents a significant step forward in enhancing accessibility and inclusivity in digital communication.

Interface design features and evaluation of batik 4.0 mobile application

The use of information and communication technology could increase the quantity and quality of small medium enterprises (SME) production, including batik industry. This study focuses on the development of batik 4.0, a custom batik mobile-based interface that makes it easier for customer which can be used to quickly produce high-quality digital batik designs. The findings of this study simplify the ordering process for potential clients who want to acquire custom batik designs. Ease of transactions, namely down payments (DP) where users can make advance payments, so that users are relieved in terms of payment transactions. In designing mobile devices, applications, and user interfaces (UI), it is important to consider the user experience (UX). This paper focuses on UX design rooted in the user-centered design (UCD) approach, placing emphasis on understanding user requirements and prioritizing empathy for users. This ensures the recognition of user needs and the creation of a high-fidelity prototype. Then it was validated by the UI experts to identify problems and user difficulties in interacting with the UI. The experts responded positively towards the application and suggest for prototype improvement. Lastly, UX testing; based on the user experience questionnaire (UEQ)-S benchmark results, the batik 4.0 mobile is included in the “Excellent” category.

User-experience testing of an evidence-to-decision framework for selecting essential medicines.

Essential medicine lists (EMLs) are important medicine prioritization tools used by the World Health Organization (WHO) EML and over 130 countries. The criteria used by WHO's Expert Committee on the Selection and Use of Essential Medicines has parallels to the GRADE Evidence-to-Decision (EtD) frameworks. In this study, we explored the EtD frameworks and a visual abstract as adjunctive tools to strengthen the integrate evidence and improve the transparency of decisions of EML applications. We conducted user-experience testing interviews of key EML stakeholders using Morville's honeycomb model. Interviews explored multifaceted dimensions (e.g., usability) on two EML applications for the 2021 WHO EML-long-acting insulin analogues for diabetes and immune checkpoint inhibitors for lung cancer. Using a pre-determined coding framework and thematic analysis we iteratively improved both the EtD framework and the visual abstract. We coded the transcripts of 17 interviews with 13 respondents in 103 locations of the interview texts across all dimensions of the user-experience honeycomb. Respondents felt the EtD framework and visual abstract presented complementary useful and findable adjuncts to the traditional EML application. They felt this would increase transparency and efficiency in evidence assessed by EML committees. As EtD frameworks are also used in health practice guidelines, including those by the WHO, respondents articulated that the adoption of the EtD by EML applications represents a tangible mechanism to align EMLs and guidelines, decrease duplication of work and improve coordination. Improvements were made to clarify instructions for the EtD and visual abstract, and to refine the design and content included. 'Availability' was added as an additional criterion for EML applications to highlight this criterion in alignment with WHO EML criteria. EtD frameworks and visual abstracts present additional important tools to communicate evidence and support decision-criteria in EML applications, which have global health impact. Access to essential medicines is important for achieving universal health coverage, and the development of essential medicine lists should be as evidence-based and trustworthy as possible.

Web Accessibility Evaluation Using WCAG 2.0 and User-Centered Design Method

ITENAS e-learning has problems such as the flow of access to complex features, and the color of the display and buttons that overlap with the background color. The design of the web interface in this study uses the WCAG 2.0 standard. With the success points of Criterion 1.4.11 Non-text Contrast in the Produce Design Solution stage in the User-Centered Design method. The results of the WCAG 2.0 test show the highest contrast ratio of 15.16:1 which has met the required contrast ratio, the criteria are successfully applied. In the design evaluation stage, this study used SUS to test the usability of the prototype. User experience testing in this study uses UEQ. The results of the evaluation of the usability of the interface with SUS in the initial test resulted in a score of 38.93 which was in grade F or bad to grade B with a score of 74.38 or good after the redesign process. The results of user experience testing with UEQ obtained an average Efficiency scale of 1.875. The Attractiveness scale is 1.667, the Clarity scale is 1.750, the Accuracy scale is 1.375, the Stimulation scale is 1.525, the last is the Novelty scale of 1.350 according to the UEQ average scale which is at a positive level. From the results of the two SUS and UEQ tests, the Usability aspect and the User Experience aspect of the Itenas e-learning web can be said to have improved.