User-centered Evaluation Research Articles

Teacher-centric educational recommender systems in K12 practice: Usage and evaluation.

Personalized learning is hard to apply in classroom practice due to the workload of teachers. Recommender systems can help teachers make choices and hence personalize the learning environment. However, few studies have investigated how teachers use recommender systems in their practice. Educational recommender systems also require more investigation into how teachers assess recommendations to provide better systems in the future. Therefore, this research aimed to provide a better insight into the practical application of a teacher-centric educational recommender system, conduct a user-centric evaluation of the recommender system, and present teacher beliefs on the use of educational recommender systems for personalized learning. Our results indicated that teachers similarly used the recommender system by creating new lessons rather than adapting suggested lessons. Their evaluations of the recommender system highlighted the importance of accuracy and utility. Diversity and serendipity were considered to be less important. Teachers perceived recommender systems as a first step to personalized learning and as a tool for simplifying such learning. However, the current system did not affect the quality of personalized learning. Based on these results, we propose to focus more on accuracy and utility as opposed to diversity and serendipity for teacher-centric recommender system evaluation. For personalized learning, we conclude that, according to teachers, recommender systems can lead to more effective personalization by allowing teachers to focus on student needs.

An Overview of the Empirical Evaluation of Explainable AI (XAI): A Comprehensive Guideline for User-Centered Evaluation in XAI

Recent advances in technology have propelled Artificial Intelligence (AI) into a crucial role in everyday life, enhancing human performance through sophisticated models and algorithms. However, the focus on predictive accuracy has often resulted in opaque black-box models that lack transparency in decision-making. To address this issue, significant efforts have been made to develop explainable AI (XAI) systems that make outcomes comprehensible to users. Various approaches, including new concepts, models, and user interfaces, aim to improve explainability, build user trust, enhance satisfaction, and increase task performance. Evaluation research has emerged to define and measure the quality of these explanations, differentiating between formal evaluation methods and empirical approaches that utilize techniques from psychology and human–computer interaction. Despite the importance of empirical studies, evaluations remain underutilized, with literature reviews indicating a lack of rigorous evaluations from the user perspective. This review aims to guide researchers and practitioners in conducting effective empirical user-centered evaluations by analyzing several studies; categorizing their objectives, scope, and evaluation metrics; and offering an orientation map for research design and metric measurement.

Emerging Trends in Real-time Recommendation Systems: A Deep Dive into Multi-behavior Streaming Processing and Recommendation for E-commerce Platforms

This paper aims to assess the effectiveness of recommendation systems, focusing on multi-behavior streaming processing to enhance accuracy. It studies recommendation system performance using AI and data science, focusing on user behaviour, algorithm refinement, and satisfaction. E-commerce, streaming, and social media datasets with 93 anonymous participants were employed in experiments. These files from 93 anonymized users show typical internet use. Data anonymization and strict data management ensured anonymity. These databases track clicks, views, purchases, and ratings for empirical research and model evaluation. Collaborative filtering, matrix factorization, and TensorFlow/Keras train and assess application-specific recommendation models. Multi-behavior streaming processing and recommendation accounts assess each method's pros and downsides, system correctness, efficacy, and user involvement. The outcome compares domain-wide recommendation system precision, recall, NDCG, and conversion rate. Multi-behavior streaming processing adapts to user preferences and interactions to improve model accuracy and adaptability. Multi-behavior streaming processing improved model accuracy and flexibility by reacting to user inputs and choices. With error margins for all significant metrics, statistical significance was confirmed. The findings suggest that recommendation systems with real-time adaptation and multi-behavior streaming processing can improve user satisfaction and engagement in the changing digital landscape. It encourages algorithm advancement, model interpretation, user-centric evaluation, and ethics to improve information retrieval and personalisation. The study concluded that algorithm refinement, transparent model interpretation, user-centric evaluation, and ethical problems including data protection and bias mitigation increase information retrieval and personalisation. For durable and flexible recommendation systems, research should improve multi-behavior processing and sectoral applications.

MobiTangibles: Enabling Physical Manipulation Experiences of Virtual Precision Hand-Held Tools' Miniature Control in VR.

Realistic simulation for miniature control interactions, typically identified by precise and confined motions, commonly found in precision hand-held tools, like calipers, powered engravers, retractable knives, etc., are beneficial for skill training associated with these kinds of tools in virtual reality (VR) environments. However, existing approaches aiming to simulate hand-held tools' miniature control manipulation experiences in VR entail prototyping complexity and require expertise, posing challenges for novice users and individuals with limited resources. Addressing this challenge, we introduce MobiTangibles-proxies for precision hand-held tools' miniature control interactions utilizing smartphone-based magnetic field sensing. MobiTangibles passively replicate fundamental miniature control experiences associated with hand-held tools, such as single-axis translation and rotation, enabling quick and easy use for diverse VR scenarios without requiring extensive technical knowledge. We conducted a comprehensive technical evaluation to validate the functionality of MobiTangibles across diverse settings, including evaluations for electromagnetic interference within indoor environments. In a user-centric evaluation involving 15 participants across bare hands, VR controllers, and MobiTangibles conditions, we further assessed the quality of miniaturized manipulation experiences in VR. Our findings indicate that MobiTangibles outperformed conventional methods in realism and fatigue, receiving positive feedback.

An User-Centered Evaluation of Two Socially Assistive Robots Integrated in a Retirement Home

Socially assistive robots are receiving a growing interest in the health and social care sectors. They are considered a promising technology to add value to the work of caregivers, and relieve them of simple and repetitive tasks. However, these robots currently face significant difficulties when deployed in everyday scenarios due to a number of factors. Most of these factors are related to insufficient consideration of the user perspective and incorrect evaluation procedures. This paper aims to address these issues. Its objective is to analyze the long-term accessibility, usability, social acceptance and user experience for two different socially assistive robots performing the same tasks in a retirement home. The evaluation procedure is based on a framework specifically designed to consider these criteria. Collected results show that both robots received an overall positive feedback. Although the number of users participating in the evaluation was not very high, due to the chosen recruitment criteria and the period of activity of this research project, during the COVID19 pandemic, these results allow to extract relevant insights towards a meaningful use of social robots in shared social care contexts.

Depth-based interactive assistive system for dementia care

Dementia often leads to reduced autonomy and the need for help with activities of daily living, such as the use of the lavatory. In response, we present ToiletHelp, a novel system to provide automated, step-by-step guidance to individuals with dementia when using the lavatory. In such an intimate environment, it is crucial to maintain the user’s privacy, which ToiletHelp achieves by sensing only depth maps. ToiletHelp uses a depth sensor to detect when the user is in need of assistance and tailors the guidance accordingly, using a combination of verbal and visual prompts. Our main contribution is the development of the first assistance system for people with dementia in such a sensitive environment as the lavatory. In addition, what sets ToiletHelp apart is its comprehensive involvement of 60 participants, 33 end-users and 27 healthcare workers, throughout its development process. A user-centered evaluation validates the interaction design, demonstrating enhanced feelings of security and independence among older adults, as well as successful comprehension of instructions by participants with dementia. Caregivers report high satisfaction with the system’s modalities and confirm its effectiveness in reducing their workload while assisting end-users. These findings not only validate the success of ToiletHelp and underscore its potential benefits but also contribute to the field of human-computer interaction design. Our comprehensive analysis of the results provides valuable information for researchers developing solutions for individuals with dementia, which is especially valuable given the limited existing literature on the subject.

Empowering geoportals HCI with task-oriented chatbots through NLP and deep transfer learning

ABSTRACT In the past ten years, chatbot development has matured to become one of the most well-distinguished outcomes of artificial intelligence. Despite some criticism, Bing AI, ChatGPT and other natural language processing (NLP) products of similar nature are becoming more popular. The creation of chatbots can close several gaps in geographic information retrieval as well. This research introduces and successfully implements, for the first time, a model for integrating task-oriented chatbots into geoportals, with the goal of easing user requests, improving access to geospatial services, and fostering human-computer interactions (HCI). Additionally, it presents a novel recommendation solution for matching the most appropriate volunteer to the user’s geospatial needs based on expertise similarity, semantic similarity, and community feedback. The three categories of finding map services, discovering geoprocessing services, and volunteer expert recommendations were shown to be the most significant geoportal bot intents. Depending on the requirement, each intent additionally includes various entities such as time, place, description, skill, etc. The notion of deep transfer learning (DTL) was then put into practice by customizing a pre-trained BERT (Bidirectional Encoder Representations from Transformers) model for our particular aim and creating a task-oriented conversational agent. According to the results, effective intent classification and entity recognition in the geospatial domain could arise from this approach. We performed the training process with 200 sample data, 20% of which were utilized in a stratified manner for testing, and we obtained f1-scores of at least 0.75. Finally, a pilot Geoportal Chabot that combines crowdsourcing and conversational agents’ approaches was put into use and tested with success. In keeping with SDI technical purposes, the system might direct users to common geospatial web services, namely WMS and WPS, in addition to including natural language understanding (NLU) and natural language generation (NLG) capabilities. Result of user-centered evaluation indicated that the integration of a chatbot significantly reduces the average time required to access geospatial data and processing services by more than 50%. Notably, this effect is even more pronounced when locating an expert, with a fivefold decrease in the time required. Finally, overall user satisfaction rose from 86% to 94%.

A Vision on User-Centered Implementation and Evaluation of Explainable AI for Predicting Hospital-Onset Bacteremia.

In recent years, artificial intelligence (AI) has gained momentum in many fields of daily live. In healthcare, AI can be used for diagnosing or predicting illnesses. However, explainable AI (XAI) is needed to ensure that users understand how the algorithm arrives at a decision. In our research project, machine learning methods are used for individual risk prediction of hospital-onset bacteremia (HOB). This paper presents a vision on a step-wise process for implementation and evaluation of user-centered XAI for risk prediction of HOB. An initial requirement analysis revealed first insights on the users' needs of explainability to use and trust such risk prediction applications. The findings were then used to propose step-wise process towards a user-centered evaluation.

User-Centered Evaluation of the Wearable Walker Lower Limb Exoskeleton; Preliminary Assessment Based on the Experience Protocol.

Using lower limb exoskeletons provides potential advantages in terms of productivity and safety associated with reduced stress. However, complex issues in human-robot interactions are still open, such as the physiological effects of exoskeletons and the impact on the user's subjective experience. In this work, an innovative exoskeleton, the Wearable Walker, is assessed using the EXPERIENCE benchmarking protocol from the EUROBENCH project. The Wearable Walker is a lower-limb exoskeleton that enhances human abilities, such as carrying loads. The device uses a unique control approach called Blend Control that provides smooth assistance torques. It operates two models simultaneously, one in the case in which the left foot is grounded and another for the grounded right foot. These models generate assistive torques combined to provide continuous and smooth overall assistance, preventing any abrupt changes in torque due to model switching. The EXPERIENCE protocol consists of walking on flat ground while gathering physiological signals, such as heart rate, its variability, respiration rate, and galvanic skin response, and completing a questionnaire. The test was performed with five healthy subjects. The scope of the present study is twofold: to evaluate the specific exoskeleton and its current control system to gain insight into possible improvements and to present a case study for a formal and replicable benchmarking of wearable robots.

Evaluating the real-world usability of BCI control systems with augmented reality: a user study protocol.

Brain-computer interfaces (BCI) enable users to control devices through their brain activity. Motor imagery (MI), the neural activity resulting from an individual imagining performing a movement, is a common control paradigm. This study introduces a user-centric evaluation protocol for assessing the performance and user experience of an MI-based BCI control system utilizing augmented reality. Augmented reality is employed to enhance user interaction by displaying environment-aware actions, and guiding users on the necessary imagined movements for specific device commands. One of the major gaps in existing research is the lack of comprehensive evaluation methodologies, particularly in real-world conditions. To address this gap, our protocol combines quantitative and qualitative assessments across three phases. In the initial phase, the BCI prototype's technical robustness is validated. Subsequently, the second phase involves a performance assessment of the control system. The third phase introduces a comparative analysis between the prototype and an alternative approach, incorporating detailed user experience evaluations through questionnaires and comparisons with non-BCI control methods. Participants engage in various tasks, such as object sorting, picking and placing, and playing a board game using the BCI control system. The evaluation procedure is designed for versatility, intending applicability beyond the specific use case presented. Its adaptability enables easy customization to meet the specific user requirements of the investigated BCI control application. This user-centric evaluation protocol offers a comprehensive framework for iterative improvements to the BCI prototype, ensuring technical validation, performance assessment, and user experience evaluation in a systematic and user-focused manner.

Exploring User-Centered Evaluation in a Tertiary-Level, Audio-Describer Training Program

Introduction: Audio description (AD) should satisfy the needs of people who are visually impaired; however, quality assessment in university training programs has rarely been based on feedback provided by AD users. Aiming to help students truly understand the needs of people who are visually impaired and their criteria for AD, this study adopted user-centered evaluation in an AD training program offered by a university in Hong Kong. Method: Three evaluators with visual impairment, representing the views of AD users, were invited to evaluate 29 students’ film clips that included AD and discuss their evaluation criteria. Eleven students participated in focus-group interviews and were asked to comment on AD products and discuss their own evaluation criteria. The evaluation criteria of the two groups were compared qualitatively and quantitatively with the aid of the software NVivo. Results: The results showed discrepancies between the users’ and the trainees’ criteria as well as differences in their focuses when evaluating the AD products. The users’ criteria were more numerous; however, certain criteria were perceived exclusively by the trainees. Discussion: Without sufficient knowledge of users’ needs, AD trainees are inclined to provide redundant information (e.g., easily identifiable film sounds and speaker information) while neglecting some aspects of AD (e.g., the prudent use of pronouns and literary expressions), which are deemed important by the users. Implications for Practitioners: The results prompted AD instructors to integrate user feedback into AD training to improve students’ understanding of how to produce AD that better meets its users’ needs.

Bridging the Gap Between Universities and Alumni: A User-Centered Evaluation of a Digital Alumni Engagement Platform

Universities face an ongoing challenge in fostering meaningful connections with their alumni communities. This paper presents the development and evaluation of the Alumni Tracer and Engagement Hub, a web-based platform designed to bridge the gap between universities and their alumni. The study employed a comprehensive approach, utilizing McCall's Software Quality Model for technical evaluation during the development and testing phases. Following deployment, a user-centric evaluation assessed the platform's effectiveness in achieving its objectives. This evaluation leveraged the ISO/IEC 25010 model and gathered feedback through surveys and interviews with alumni, administrators, and staff. The evaluation results revealed a strong technical foundation, with high marks for security, self-documentation, code clarity, and user-friendliness (McCall's model). User feedback (ISO/IEC 25010 model) indicated positive experiences with core functionalities like functional completeness, performance, and learnability. However, areas for improvement were identified in the user interface aesthetics, clarity of element purpose, and user perceptions of data security. This paper discusses these findings and proposes recommendations for optimizing the platform's user experience, strengthening data security measures, and ensuring long-term technical maintainability. By implementing these recommendations, the Alumni Tracer and Engagement Hub has the potential to become a valuable tool for universities to strengthen alumni connections and build a thriving alumni network.

Learning by Doing in VR: A User-Centric Evaluation of Lathe Operation Training

This study presents the development and evaluation of an immersive virtual reality (VR) application designed for lathe operation training. The VR application, built using Unity for Oculus Rift headsets, aims to simulate a realistic lathe machining experience, allowing users to interact with the machine’s various controls and levers. The experimental analysis involved 20 s-year Mechanical Engineering students who performed machining tasks in the virtual environment. The usability and user experience of the application were assessed using the System Usability Scale (SUS) and a 12-item questionnaire. The SUS results yielded a high mean score of 96.25 (SD = 6.41), indicating excellent usability. The user experience evaluation also showed positive feedback, with high ratings for the sense of presence, realism, and usefulness for training purposes. However, some users reported minor physical discomforts such as dizziness. The study concludes that immersive VR is a valuable tool for enhancing training in lathe operations, offering an engaging and realistic experience that encourages active learning. Future work should focus on reducing physical discomfort and further improving the application’s realism and interactivity.

Usability assessment of a greenhouse context-aware alert system for small-scale farmers

In the dynamic landscape of modern agriculture, integrating technology holds immense potential to enhance efficiency and productivity for small-scale farmers. This study presents a user-centric evaluation of an intelligent context-aware alert system, tailored for small-scale greenhouse farming. We employed standardized questionnaires, including the NASA Task Load Index and the User Experience Questionnaire, to assess the system's perceived utility, mental workload, and overall user experience. Our findings reveal the high perceived utility of the system among farmers. Farmers participating in the assessment indicated a strong intention to utilize the system for crop monitoring. Moreover, the system demonstrated a moderate mental workload, suggesting ease of use and potential acceptance by users. Our evaluation highlighted an excellent user experience, with scores ranging from very good to extremely good across all dimensions. Furthermore, user preferences for alert mechanisms underscored the importance of adaptable notifications, with voice and text alerts favored for comprehensive information dissemination. Light and voice alerts were preferred during manual tasks. This study highlights the significance of user-centered design in agricultural technology, offering insights to enhance the usability and the adoption of alert systems in small-scale farming environments. The positive reception of the system's utility and the moderate mental workload suggest that such technology can be readily adopted by farmers, thereby improving monitoring and management practices in greenhouse farming. The preference for adaptable alert mechanisms further emphasizes the need for flexible and context-sensitive solutions in agricultural technology.

Report on the 8th Workshop on Search-Oriented Conversational Artificial Intelligence (SCAI 2024) at CHIIR 2024

Conversational Agents are increasingly integrated into our daily routines, assisting us with various tasks, from simple commands such as scheduling events to more complex conversational search interactions. Such conversational search systems are traditionally evaluated with word-overlap metrics such as F1 score and accuracy. The full-day workshop on Search-Oriented Conversational Artificial Intelligence (SCAI) at CHIIR 2024 explored the evaluation of conversational search systems from the user's perspective. This interactive workshop included multiple panel discussions and working groups focused on developing and discussing innovative, user-centered evaluation methods for these systems. This paper, co-authored by both organizers and participants of the workshop, presents a summary of the insights gathered from the panel discussions and working groups. Date : 14 March 2024. Website : https://scai.info/scai-2024/.

Developing feedback visualizations to support older adults’ medication adherence

BackgroundBlood pressure control is critical for older adults because the prevalence of hypertension and resulting cardiovascular illness increases with age. Antihypertension medications are central to blood pressure treatment. However, nonadherence to antihypertension medications is high. Health technology such as smartphone apps provide an opportunity for users to manage their medication regimen and support processes related to medication-taking. ObjectiveWe implemented a user-centered evaluation approach to develop and refine adherence feedback visualizations for the MEDSReM© medication adherence app for older adults with hypertension. MethodsWe conducted a literature review and iterative usability testing to achieve this objective. We identified adherence goals, information needs, as well as design guidelines by reviewing theoretical frameworks and existing scientific evidence. We then used a two-phase iterative user-centered study and subject matter expert evaluation. Both quantitative and qualitative data were used to select and improve the current prototype and evolve to the next prototype. ResultsThe need for daily, weekly, and monthly adherence performance information as well as visualization formats for conveying this information was identified from the literature review. Overall, the information shown in visualization prototypes was successfully interpreted by participants. Comprehension issues of visualizations were identified and addressed from visual prototype revisions. Insights from both user and subject matter expert groups were used to select and refine the prototypes for the MEDSReM app. ConclusionEvidence-based and user-centered approaches were effective for developing visualizations about adherence performance feedback in the MEDSReM app and provided insight into how the app can be made easy to understand and use by older adults with hypertension, which will be evaluated in future effectiveness testing.

User-Centered Evaluation Framework to Support the Interaction Design for Augmented Reality Applications

The advancement of Augmented Reality (AR) technology has been remarkable, enabling the augmentation of user perception with timely information. This progress holds great promise in the field of interaction design. However, the mere advancement of technology is not enough to ensure widespread adoption. The user dimension has been somewhat overlooked in AR research due to a lack of attention to user motivations, needs, usability, and perceived value. The critical aspects of AR technology tend to be overshadowed by the technology itself. To ensure appropriate future assessments, it is necessary to thoroughly examine and categorize all the methods used for AR technology validation. By identifying and classifying these evaluation methods, researchers and practitioners will be better equipped to develop and validate new AR techniques and applications. Therefore, comprehensive and systematic evaluations are critical to the advancement and sustainability of AR technology. This paper presents a theoretical framework derived from a cluster analysis of the most efficient evaluation methods for AR extracted from 399 papers. Evaluation methods were clustered according to the application domains and the human–computer interaction aspects to be investigated. This framework should facilitate rapid development cycles prioritizing user requirements, ultimately leading to groundbreaking interaction methods accessible to a broader audience beyond research and development centers.

CRS-Que : A User-centric Evaluation Framework for Conversational Recommender Systems

An increasing number of recommendation systems try to enhance the overall user experience by incorporating conversational interaction. However, evaluating conversational recommender systems (CRSs) from the user’s perspective remains elusive. The GUI-based system evaluation criteria may be inadequate for their conversational counterparts. This article presents our proposed unifying framework, CRS-Que , to evaluate the user experience of CRSs. This new evaluation framework is developed based on ResQue , a popular user-centric evaluation framework for recommender systems. Additionally, it includes user experience metrics of conversation (e.g., understanding, response quality, humanness) under two dimensions of ResQue (i.e., Perceived Qualities and User Beliefs). Following the psychometric modeling method, we validate our framework by evaluating two conversational recommender systems in different scenarios: music exploration and mobile phone purchase . The results of the two studies support the validity and reliability of the constructs in our framework and reveal how conversation constructs and recommendation constructs interact and influence the overall user experience of the CRS. We believe this framework could help researchers conduct standardized user-centric research for conversational recommender systems and provide practitioners with insights into designing and evaluating a CRS from users’ perspectives.

User-centric evaluation of EHR software through NLP-driven investigation: Implications for product development and user experience

In today's rapidly digitising world, organisations often face challenges in achieving successful technology implementation, with low returns on investment and a significant gap between technology utilisation and acceptance. Electronic Health Record (EHR) software provides a comprehensive platform for storing, managing, and gaining access to patient data, enabling healthcare providers to provide high-quality care and make informed decisions. Conducting a comprehensive analysis of EHR software decision-level user reviews can contribute to improving and optimising EHR systems. Incorporating principles of user innovation, the study aims to investigate the significance of online review analysis in product development and reduce the technology acceptance gap. The study employs document-topic theme mining techniques, specifically BERTopic, a topic modelling technique, VADER sentiment analysis, and NRCLex Affect dictionary-based emotion analysis to extract meaningful aspect-based insights from the vast unstructured data of online reviews about EHR software. The study helps healthcare organisations get insights from user perspectives and highlights the importance of user review evaluation in purchasing or switching EHR software. It contributes to the existing literature on software evaluation, software requirement analysis, marketing and positioning strategy analysis, and online user review analysis. In addition to the challenges faced in the healthcare sector, the study recognises that technology transition issues are pervasive across various domains. Common challenges include the need for seamless integration with existing workflows, addressing resistance to change, navigating data security and privacy concerns, and fostering a technology-friendly culture. By exploring these challenges within the context of EHR software, the research contributes valuable insights with broader implications for diverse industries undergoing digital transformation.

Evaluating the clinical utility of artificial intelligence assistance and its explanation on the glioma grading task

Clinical evaluation evidence and model explainability are key gatekeepers to ensure the safe, accountable, and effective use of artificial intelligence (AI) in clinical settings. We conducted a clinical user-centered evaluation with 35 neurosurgeons to assess the utility of AI assistance and its explanation on the glioma grading task. Each participant read 25 brain MRI scans of patients with gliomas, and gave their judgment on the glioma grading without and with the assistance of AI prediction and explanation. The AI model was trained on the BraTS dataset with 88.0% accuracy. The AI explanation was generated using the explainable AI algorithm of SmoothGrad, which was selected from 16 algorithms based on the criterion of being truthful to the AI decision process. Results showed that compared to the average accuracy of 82.5±8.7% when physicians performed the task alone, physicians’ task performance increased to 87.7±7.3% with statistical significance (p-value = 0.002) when assisted by AI prediction, and remained at almost the same level of 88.5±7.0% (p-value = 0.35) with the additional assistance of AI explanation. Based on quantitative and qualitative results, the observed improvement in physicians’ task performance assisted by AI prediction was mainly because physicians’ decision patterns converged to be similar to AI, as physicians only switched their decisions when disagreeing with AI. The insignificant change in physicians’ performance with the additional assistance of AI explanation was because the AI explanations did not provide explicit reasons, contexts, or descriptions of clinical features to help doctors discern potentially incorrect AI predictions. The evaluation showed the clinical utility of AI to assist physicians on the glioma grading task, and identified the limitations and clinical usage gaps of existing explainable AI techniques for future improvement.