Experience Feedback Research Articles

A Multiparty Collaboration to Engage Diverse Populations in Community-Centered Artificial Intelligence Research

Artificial Intelligence (AI)-enabled technology has the potential to expand access to high-quality health information and healthcare services. Learning how diverse users interact with technology enables improvements to the AI model and the user interface, maximizing its potential benefit for a greater number of people. This narrative describes how technology developers, academic researchers, and representatives from a community-based organization collaborated to conduct a community-centered project on emerging health technologies. Our project team comprised representatives from Stanford Medicine, Google, and Santa Clara Family Health Plan’s Blanca Alvarado Community Resource Center. We aimed to understand the usability and acceptability of an AI-driven dermatology tool among East San Jose, CA, community members. Specifically, our objectives were to: test a model for cross-sector research of AI-based health technology; determine the utility of the tool in an ethnically and age-diverse population; obtain in-depth user experience feedback from participants recruited during community events; offer free skin health consultations; and provide resources for receiving follow-up care. We describe a collaborative approach in which each party contributed expertise: knowledge of the community from the community health partner, clinical expertise from the academic research institution, and software and AI expertise from the technology company. Through an iterative process, we identified important community needs, including technological, language, and privacy support. Our approach allowed us to recruit and engage a diverse cohort of participants, over 70% of whom preferred a language other than English. We distill learnings from planning and executing this case study that may help other collaborators bridge the gap between academia, industry, and community in AI healthcare innovation.

Virtual Reality for the Visualised-Guided Tours of the Notre Dame Museum in Paris

This study aims to explore the application of virtual reality technology for visualization in museums, using Notre Dame de Paris, France, as an example. It is more rich in scenes, more interactive graphics, and other visualizations, which enables viewers to enjoy the reconstructed Notre Dame online with the best audio-visual effects. We used the Likert scale research method for a survey. The questionnaire was administered to 15 students and 15 experts for basic information collection and experience feedback. Using data statistics and qualitative analysis, the effectiveness of the guided tours was validated. The result showed that VR museum tours were supported with great potential. Its more interactive guided tour experience enhanced the audience's experience and improved their understanding and interest in history and culture. Virtual reality technology was successfully applied to guide NDDP by providing a richer and more interactive guided experience and promote the cultural heritage and development of the cathedral. The research result provides a reference and inspiration for the future development of guided tours of museums.

The SHOW RESPECT adaptable framework of considerations for planning how to share trial results with participants, based on qualitative findings from trial participants and site staff

BackgroundSharing trial results with participants is a moral imperative, but too often does not happen in appropriate ways.MethodsWe carried out semi-structured interviews with patients (n = 13) and site staff (n = 11), and surveyed 180 patients and 68 site staff who were part of the Show RESPECT study, which tested approaches to sharing results with participants in the context of the ICON8 ovarian cancer trial (ISRCTN10356387). Qualitative and free-text data were analysed thematically, and findings used to develop the SHOW RESPECT adaptable framework of considerations for planning how to share trial results with participants. This paper presents the framework, with illustrations drawn from the Show RESPECT study.ResultsOur adaptable ‘SHOW RESPECT’ framework covers (1) Supporting and preparing trial participants to receive results, (2) HOw will the results reach participants?, (3) Who are the trial participants?, (4) REsults—what do they show?, (5) Special considerations, (6) Provider—who will share results with participants?, (7) Expertise and resources, (8) Communication tools and (9) Timing of sharing results. While the data upon which the framework is based come from a single trial, many of our findings are corroborated by findings from other studies in this area, supporting the transferability of our framework to trials beyond the UK ovarian cancer setting in which our work took place.ConclusionsThis adaptable ‘SHOW RESPECT’ framework can guide researchers as they plan how to share aggregate trial results with participants. While our data are drawn from a single trial context, the findings from Show RESPECT illustrate how approaches to communication in a specific trial can influence patient and staff experiences of feedback of trial results. The framework generated from these findings can be adapted to fit different trial contexts and used by other researchers to plan the sharing of results with their own participants.Trial registrationISRCTN96189403. Registered on February 26, 2019. Show RESPECT was supported by the Medical Research Council (MC_UU_12023/24 and MC_UU_00004/08) and the NIHR CRN.

Scaffolding Feedback Literacy: Designing a Feedback Analytics Tool with Students

Feedback is essential in learning. The emerging concept of feedback literacy underscores the skills students require for effective use of feedback. This highlights students’ responsibilities in the feedback process. Yet, there is currently a lack of mechanisms to understand how students make sense of feedback and whether they act on it. This gap makes it hard to effectively support students in feedback literacy development and improve the quality of feedback. As a specific application of learning analytics, feedback analytics (analytics on learner engagement with feedback) can offer insights into students’ learning engagement and progression, which can in turn be used to scaffold student feedback literacy. This study proposes a feedback analytics tool, designed with students, aimed at aiding students to synthesize feedback received from multiple sources, scaffold the sense-making process, and prompt deeper reflections or actions on feedback based on data about students’ interactions with feedback. We held focus group discussions with 38 students to learn about their feedback experiences and identified tool features. Based on identified user requirements, a prototype was developed and validated with 16 students via individual interviews. Based on the findings, we envision a feedback analytics tool with the aim of scaffolding student feedback literacy.

Measuring the relationship between museum attributes and visitors: An application of topic model on museum online reviews.

In recent years, cultural tourism has increasingly embraced museum visits. Museums serve as both cultural heritage guardians and integral parts of tourist destinations, significantly impacting visitor satisfaction and experience. Moreover, online museum reviews have become a crucial indicator of museum service quality, visitor experience, and public feedback in the digital age. An analysis of online reviews on major tourism websites and social media platforms can assist museums in developing appropriate management strategies. This study employed the structural topic model (STM) to analyze online museum reviews, identifying three primary attributes of museums and visitors' personal experiences, as well as 19 sub-attributes. The study confirmed that core offerings have a positive impact on visitor experience and satisfaction, while peripheral services and overall ambiance are also positively related to visitor experience and satisfaction. Furthermore, the results of structural equation modeling demonstrated that visitors' personal experiences have a positive impact on satisfaction. The results of structural equation modeling analysis support all seven hypothesized relationships. These findings will assist museum managers in developing effective management strategies and future plans.

How is feedback perceived by Brazilian students and faculty from a nursing school?

AimThis study aims to explore the perceptions of feedback among undergraduate students and faculty members at a Brazilian private nursing school. BackgroundFeedback plays a crucial role in the socioemotional development of learners, with its interpretation varying across different sociocultural contexts. Student evaluations frequently express dissatisfaction regarding both the quality and quantity of feedback received. Conversely, delivering feedback poses a challenge for faculty, requiring the establishment of an empathetic connection that fosters trust and credibility. Brazil, being a developing country characterized by social disparities and economic challenges, presents a unique backdrop for examining feedback dynamics. DesignQualitative research, employing Inductive Content Analysis, was used to understand feedback perceptions in Brazilian nursing education. Symbolic interactionism was adopted as methodological framework and guided data interpretation. MethodsWe carried out five virtual focus groups composed of a group of teachers (n=5) and four of students (n=34). Semi-structured interviews guided data collection. The recorded sessions were subsequently analyzed to identify key themes and codes. Symbolic interactionism was employed as a framework to derive meaning from qualitative data. ResultsContent analysis generated two categories that reveal the perception of teachers and students in the feedback process. The first, called “Feedback in Education: Sociocultural Influences for Students and Teachers”, expresses the beliefs and interpretations of students and teachers within the shared feedback environment. The second called “Challenging resonance, transformative construction: Navigating the dualities of feedback for teachers and students”, which elucidated how relational dynamics shape behaviors and attitudes, promoting the development of social skills and learning. Faculty's previous feedback experiences significantly influence their self-perception and behavior with students. As a result of the resonance of these past interactions, we recognize that the teaching self also plays a crucial role in the quality and perception of feedback. Furthermore, students construct social reality with similar beliefs and values, they believe in the learning potential generated by feedback. Our findings also corroborate that perceptions of feedback are deeply influenced by the sociocultural context and the narratives corroborate previous findings indicating that, in Brazil, honest feedback can be implicitly perceived as criticism rather than an opportunity for growth. ConclusionsFaculty members often draw on their past experiences when providing feedback, highlighting the adaptive nature of feedback interactions. Additionally, the feedback process is consistently influenced by the commitment to maintaining positive relationships with students. Students recognize the constructive dimension of feedback as a valuable tool for learning and personal growth.

Wear + Touch: An Exploration of Wearables for Vibrotactile Feedback During Touchscreen Input

Vibrotactile feedback is known to have a positive impact on the user performance and experience of touchscreen input. Unfortunately, not all touchscreen devices have built-in haptics capabilities, public displays being a notable example, while the design options for vibrotactile feedback on conventional mobile devices, such as smartphones and tablets, are still limited. In this work, we propose leveraging commodity wearable devices, such as smartwatches, rings, and armbands, to enable vibrotactile feedback as closely as possible to the touch point when the touchscreen does not natively support haptics, and to complement existing haptics capabilities when the touchscreen already features them. To this end, we introduce Wear + Touch, a set of four interaction techniques designed to augment touchscreen input with vibrations delivered through wearable devices, from substituting to complementing, extending, and expanding on-screen vibrotactile feedback. We demonstrate Wear + Touch with several practical implementations involving a large touchscreen display, smartphone, tablet, smartwatch, smart ring, and armband as well as a custom wearable device that we designed for vibrations delivered simultaneously on the user’s index finger, wrist, and forearm. We also report empirical results from two controlled experiments involving a total of 40 participants, conducted to understand the multi-faceted user experience of vibrotactile feedback delivered through the Wear + Touch techniques. In the first experiment, we reveal a favorable user experience of vibrations delivered on the index finger, wrist, and forearm, in this order, when the touchscreen lacks haptics capabilities. In the second experiment, we reveal benefits of reinforcing and expanding on-screen vibrations with vibrotactile feedback on the index finger, wrist, and arm, respectively. We use our findings to suggest future work opportunities for combined vibrotactile feedback involving both touchscreens and wearable devices.

26. Co-innovation to conceptualize the poultry farms of tomorrow: Experience feedback from a participatory research

26. Co-innovation to conceptualize the poultry farms of tomorrow: Experience feedback from a participatory research

From Reporting to Improving: How Root Cause Analysis in Teams Shape Patient Safety Culture.

Given the increasing focus on patient safety in healthcare systems worldwide, understanding the impact of Continuous Quality Improvement Programs (QIPs) is crucial. QIPs, including Morbidity and Mortality Conferences (MMCs) and Experience Feedback Committees (EFCs), have been identified as effective strategies for enhancing patient safety culture. These programs engage healthcare professionals in the identification and analysis of adverse events to foster a culture of safety (ie the product of individual and group value, attitudes, and perceptions about quality and safety). This study aimed to determine whether patient safety culture differed regarding care provider participation in MMCs and EFCs activities. A cross-sectional web-only survey was conducted in 2022 using the Hospital Survey on Patient Safety Culture (HSOPS) among 4780 employees at an 1836-bed, university-affiliated hospital in France. We quantified the mean differences in the 12 HSOPS dimension scores according to MMCs and EFCs participation, using Cohen d effect size. We performed a multivariate analysis of variance to examine differences in dimension scores after adjusting for background characteristics. Of 4780 eligible employees, 1457 (30.5%) participated in the study. Among the respondents, 571 (39.2%) participated in MMCs or EFCs activities. Participants engaged in MMCs or EFCs reported significantly higher scores in six out of twelve HSOPS dimensions, particularly in "Nonpunitive response to error", "Feedback and communication about error", and "Organizational learning" (Overall effect size = 0.14, 95% confidence interval = 0.11 to 0.17, P<0.001). Notably, involvement in both MMCs and EFCs was associated with higher improvements in patient safety culture compared to non-participation or singular involvement in either program. However, certain dimensions such as "Staffing", "Hospital management support", and "Hospital handoffs and transition" showed no significant association with MMCs or EFCs participation, highlighting broader systemic challenges. The study confirms the positive association between participation in MMCs or EFCs and an enhanced culture of patient safety, emphasizing the importance of such programs in fostering an environment conducive to learning, communication, and nonpunitive responses to errors. While MMCs or EFCs are effective in promoting certain aspects of patient safety culture, addressing broader systemic challenges remains crucial for comprehensive improvements in patient safety.

The commitment of French midwifery teachers to doctoral studies: A qualitative study based on experience feedback

The commitment of French midwifery teachers to doctoral studies: A qualitative study based on experience feedback

Near real-time patient experience feedback with data relay to providers: a systematic review of its effectiveness.

Near Real-Time Feedback (NRTF) on the patient's experience with care, coupled with data relay to providers, can inform quality-of-care improvements, including at the point of care. The objective is to systematically review contemporary literature on the impact of the use of NRTF and data relay to providers on standardized patient experience measures. Six scientific databases and five specialty journals were searched supplemented by snowballing search strategies, according to the registered study protocol. Eligibility included studies in English (2015-2023) assessing the impact of NRTF and data relay on standardized patient-reported experience measures as a primary outcome. Eligibility and quality appraisals were performed by two independent reviewers. An expert former patient (Patient and Family Advisory Council and communication sciences background) helped interpret the results. Eight papers met review eligibility criteria, including three randomized controlled trials (RCTs) and one non-randomized study. Three of these studies involved in-person NRTF prior to data relay (patient-level data for immediate corrective action or aggregated and peer-compared) and led to significantly better results in all or some of the experience measures. In turn, a kiosk-based NRTF achieved no better experience results. The remaining studies were pre-post designs with mixed or neutral results and greater risks of bias. In-person NRTF on the patient experience followed by rapid data relay to their providers, either patient-level or provider-level as peer-compared, can improve the patient experience of care. Reviewed kiosk-based or self-reported approaches combined with data relay were not effective. Further research should determine which approach (e.g. who conducts the in-person NRTF) will provide better, more efficient improvements and under which circumstances.

Real time feedback and E-learning intelligent entertainment experience in computer English communication based on deep learning

Traditional E-learning usually lacks real-time feedback and interactivity, which can not meet the needs of students’ personalized learning. Computer English communication is an important aspect of cultivating students’ language communication ability, which has a positive impact on students’ learning results. In order to realize the real-time feedback of computer English communication and E-learning intelligent entertainment experience, this study stimulates students’ learning interest and improves their learning effect by providing real-time feedback and personalized learning content. With the development of computer network technology, E-learning English platforms can provide students with intelligent entertainment experiences and real-time learning feedback. This article analyzes the real-time feedback and E-learning intelligent entertainment experience of computer English communication based on deep learning. Firstly, an in-depth analysis was conducted on the characteristics of English communication. English communication has specific grammatical structures and professional terms, which require quick and accurate understanding and expression of information. On this basis, a training dataset was constructed, preprocessed and annotated, cleaned to remove irrelevant information and noise, and then corresponding labels were added to each data sample. Subsequently, the constructed dataset was used to optimize the deep learning and recurrent neural network models, and the model parameters were optimized using a backpropagation algorithm. The experimental results show that the method designed in this article has higher accuracy and real-time performance, can timely identify learners’ problems and provide personalized feedback, accurately evaluate learners’ communication ability, and provide targeted feedback.

Reinforcement Learning from Experience Feedback: Application to Economic Policy

Reinforcement Learning from Experience Feedback: Application to Economic Policy

Teaching note: personal attack or the personal touch? Evaluating the use of video feedback methods with qualifying social workers

ABSTRACT Social work skills such as the ability to reflect on self and to recognize and manage emotions are fostered in part through using and receiving feedback, however there is limited research about feedback methods that effectively enable social work students and post-qualifying workers to engage with the emotional aspects of learning and feedback. This teaching note reports on a pilot project in which video feedback on academic assignments was provided to first year undergraduate social work students studying at a university in England. The project findings underscore the importance of engaging with the emotional dimensions of feedback processes and recognizing how feedback experiences can shape emerging learner and professional identities. The findings also underline the psychological and social skills required to engage in more performative aspects of contemporary education and practice. We argue that social work students are likely to benefit from support to develop meta-cognition (the process of thinking about one’s thinking processes) and self-regulation (regulation of emotion and behavior) skills before they can make sense of academic and practice-orientated feedback. Video feedback on academic assignments is therefore best utilized when it is underpinned by a dialogical approach to teaching and learning in the ‘classroom’ and the ‘field’.

A Gamified Platform to Encourage Sustainable Behaviours

Abstract. The production, use, and disposal of goods and services require the use of natural resources, resulting in waste and environmental pollution. A growing number of people are embracing more sustainable lifestyles in alignment with the 3Rs (Reducing, Reusing, and Recycling) to counter these effects, but many remain uninformed of their options, or lack the motivation to adopt these practices. The work described in this paper addresses these issues through the development of the Sustainable App, a mobile application incorporating techniques of persuasion and gamification which offers its users eco-friendly alternatives to several types of daily activities, made available via an interactive map. The app employs diverse strategies to improve engagement, including a pet-nurturing game-like feature that responds to user actions related to sustainable transport choices, sustainable retail, and recycling. Additionally, the complete platform associated with the app also delivers Just-In-Time (JIT) messages concerning nearby sustainable choices based on location and time. Upon evaluation of the platform, positive user experience and overall good feedback were noted. Experts also weighed in on the application’s evaluation, offering valuable insights to refine future iterations.

Bringing a Lens of Equity to Geoscience Qualifying Examinations

AbstractQualifying examinations are an important milestone in geoscience graduate programs, but students with marginalized identities are disproportionately lost from graduate programs around the time of these exams. Inequity in qualifying exams can enter at multiple stages throughout the exam design, student mentorship experience, exam administration, and post exam feedback. Therefore, robust assessment is necessary when building an equitable examination. We provide concrete suggestions for graduate programs to evaluate and modify their qualifying examinations. The data‐driven and iterative process encourages graduate programs to outline specific expectations for success, employ best‐practice pedagogy, proactively support students, and use data to measure progress and inform changes in the examination.

Pneumonia diagnosis performance in the emergency department: a mixed-methods study about clinicians' experiences and exploration of individual differences and response to diagnostic performance feedback.

We sought to (1) characterize the process of diagnosing pneumonia in an emergency department (ED) and (2) examine clinician reactions to a clinician-facing diagnostic discordance feedback tool. We designed a diagnostic feedback tool, using electronic health record data from ED clinicians' patients to establish concordance or discordance between ED diagnosis, radiology reports, and hospital discharge diagnosis for pneumonia. We conducted semistructured interviews with 11 ED clinicians about pneumonia diagnosis and reactions to the feedback tool. We administered surveys measuring individual differences in mindset beliefs, comfort with feedback, and feedback tool usability. We qualitatively analyzed interview transcripts and descriptively analyzed survey data. Thematic results revealed: (1) the diagnostic process for pneumonia in the ED is characterized by diagnostic uncertainty and may be secondary to goals to treat and dispose the patient; (2) clinician diagnostic self-evaluation is a fragmented, inconsistent process of case review and follow-up that a feedback tool could fill; (3) the feedback tool was described favorably, with task and normative feedback harnessing clinician values of high-quality patient care and personal excellence; and (4) strong reactions to diagnostic feedback varied from implicit trust to profound skepticism about the validity of the concordance metric. Survey results suggested a relationship between clinicians' individual differences in learning and failure beliefs, feedback experience, and usability ratings. Clinicians value feedback on pneumonia diagnoses. Our results highlight the importance of feedback about diagnostic performance and suggest directions for considering individual differences in feedback tool design and implementation.

P.016 Re-norming medical education: centering patient experience and diverse bodies in Lumbar Puncture (LP) instruction

Background: Medical curricula are often created with limited patient and student input and underrepresent certain body types. Traditional medical education often prepares learners to perform procedures, such as lumbar punctures (LPs), on a young white able-bodied 70kg male. When approaching diverse patients, this educational gap can lead to medical learners’ lack of confidence, skill, and knowledge, resulting in poor patient experiences. Methods: This co-design project involves patient and student input. We interviewed five patients who underwent LPs and explored their experience through a trauma-informed approach. To visualize landmarking across body types, we recruited nine volunteers of diverse body sizes, ages, tattoos, and skin colour (Fitzpatrick Scale). Incorporating patient narratives, as well as videos and photographs showing landmarking on diverse bodies, we crafted an online LP instructional module. Focus groups of 6-10 students will be held to collect student perception of the effectiveness of the module. Results: Our learning module and related media will be built into Western University’s Undergraduate Medical Education curriculum, available under a Creative Commons license through the Western Health Education Media Library. Conclusions: Integrating patient experience and student feedback, we are developing a comprehensive educational tool to better equip medical learners to deliver patient-centered LPs across diverse body types.

Medical students', residents', and nurses's feedback to clinical educators in Taiwan: A qualitative study.

Feedback is a crucial element in learning. While studies in the field of healthcare professions education have highlighted the process of educators feeding back to learners, relatively little investigation exists on learners feeding back to educators in Asian cultures. Studies show that recipients of effective feedback develop educational skills and reflective practice, but the process of giving feedback seems to have been mainly studied through surveys and questionnaires. Such research offers little to no insights on feedback providers' and recipients' experiences of feedback. To fill the gap, in the context of multi-source feedback, we investigate medical students, residents, and nurses feedback giving to clinical educators (and their receiving of this) following a case presentation training course. We aim to understand the facilitators and inhibitors that encourage and/or prevent feedback provision alongside educators' uptake and reactions. We used semi-structured group interviews. Participants comprised five different categories of participants: year-4 medical students (n = 6); residents (n = 5); nurses (n = 4); junior clinical educators (n = 9); senior clinical educators (n = 3). We asked them about their experiences of providing feedback to educators and educators receiving of feedback on their teaching. Group interviews were conducted in the largest healthcare institution in Taiwan. Data were analysed using thematic Framework Analysis and managed in ATLAS.ti 8.0. We identified two major themes with respective sub-themes: (1) Factors affecting feedback giving (including desire for improvement, feedback content, process of feedback, feedback fears, feedback prevention and medical hierarchy); and (2) Educators' reactions to receiving feedback (including validity of feedback, face-saving and emotional reactions to receiving feedback). Feedback provision to educators on their teaching, and educators' receiving of this feedback in an Asian culture brings forth issues around medical hierarchy, in-person feedback and face-saving, which have important implications for effective and optimal delivery of feedback. Curricular developers should consider the context of feedback (e.g. anonymously online), facilitating students as active participants for the development of educational quality, and educators' mindful practice when engaging with student feedback.

Customer Feedback Analysis Using Text Mining

Complexity surrounding the holistic nature of customer experience has made measuring customer perceptions of interactive service experiences challenging. At the same time, advances in technology and changes in methods for collecting explicit customer feedback are generating increasing volumes of unstructured textual data, making it difficult for managers to analyze and interpret this information. Consequently, text mining, a method enabling automatic extraction of information from textual data, is gaining in popularity. However, this method has performed below expectations in terms of depth of analysis of customer experience feedback and accuracy. In this study, we advance linguistics-based text mining modeling to inform the process of developing an improved framework. The proposed framework incorporates important elements of customer experience, service methodologies and theories such as co-creation processes, interactions and context. This more holistic approach for analyzing feedback facilitates a deeper analysis of customer feedback experiences, by encompassing three value creation elements: activities, resources, and context (ARC). Empirical results show that the ARC framework facilitates the development of a text mining model for analysis of customer textual feedback that enables companies to assess the impact of interactive service processes on customer experiences. The proposed text mining model shows high accuracy levels and provides flexibility through training. As such, it can evolve to account for changing contexts over time and be deployed across different (service) business domains; we term it an “open learning” model. The ability to timely assess customer experience feedback represents a pre-requisite for successful co-creation processes in a service environment.