Multimodal Feedback Research Articles

The Role of Augmented Feedback on Motor Learning: A Systematic Review.

In motor learning, augmented feedback (AF) is the information provided by sources outside the body and encompasses visual feedback, auditory feedback, and multimodal augmented feedback. This review aims to examine the most recent evidence on these different modality types in healthy and diseased populations and athletes.The reporting of this review was guided by the standards of the “Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA)” statement with the aim to examine the most recent evidence on these feedback types in healthy and diseased populations and athletes. The literature search for this review has been limited to electronic journals with the search engines ISI Web of Knowledge, OvidSP EMBASE, and PubMed databases.This review considers visual feedback as the cornerstone of all augmented feedback types by citing its superiority in learning complex skills by medical students and balance maintenance by older adults. The review also deciphers the role of auditory augmented feedback in encumbering already burdened visual connections in the rehabilitation of patients with Parkinson’s disease (PD) and prosthetic limbs and further argues why the multimodal feedback model seems to be the most efficient way to offer feedback in learning motor tasks by alluding to multifaceted “skill training communication model” in the prevention of sports injuries such as anterior cruciate ligament tears.Multimodal augmented feedback seems to be the most effective and appropriate way to give feedback during motor learning in healthy and diseased populations and athletes as its stimuli are perceived faster and tend to be retained longer compared with the unimodal stimuli of auditory and visual feedback mechanisms.

Keep Me in the Loop: Real-Time Feedback with Multimodal Data

This paper describes the CPR Tutor, a real-time multimodal feedback system for cardiopulmonary resuscitation (CPR) training. The CPR Tutor detects training mistakes using recurrent neural networks. The CPR Tutor automatically recognises and assesses the quality of the chest compressions according to five CPR performance indicators. It detects training mistakes in real-time by analysing a multimodal data stream consisting of kinematic and electromyographic data. Based on this assessment, the CPR Tutor provides audio feedback to correct the most critical mistakes and improve the CPR performance. The mistake detection models of the CPR Tutor were trained using a dataset from 10 experts. Hence, we tested the validity of the CPR Tutor and the impact of its feedback functionality in a user study involving additional 10 participants. The CPR Tutor pushes forward the current state of the art of real-time multimodal tutors by providing: (1) an architecture design, (2) a methodological approach for delivering real-time feedback using multimodal data and (3) a field study on real-time feedback for CPR training. This paper details the results of a field study by quantitatively measuring the impact of the CPR Tutor feedback on the performance indicators and qualitatively analysing the participants’ questionnaire answers.

Motor Imagery-Based Brain-Computer Interface Combined with Multimodal Feedback to Promote Upper Limb Motor Function after Stroke: A Preliminary Study.

Background Recently, the brain-computer interface (BCI) has seen rapid development, which may promote the recovery of motor function in chronic stroke patients. Methods Twelve stroke patients with severe upper limb and hand motor impairment were enrolled and randomly assigned into two groups: motor imagery (MI)-based BCI training with multimodal feedback (BCI group, n = 7) and classical motor imagery training (control group, n = 5). Motor function and electrophysiology were evaluated before and after the intervention. The Fugl-Meyer assessment-upper extremity (FMA-UE) is the primary outcome measure. Secondary outcome measures include an increase in wrist active extension or surface electromyography (the amplitude and cocontraction of extensor carpi radialis during movement), the action research arm test (ARAT), the motor status scale (MSS), and Barthel index (BI). Time-frequency analysis and power spectral analysis were used to reflect the electroencephalogram (EEG) change before and after the intervention. Results Compared with the baseline, the FMA-UE score increased significantly in the BCI group (p = 0.006). MSS scores improved significantly in both groups, while ARAT did not improve significantly. In addition, before the intervention, all patients could not actively extend their wrists or just had muscle contractions. After the intervention, four patients regained the ability to extend their paretic wrists (two in each group). The amplitude and area under the curve of extensor carpi radialis improved to some extent, but there was no statistical significance between the groups. Conclusion MI-based BCI combined with sensory and visual feedback might improve severe upper limb and hand impairment in chronic stroke patients, showing the potential for application in rehabilitation medicine.

One Does Not Simply Teach Idioms: Meme Creation as Innovative Practice for Virtual EFL Learners

To maximize the advantages of virtual learning, the present study highlights the potential for Internet meme design and creation in English language learning (ELL) courses as an innovative activity that raises student agency, increases multimodal literacy, inculcates intercultural communication, and teaches idiomatic expression. Memes resonate a multimodal feedback loop of popular culture. In the context of language education, multimodal literacy is a necessity for 21st-century education because the affordances of digital learning platforms present the world told alongside the world shown. While some studies feature the usefulness of memes in English as a foreign language (EFL) learning, none have underscored meme creation as a learning activity. To demonstrate the activity in situ, a vignette at two Korean universities features two instructors who ask their respective students ( N = 49) to design one meme using an idiom discovered in their ELL materials from a prescribed list, then asks: 1) What common power relations and ideologies emerge in the multimodal discourse of the collected pool of student “idiomemes”? 2) What do the findings tell us about student attitudes and engagement with the activity? 3) What do the findings tell us about the importance of multimodal discourse in EFL learning? Using a multimodal critical discourse analysis of the student-created Internet memes, the findings reveal that students chose culturally familiar images to complete the assignment, suggesting that their engagement and understanding of multimodal, English discourse increases commensurately with content intuitive to their culture. The implications suggest that empowering students with a measure of agency in expressing culturally relevant, multimodal discourse in ELL course content increases their engagement in virtual classrooms. Designing idiomemes, as a virtual learning activity, is further explored as a curricular augmentation that increases the value of a student's language-learning investment.

Big data visualization using multimodal feedback in education

Currently, Big Data Analytics help educators better analyze students and monitor their progress and likelihood of success. The online learning experience applies to the experiences in the online computer-mediated digital system to collect information. An essential part of education training programs is challenging and effectively implemented between teachers and students in the online learning feedback system. Hence, a Big Data Visualization assisted Multimodal Feedback Framework (BDVMFF) is proposed to boost students' confidence, self-consciousness, and motivation in the online learning platform. The BDVMFF offers the teacher a digital workflow to effectively exchange both the writing and the input to use multimodal feedback effectively. This system provides teachers and students with an effective and straightforward digital learning environment. The simulation results of BDVMFF show the highest performance ratio (97.9%), the efficiency ratio (96.1%), the grade analysis ratio (93.5%), the computation ratio (95.3%), and the lowest response time compared to existing methods.

Assessing the Role of Gaze Tracking in Optimizing Humans-In-The-Loop Telerobotic Operation Using Multimodal Feedback.

A key challenge in achieving effective robot teleoperation is minimizing teleoperators’ cognitive workload and fatigue. We set out to investigate the extent to which gaze tracking data can reveal how teleoperators interact with a system. In this study, we present an analysis of gaze tracking, captured as participants completed a multi-stage task: grasping and emptying the contents of a jar into a container. The task was repeated with different combinations of visual, haptic, and verbal feedback. Our aim was to determine if teleoperation workload can be inferred by combining the gaze duration, fixation count, task completion time, and complexity of robot motion (measured as the sum of robot joint steps) at different stages of the task. Visual information of the robot workspace was captured using four cameras, positioned to capture the robot workspace from different angles. These camera views (aerial, right, eye-level, and left) were displayed through four quadrants (top-left, top-right, bottom-left, and bottom-right quadrants) of participants’ video feedback computer screen, respectively. We found that the gaze duration and the fixation count were highly dependent on the stage of the task and the feedback scenario utilized. The results revealed that combining feedback modalities reduced the cognitive workload (inferred by investigating the correlation between gaze duration, fixation count, task completion time, success or failure of task completion, and robot gripper trajectories), particularly in the task stages that require more precision. There was a significant positive correlation between gaze duration and complexity of robot joint movements. Participants’ gaze outside the areas of interest (distractions) was not influenced by feedback scenarios. A learning effect was observed in the use of the controller for all participants as they repeated the task with different feedback combination scenarios. To design a system for teleoperation, applicable in healthcare, we found that the analysis of teleoperators’ gaze can help understand how teleoperators interact with the system, hence making it possible to develop the system from the teleoperators’ stand point.

Screencast Video Feedback and its Implication on English as a Foreign Language (EFL) Writing

Providing feedback to students’ written work has always been a challenging experience for English as a foreign language (EFL) teachers and learners. High-quality feedback promotes students’ engagement in learning processes and enhances writing performance. Traditional written corrective feedback has often been criticized for not being able to achieve its purpose. 21st-century technological development brought the necessity to provide audio and video feedback through screencast technology. The letter enables EFL teachers to provide multimodal feedback by recording the teacher’s screen while commenting on a student’s written work. Although there have been some studies conducted in the field of oral feedback via screencast, video feedback is still relatively new in many educational settings. For this reason, the paper aims to provide a brief overview of screencast video feedback, potential affordances and challenges faced by EFL teachers and learners. For this article, recent research studies have been collected to review the use of screencast feedback in EFL class and discuss its implications on EFL students’ writing. Furthermore, the paper provides an overview of the most widely-used screencast software in educational settings and concludes with some practical guidelines for the effective implementation of screencast technology.

Emotional Response to Vibrothermal Stimuli

Emotional response to haptic stimuli is a widely researched topic, but the combination of vibrotactile and thermal stimuli requires more attention. The purpose of this study is to investigate emotional response to vibrothermal stimulation by combining spatiotemporal vibrotactile stimulus with dynamic thermal stimulus (hot or cold). The vibrotactile and thermal stimuli were produced using the Haptic Chair and the Embr wave thermal bracelet, respectively. The results show that spatiotemporal vibrotactile patterns and their duration, and dynamic thermal stimulation, have an independent effect on the emotional response. Increasing duration generally increases the valence and arousal of emotional response. Shifting the dynamic temperature from cold to hot generally decreases the valence of emotional response but has no significant effect on arousal. Nevertheless, certain spatiotemporal patterns do exhibit unique responses to changes in dynamic temperature, although no interaction effects were found. The results show the potential of designing affective haptic interfaces using multimodal vibrothermal feedback.

Multimodal Feedback on speaking: Effectiveness and Students’ Perspective

Abstract. Multimodal feedback needs to be studied and researched to enhance student’s ability. Thus, this research aims to examine the feedback which using technology and conventional without using technology in speaking, wanted to explore how multimodal feedback could influence the students’ speaking ability, know the student’s perception on multimodal feedback and the reason multimodal feedback could improve students’ speaking ability. The research used a sequential mixed method design with test and interview as instruments. The sample was 54 students for experimental and control group. The result showed that there is not significant different between multimodal feedback with using technology and without using technology. However, in terms of mean score, multimodal feedback with using technology is higher than multimodal feedback without using technology on improving students’ speaking ability. Another result showed that the students had positive response towards the use of multimodal feedback on learning speaking because it can give them much knowledge and upgrade their ability in speaking class. Keywords: Multimodal feedback; using technology; speaking ability

Multimodal Warnings in Remote Operation: The Case Study on Remote Driving

Developments in sensor technology, artificial intelligence, and network technologies like 5G has made remote operation a valuable method of controlling various types of machinery. The benefits of remote operations come with an opportunity to access hazardous environments. The major limitation of remote operation is the lack of proper sensory feedback from the machine, which in turn negatively affects situational awareness and, consequently, may risk remote operations. This article explores how to improve situational awareness via multimodal feedback (visual, auditory, and haptic) and studies how it can be utilized to communicate warnings to remote operators. To reach our goals, we conducted a controlled, within-subjects experiment in eight conditions with twenty-four participants on a simulated remote driving system. Additionally, we gathered further insights with a UX questionnaire and semi-structured interviews. Gathered data showed that the use of multimodal feedback positively affected situational awareness when driving remotely. Our findings indicate that the combination of added haptic and visual feedback was considered the best feedback combination to communicate the slipperiness of the road. We also found that the feeling of presence is an important aspect of remote driving tasks, and a requested one, especially by those with more experience in operating real heavy machinery.

Improvement in Glucose Regulation Using a Digital Tracker and Continuous Glucose Monitoring in Healthy Adults and Those with Type2 Diabetes.

IntroductionWhile continuous glucose monitoring (CGM) has been shown to decrease both hyper- and hypoglycemia in insulin-treated diabetes, its value in non-insulin-treated type 2 diabetes (T2D) and prediabetes is unclear. Studies examining the reduction in hyperglycemia with the use of CGM in non-insulin-treated T2D are limited.MethodsWe investigated the potential benefit of CGM combined with a mobile app that links each individual’s glucose tracing to meal composition, heart rate, and physical activity in a cohort of 1022 individuals, ranging from nondiabetic to non-insulin-treated T2D, spanning a wide range of demographic, geographic, and socioeconomic characteristics. The primary endpoint was the change in time in range (TIR), defined as 54–140 mg/dL for healthy and prediabetes, and 54–180 mg/dL for T2D, from the beginning to end of a 10-day period of use of the Freestyle Libre CGM. Logged food intake, physical activity, continuous glucose, and heart rate data were captured by a smartphone-based app that continuously provided feedback to participants, overlaying daily glucose patterns with activity and food intake, including macronutrient breakdown, glycemic load (GL), and glycemic index (GI).ResultsA total of 665 participants meeting eligibility and data requirements were included in the final analysis. Among self-reported nondiabetic participants, CGM identified glucose excursions in the diabetic range among 15% of healthy and 36% of those with prediabetes. In the group as a whole, TIR improved significantly (p < 0.001). Among the 51.4% of participants who improved, TIR increased by an average of 6.4% (p < 0.001). Of those with poor baseline TIR, defined as TIR below comparable A1c thresholds for T2D and prediabetes, 58.3% of T2D and 91.7% of healthy/prediabetes participants improved their TIR by an average of 22.7% and 23.2%, respectively. Predictors of improved response included no prior diagnosis of T2D and lower BMI.ConclusionsThese results indicate that 10-day use of CGM as a part of multimodal data collection, with synthesis and feedback to participants provided by a mobile health app, can significantly reduce hyperglycemia in non-insulin-treated individuals, including those with early stages of glucose dysregulation.Supplementary InformationThe online version contains supplementary material available at 10.1007/s13300-021-01081-3.

Bio-Inspired Haptic Feedback for Artificial Palpation in Robotic Surgery.

Adding haptic feedback has been reported to improve the outcome of minimally invasive robotic surgery. In this study, we seek to determine whether an algorithm based on simulating responses of a cutaneous afferent population can be implemented to improve the performance of presenting haptic feedback for robot-assisted surgery. We propose a bio-inspired controlling model to present vibration and force feedback to help surgeons localize underlying structures in phantom tissue. A single pair of actuators was controlled by outputs of a model of a population of cutaneous afferents based on the pressure signal from a single sensor embedded in surgical forceps. We recruited 25 subjects including 10 expert surgeons to evaluate the performance of the bio-inspired controlling model in an artificial palpation task using the da Vinci surgical robot. Among the control methods tested, the bio-inspired system was unique in allowing both novices and experts to easily identify the locations of all classes of tumors and did so with reduced contact force and tumor contact time. This work demonstrates the utility of our bio-inspired multi-modal feedback system, which resulted in superior performance for both novice and professional users, in comparison to a traditional linear and the existing piecewise discrete algorithms of haptic feedback.

Policy Blending and Recombination for Multimodal Contact-Rich Tasks

Multimodal information such as tactile, proximity and force sensing is essential for performing stable contact-rich manipulations. However, coupling multimodal information and motion control still remains a challenging topic. Rather than learning a monolithic skill policy that takes in all feedback signals at all times, skills should be divided into phases and learn to only use the sensor signals applicable to that phase. This makes learning the primitive policies for each phase easier, and allows the primitive policies to be more easily reused among different skills. However, stopping and abruptly switching between each primitive policy results in longer execution times and less robust behaviours. We therefore propose a blending approach to seamlessly combining the primitive policies into a reliable combined control policy. We evaluate both time-based and state-based blending approaches. The resulting approach was successfully evaluated in simulation and on a real robot, with an augmented finger vision sensor, on: opening a cap, turning a dial and flipping a breaker tasks. The evaluations show that the blended policies with multimodal feedback can be easily learned and reliably executed.

Image retrieval with mixed initiative and multimodal feedback

How would you search for a unique, flamboyant shoe that a friend wore and you want to buy? What if you did not take a picture? Existing approaches propose interactive image search, but they either entrust the user with taking the initiative to provide informative feedback, or give all control to the system which determines informative questions to ask. Instead, we propose a mixed-initiative framework where both the user and system can be active participants, depending on whose input will be more beneficial for obtaining high-quality search results. We develop a reinforcement learning approach which dynamically decides which of four interaction opportunities to give to the user: drawing a sketch, marking images as relevant or not, providing free-form attribute feedback, or answering attribute-based questions. By allowing these four options, our system optimizes both the informativeness of feedback, and the ability of the user to explore the data, allowing faster image retrieval. We outperform five baselines on three datasets under extensive settings.

Multi-Session Influence of Two Modalities of Feedback and Their Order of Presentation on MI-BCI User Training

By performing motor-imagery tasks, for example, imagining hand movements, Motor-Imagery based Brain-Computer Interfaces (MI-BCIs) users can control digital technologies, for example, neuroprosthesis, using their brain activity only. MI-BCI users need to train, usually using a unimodal visual feedback, to produce brain activity patterns that are recognizable by the system. The literature indicates that multimodal vibrotactile and visual feedback is more effective than unimodal visual feedback, at least for short term training. However, the multi-session influence of such multimodal feedback on MI-BCI user training remained unknown, so did the influence of the order of presentation of the feedback modalities. In our experiment, 16 participants trained to control a MI-BCI during five sessions with a realistic visual feedback and five others with both a realistic visual feedback and a vibrotactile one. training benefits from a multimodal feedback, in terms of performances and self-reported mindfulness. There is also a significant influence of the order presentation of the modality. Participants who started training with a visual feedback had higher performances than those who started training with a multimodal feedback. We recommend taking into account the order of presentation for future experiments assessing the influence of several modalities of feedback.

The voice, text, and the visual as semiotic companions: an analysis of the materiality and meaning potential of multimodal screen feedback

The gap between how learners interpret and act upon feedback has been widely documented in the research literature. What is less certain is the extent to which the modality and materiality of the feedback influence students’ and teachers’ perceptions. This article explores the semiotic potential of multimodal screen feedback to enhance written feedback. Guided by an “Inquiry Graphics” approach, situated within a semiotic theory of learning edusemiotic conceptual framework, constructions of meaning in relation to screencasting feedback were analysed to determine how and whether it could be incorporated into existing feedback practices. Semi-structured video elicitation interviews with student teachers were used to incorporate both micro and macro levels of analysis. The findings suggested that the relationship between the auditory, visual and textual elements in multimodal screen feedback enriched the feedback process, highlighting the importance of form in addition to content to aid understanding of written feedback. The constitutive role of design and material artefacts in feedback practices in initial teacher training pertinent to these findings is also discussed.

Show, Don't Tell: Multimodal Story Feedback in a K–1 Play-Based Writing Unit

BackgroundWith standardization ever squeezing creative curricula in K–1 classrooms, creating time for a play-based multimodal writing curriculum that leverages children's strengths as storytellers is revolutionary. Due in part to accountability policy pressures, print-based writing and verbocentric writing feedback are still often privileged in school curricula. And yet, children are natural whole-body storytellers who will be asked to write and present ideas in all sorts of forms. In order to leverage children's storytelling strengths, we need to teach writing through multiple modes: This means expanding both writing instruction and the types of feedback offered to writers in primary classrooms.Research QuestionsThis study examines two questions: How is feedback being given, and what impact does it have on children's storytelling? How is play/storying being sanctioned?Setting &amp; ParticipantsThe study took place in a K–1 classroom in an inquiry-based, project-based school in the U.S. Midwest during a month-long storytelling workshop unit. Participants included two co-teachers and 46 children aged 5 to 7.Research DesignThis qualitative study used ethnographic methods and participant observation.Data Collection &amp; AnalysisVideo data were collected during workshop each day for one month, including minilessons, writing time, and share time, which is the focus of this article. Discourse analysis and a multimodality theoretical lens were used to analyze how children gave one another feedback on their stories through embodied demonstration, gesture, acting, out, or copying one another's storytelling devices.FindingsFindings indicate that children's acting/embodiment, humor/parody, and copying all worked as effective forms of multimodal feedback, which ultimately functioned as teaching for developing peers’ storytelling strategies and skills. However, teachers inadvertently privileged language alone via narration, or language with demonstration in feedback sessions.ConclusionsTeacher/researcher collaborations should explore ways to reimagine forms of writer's feedback that include and account for demonstration, copying, and impromptu performance and that, ultimately, open up the definition of what counts as writing at school. Um, you should work on making your story, like, real. Because, um, you're going all over the place [wiggles entire body to illustrate]—Allen, age 6

A Survey on Simulation for Weight Perception in Virtual Reality

Virtual reality (VR) technology can provide users with an immersive experience as if they are in the real world, which can be applied in the fields of entertainment, education and scientific research, etc. In order to improve the sense of presence and immersion in VR, the design of multimodal feedback is an important component. In particular, the simulation of weight of virtual objects poses many challenges due to the limitations of hardware and software. Many researchers focused on this issue in various ways. These methods are mainly divided into two categories: device-based simulation and software-based simulation. This paper investigates the focus of software-based simulation, particularly for the virtual feedback methods proposed by researchers in recent years. We introduce the background of these proposed methods, technical implementation principles, application scenarios, the advantages and disadvantages of these simulation methods, and the evaluation criteria. We also propose the future challenges and the development of simulation methods for weight perception of virtual objects in VR.

Towards Automatic Modeling of Volleyball Players’ Behavior for Analysis, Feedback, and Hybrid Training

Automatic tagging of video recordings of sports matches and training sessions can be helpful to coaches and players and provide access to structured data at a scale that would be unfeasible if one were to rely on manual tagging. Recognition of different actions forms an essential part of sports video tagging. In this paper, the authors employ machine learning techniques to automatically recognize specific types of volleyball actions (i.e., underhand serve, overhead pass, serve, forearm pass, one hand pass, smash, and block which are manually annotated) during matches and training sessions (uncontrolled, in the wild data) based on motion data captured by inertial measurement unit sensors strapped on the wrists of eight female volleyball players. Analysis of the results suggests that all sensors in the inertial measurement unit (i.e., magnetometer, accelerometer, barometer, and gyroscope) contribute unique information in the classification of volleyball actions types. The authors demonstrate that while the accelerometer feature set provides better results than other sensors, overall (i.e., gyroscope, magnetometer, and barometer) feature fusion of the accelerometer, magnetometer, and gyroscope provides the bests results (unweighted average recall = 67.87%, unweighted average precision = 68.68%, and κ = .727), well above the chance level of 14.28%. Interestingly, it is also demonstrated that the dominant hand (unweighted average recall = 61.45%, unweighted average precision = 65.41%, and κ = .652) provides better results than the nondominant (unweighted average recall = 45.56%, unweighted average precision = 55.45, and κ = .553) hand. Apart from machine learning models, this paper also discusses a modular architecture for a system to automatically supplement video recording by detecting events of interests in volleyball matches and training sessions and to provide tailored and interactive multimodal feedback by utilizing an HTML5/JavaScript application. A proof of concept prototype developed based on this architecture is also described.

A Multimodal Real-Time Feedback Platform Based on Spoken Interactions for Remote Active Learning Support.

While technology has helped improve process efficiency in several domains, it still has an outstanding debt to education. In this article, we introduce NAIRA, a Multimodal Learning Analytics platform that provides Real-Time Feedback to foster collaborative learning activities’ efficiency. NAIRA provides real-time visualizations for students’ verbal interactions when working in groups, allowing teachers to perform precise interventions to ensure learning activities’ correct execution. We present a case study with 24 undergraduate subjects performing a remote collaborative learning activity based on the Jigsaw learning technique within the COVID-19 pandemic context. The main goals of the study are (1) to qualitatively describe how the teacher used NAIRA’s visualizations to perform interventions and (2) to identify quantitative differences in the number and time between students’ spoken interactions among two different stages of the activity, one of them supported by NAIRA’s visualizations. The case study showed that NAIRA allowed the teacher to monitor and facilitate the learning activity’s supervised stage execution, even in a remote learning context, with students working in separate virtual classrooms with their video cameras off. The quantitative comparison of spoken interactions suggests the existence of differences in the distribution between the monitored and unmonitored stages of the activity, with a more homogeneous speaking time distribution in the NAIRA supported stage.