Real-time Visual Feedback Research Articles

Intraoperative computed tomography for the management of traumatic complex facial fractures: a narrative review

Successful facial reconstruction requires accurate restoration of the concealed three-dimensional facial skeleton. The key rationale for intraoperative computed tomography (iCT) is that it facilitates the most comprehensive assessment of fracture reduction. Real-time visual feedback is now considered an essential component in hand, spine and general orthopaedic surgery. Frequently proposed as the new gold standard of care, iCT has been widely adopted in maxillofacial surgery, especially internationally. In contrast, our Australian experience is that access to iCT and therefore uptake is relatively lacking. The craniofacial skeleton is a complex structure owing to its various convexities and concavities, sinonasal elements, biomechanical buttresses, visceral and occlusal components and neurovascular structures. Classic approaches to open reduction internal fixation aim to reduce incisions to prevent complications such as ectropion, salivary leak or nerve damage. However, intraoperative assessment of fractures in minimally-invasive techniques are hindered by the limited extent of exposure, especially in the setting of complex fracture patterns or those that involve dynamic or functional endpoints. With iCT, surgeons may visualise, adjust and revise fixation without wide exposure while the patient is on the operating table. This obviates an additional operation that may be required due to an unsatisfactory fixation. iCT is cost-effective and reduces radiation exposure. Challenges in integrating iCT include competing for availability with other specialties, training staff and lack of awareness about the benefits this technology offers. This narrative review discusses the role of iCT in the modern Australian operating theatre.

Development and evaluation of a BCI-neurofeedback system with real-time EEG detection and electrical stimulation assistance during motor attempt for neurorehabilitation of children with cerebral palsy.

In the realm of motor rehabilitation, Brain-Computer Interface Neurofeedback Training (BCI-NFT) emerges as a promising strategy. This aims to utilize an individual's brain activity to stimulate or assist movement, thereby strengthening sensorimotor pathways and promoting motor recovery. Employing various methodologies, BCI-NFT has been shown to be effective for enhancing motor function primarily of the upper limb in stroke, with very few studies reported in cerebral palsy (CP). Our main objective was to develop an electroencephalography (EEG)-based BCI-NFT system, employing an associative learning paradigm, to improve selective control of ankle dorsiflexion in CP and potentially other neurological populations. First, in a cohort of eight healthy volunteers, we successfully implemented a BCI-NFT system based on detection of slow movement-related cortical potentials (MRCP) from EEG generated by attempted dorsiflexion to simultaneously activate Neuromuscular Electrical Stimulation which assisted movement and served to enhance sensory feedback to the sensorimotor cortex. Participants also viewed a computer display that provided real-time visual feedback of ankle range of motion with an individualized target region displayed to encourage maximal effort. After evaluating several potential strategies, we employed a Long short-term memory (LSTM) neural network, a deep learning algorithm, to detect the motor intent prior to movement onset. We then evaluated the system in a 10-session ankle dorsiflexion training protocol on a child with CP. By employing transfer learning across sessions, we could significantly reduce the number of calibration trials from 50 to 20 without compromising detection accuracy, which was 80.8% on average. The participant was able to complete the required calibration trials and the 100 training trials per session for all 10 sessions and post-training demonstrated increased ankle dorsiflexion velocity, walking speed and step length. Based on exceptional system performance, feasibility and preliminary effectiveness in a child with CP, we are now pursuing a clinical trial in a larger cohort of children with CP.

Integrating augmented reality, gesture recognition, and NLP for enhancing underwater human-robot interaction

This paper presents an in-depth exploration into the integration of augmented reality (AR), gesture recognition, and natural language processing (NLP) to enhance human-robot interaction (HRI) within the context of underwater robotics. It highlights the significant potential these technologies hold in addressing the unique challenges faced in underwater environments, such as limited visibility, complex navigation, and the need for precise, intuitive communication between divers and robots. By reviewing current technological advancements and applications, the study underscores the critical role of AR in providing real-time visual feedback, gesture recognition in enabling more natural control mechanisms, and NLP in facilitating voice-driven commands and interactions. The research further discusses the development of a conceptual framework for an AR-based intuitive interface that synergizes gesture recognition and NLP, aiming to revolutionize underwater HRI by making it more efficient, safe, and user-friendly. Through this investigation, the paper seeks to contribute to the advancement of underwater robotics, proposing innovative solutions that could significantly improve human-robot collaboration in challenging aquatic missions.

Reinforcement Learning during Locomotion.

When learning a new motor skill, people often must use trial and error to discover which movement is best. In the reinforcement learning framework, this concept is known as exploration and has been linked to increased movement variability in motor tasks. For locomotor tasks, however, increased variability decreases upright stability. As such, exploration during gait may jeopardize balance and safety, making reinforcement learning less effective. Therefore, we set out to determine if humans could acquire and retain a novel locomotor pattern using reinforcement learning alone. Young healthy male and female participants walked on a treadmill and were provided with binary reward feedback (indicated by a green checkmark on the screen) that was tied to a fixed monetary bonus, to learn a novel stepping pattern. We also recruited a comparison group who walked with the same novel stepping pattern but did so by correcting for target error, induced by providing real-time veridical visual feedback of steps and a target. In two experiments, we compared learning, motor variability, and two forms of motor memories between the groups. We found that individuals in the binary reward group did, in fact, acquire the new walking pattern by exploring (increasing motor variability). Additionally, while reinforcement learning did not increase implicit motor memories, it resulted in more accurate explicit motor memories compared with the target error group. Overall, these results demonstrate that humans can acquire new walking patterns with reinforcement learning and retain much of the learning over 24 h.

On-Screen Visual Feedback Effect on Static Balance Assessment with Perturbations.

In this study, the novel mobile dynamometric platform, OREKA, was utilized to perform an extensive analysis of the centre of pressure behaviour during different tilt motion exercises. This platform is based on a parallel manipulator mechanism and can perform rotations around both horizontal axes and a vertical translation. A group of participants took part in an experimental campaign involving the completion of a set of exercises. The aim was to evaluate the platform's potential practical application and investigate the impact of visual on-screen feedback on centre of pressure motion through multiple balance indicators. The use of the OREKA platform enables the study of the impact on a user's balance control behaviour under different rotational perturbations, depending on the availability of real-time visual feedback on a screen. Furthermore, it presented data identifying postural control variations among clinically healthy individuals. These findings are fundamental to comprehending the dynamics of body balance. Further investigation is needed to explore these initial findings and fully unlock the potential of the OREKA platform for balance assessment methodologies.

Validation of SPARCC MRI-RETIC e-tools for increasing scoring proficiency of MRI sacroiliac joint lesions in axial spondyloarthritis

BackgroundThe Spondyloarthritis Research Consortium of Canada (SPARCC) developers have created web-based calibration modules for the SPARCC MRI sacroiliac joint (SIJ) scoring methods. We aimed to test the impact of applying...

The Use of the Voice Trainer App for Vocal Control in People with a Degenerative Ataxia: A Pilot Intervention Study.

Dysarthria is disabling in persons with degenerative ataxia. There is limited evidence for speech therapy interventions. In this pilot study, we used the Voice trainer app, which was originally developed for patients with Parkinson's disease, as a feedback tool for vocal control. We hypothesized that patients with ataxic dysarthria would benefit from the Voice trainer app to better control their loudness and pitch, resulting in a lower speaking rate and better intelligibility. This intervention study consisted of five therapy sessions of 30min within 3weeks using the principles of the Pitch Limiting Voice Treatment. Patients received real-time visual feedback on loudness and pitch during the exercises. Besides, they were encouraged to practice at home or to use the Voice trainer in daily life. We used observer-rated and patient-rated outcome measures. The primary outcome measure was intelligibility, as measured by the Dutch sentence intelligibility test. Twenty-one out of 25 included patients with degenerative ataxia completed the therapy. We found no statistically significant improvements in intelligibility (p = .56). However, after the intervention, patients were speaking slower (p = .03) and the pause durations were longer (p < .001). The patients were satisfied about using the app. At the group level, we found no evidence for an effect of the Voice trainer app on intelligibility in degenerative ataxia. Because of the heterogeneity of ataxic dysarthria, a more tailor-made rather than generic intervention seems warranted.

Development and validation of an art-inspired multimodal interactive technology system for a multi-component intervention for older people: a pilot study

IntroductionThe World Health Organization (WHO) acknowledges the presence of a significant body of research on the positive effects of the arts on health, considering a variety of factors including physical well-being, quality of life, and social and community impact. The model that underlies cultural welfare puts the performing arts, visual arts, and cultural heritage at the service of people personal and societal well-being. The potential connections between movements of the body and artistic content have been extensively studied over time, considering movement as a non-verbal language with a universal character.MethodsThis pilot study presents the results of the validation of an innovative multimodal system, the DanzArTe-Emotional Wellbeing Technology, designed to support active and participative experience of older people providing physical and cognitive activation through a full-body physical interaction with a traditional visual work of art of religious subject. DanzArTe supports a replicable treatment protocol for multidimensional frailty, administered through a low cost and scalable technological platform capable of generating real-time visual and auditory feedback (interactive sonification) from the automated analysis of individual as well as joint movement expressive qualities. The study involved 45 participants, 23 of whom participated in the DanzArTe program and 22 who were included in the control group.ResultsThe two groups were similar in terms of age (p = 0.465) and gender (p = 0.683). The results showed that the DanzArTe program had a positive impact on participants' self-perceived psychological health and well-being (Mean Psychological General Well-Being Index—Short T1 = 19.6 ± 4.3 Vs. T2 = 20.8 ± 4.9; p = 0.029). The same trend was not observed in the control group (p = 0.389).DiscussionThe findings suggest that such programs may have a significant impact particularly on the mental and social well-being of older adults and could be a valuable tool for promoting healthy aging and improving quality of life.

Development of a motion-based video game for postural training: a feasibility study on older adults with adult degenerative scoliosis.

Forward sagittal alignment affects physical performance, is associated with pain and impacts the health-related quality of life of the elderly. Interventions that help seniors to improve sagittal balance are needed to inhibit the progression of pain and disability. A motion-sensing video game (active game) is developed in this study to monitor sitting and standing postures in real-time and facilitate the postural learning process by using optical sensors to measure body movement and a video game to provide visual feedback. Ten female subjects (mean age: 60.0 ± 5.2 years old; mean BMI: 21.4 ± 1.9) with adult degenerative scoliosis (mean major Cobb's angle: 38.1°± 22.7°) participate in a 6-week postural training programme with three one-hour postural training sessions a week. Eleven body alignment measurements of their perceived "ideal" sitting and standing postures are obtained before and after each training session to evaluate the effectiveness of postural learning with the game. The participants learn to sit and stand with increased sagittal alignment with a raised chest and more retracted head position. The forward shift of their head and upper body is significantly reduced after each training session. Although this immediate effect only partially sustained after the 6-week program, the participants learned to adjust their shoulder and pelvis level for a better lateral alignment in standing. The proposed postural training system, which is presented as a gameplay with real-time visual feedback, can effectively help players to improve their postures. This pilot feasibility study explores the development and initial assessment of a motion-based video game designed for postural training in older adults with adult degenerative scoliosis, and demonstrates the usability and benefits of active gameplay in motor training.

Accuracy of routine external ventricular drain placement following a mixed reality-guided twist-drill craniostomy.

The traditional freehand placement of an external ventricular drain (EVD) relies on empirical craniometric landmarks to guide the craniostomy and subsequent passage of the EVD catheter. The diameter and trajectory of the craniostomy physically limit the possible trajectories that can be achieved during the passage of the catheter. In this study, the authors implemented a mixed reality-guided craniostomy procedure to evaluate the benefit of an optimally drilled craniostomy to the accurate placement of the catheter. Optical marker-based tracking using an OptiTrack system was used to register the brain ventricular hologram and drilling guidance for craniostomy using a HoloLens 2 mixed reality headset. A patient-specific 3D-printed skull phantom embedded with intracranial camera sensors was developed to automatically calculate the EVD accuracy for evaluation. User trials consisted of one blind and one mixed reality-assisted craniostomy followed by a routine, unguided EVD catheter placement for each of two different drill bit sizes. A total of 49 participants were included in the study (mean age 23.4 years, 59.2% female). The mean distance from the catheter target improved from 18.6 ± 12.5 mm to 12.7 ± 11.3 mm (p = 0.0008) using mixed reality guidance for trials with a large drill bit and from 19.3 ± 12.7 mm to 10.1 ± 8.4 mm with a small drill bit (p < 0.0001). Accuracy using mixed reality was improved using a smaller diameter drill bit compared with a larger bit (p = 0.039). Overall, the majority of the participants were positive about the helpfulness of mixed reality guidance and the overall mixed reality experience. Appropriate indications and use cases for the application of mixed reality guidance to neurosurgical procedures remain an area of active inquiry. While prior studies have demonstrated the benefit of mixed reality-guided catheter placement using predrilled craniostomies, the authors demonstrate that real-time quantitative and visual feedback of a mixed reality-guided craniostomy procedure can independently improve procedural accuracy and represents an important tool for trainee education and eventual clinical implementation.

The Development of a Rehabilitation Orthotic Walker with a Real-time Visual Feedback System of the Gait Symmetry

Patients who suffer from a disability or face temporary walking difficulty due to accidents or medical conditions have to go through rehabilitation treatment, which requires them to use an orthotic walking assistive device such as a walker, cane, or crutches. It is estimated that there are more than 7.4 million people in the world who depend on walkers to assist them in their walking either due to a disability, old age, or as a part of their rehabilitation following accidents or medical conditions. The use of orthotic assistive devices may last for several weeks or months and encourage patients to keep the weight off the injured or weak limb and exert more force on the healthy limb. Such prolonged use of walking assistive devices and heavy reliance on healthy lower limbs may cause negative gait disorders for patients. Negative gait disorders are due to unbalanced gait because of the lack of continuous feedback to patients about their gait. Physical therapist feedback to patients is limited to their physical presence with patients; hence, it is not practical to give the patient corrective feedback about their gait with every walking step. Hence, to overcome this pitfall, a visual feedback system of the gait symmetry, which could be installed on any traditional rehabilitation orthotic walker, was designed, developed, and proven. The visual feedback system was 3D modeled utilizing SolidWorks 2021, and parts were 3D printed utilizing the Original Prusa i3 MK3S+ 3D printer. This real-time visual feedback system relies on load cells installed in the tips of the orthotic walker legs to measure the force and weight being exerted on each side of the orthotic walker. The visual feedback system was tested to give the patients visual feedback, encouraging them to correct their unbalanced gait by shifting their weight bearing on either of their two lower limbs. This novel and innovative system mitigates the negative effects of the traditional orthotic walker on the overall gait and helps to mitigate negative gait disorders in patients.

Extended Reality as a Training Approach for Visual Real-Time Feedback in Golf

Extended Reality as a Training Approach for Visual Real-Time Feedback in Golf

Development of a real-time RGB-D visual feedback-assisted pulmonary rehabilitation system

BackgroundFollowing surgery, perioperative pulmonary rehabilitation (PR) is important for patients with early-stage lung cancer. However, current inpatient programs are often limited in time and space, and outpatient settings have access barriers. Therefore, we aimed to develop a background-free, zero-contact thoracoabdominal movement-tracking model that is easily set up and incorporated into a pre-existing PR program or extended to home-based rehabilitation and remote monitoring. We validated its effectiveness in providing preclinical real-time RGB-D (colour-depth camera) visual feedback. MethodsTwelve healthy volunteers performed deep breathing exercises following audio instruction for three cycles, followed by audio instruction and real-time visual feedback for another three cycles. In the visual feedback system, we used a RealSense™ D415 camera to capture RGB and depth images for human pose-estimation with Google MediaPipe. Target-tracking regions were defined based on the relative position of detected joints. The processed depth information of the tracking regions was visualised on a screen as a motion bar to provide real-time visual feedback of breathing intensity. Pulmonary function was simultaneously recorded using spirometric measurements, and changes in pulmonary volume were derived from respiratory airflow signals. ResultsOur movement-tracking model showed a very strong correlation (r = 0.90 ± 0.05) between thoracic motion signals and spirometric volume, and a strong correlation (r = 0.73 ± 0.22) between abdominal signals and spirometric volume. Displacement of the chest wall was enhanced by RGB-D visual feedback (23 vs 20 mm, P = 0.034), and accompanied by an increased lung volume (2.58 vs 2.30 L, P = 0.003). ConclusionWe developed an easily implemented thoracoabdominal movement-tracking model and reported the positive impact of real-time RGB-D visual feedback on self-promoted external chest wall expansion, accompanied by increased internal lung volumes. This system can be extended to home-based PR.

Improving Balance and Movement Control in Fencing Using IoT and Real-Time Sensorial Feedback.

Fencing, a sport emphasizing the equilibrium and movement control of participants, forms the focal point of inquiry in the current study. The research endeavors to assess the efficacy of a novel system designed for real-time monitoring of fencers' balance and movement control, augmented by modules incorporating visual feedback and haptic feedback, to ascertain its potential for performance enhancement. Over a span of five weeks, three distinct groups, each comprising ten fencers, underwent specific training: a control group, a cohort utilizing the system with a visual real-time feedback module, and a cohort using the system with a haptic real-time feedback module. Positive outcomes were observed across all three groups, a typical occurrence following a 5-week training regimen. However, noteworthy advancements were particularly discerned in the second group, reaching approximately 15%. In contrast, the improvements in the remaining two groups were below 5%. Statistical analyses employing the Wilcoxon signed-rank test for repeated measures were applied to assess the significance of the results. Significance was solely ascertained for the second group, underscoring the efficacy of the system integrated with visual real-time feedback in yielding statistically noteworthy performance enhancements.

RealTHASC—a cyber-physical XR testbed for AI-supported real-time human autonomous systems collaborations

Today’s research on human-robot teaming requires the ability to test artificial intelligence (AI) algorithms for perception and decision-making in complex real-world environments. Field experiments, also referred to as experiments “in the wild,” do not provide the level of detailed ground truth necessary for thorough performance comparisons and validation. Experiments on pre-recorded real-world data sets are also significantly limited in their usefulness because they do not allow researchers to test the effectiveness of active robot perception and control or decision strategies in the loop. Additionally, research on large human-robot teams requires tests and experiments that are too costly even for the industry and may result in considerable time losses when experiments go awry. The novel Real-Time Human Autonomous Systems Collaborations (RealTHASC) facility at Cornell University interfaces real and virtual robots and humans with photorealistic simulated environments by implementing new concepts for the seamless integration of wearable sensors, motion capture, physics-based simulations, robot hardware and virtual reality (VR). The result is an extended reality (XR) testbed by which real robots and humans in the laboratory are able to experience virtual worlds, inclusive of virtual agents, through real-time visual feedback and interaction. VR body tracking by DeepMotion is employed in conjunction with the OptiTrack motion capture system to transfer every human subject and robot in the real physical laboratory space into a synthetic virtual environment, thereby constructing corresponding human/robot avatars that not only mimic the behaviors of the real agents but also experience the virtual world through virtual sensors and transmit the sensor data back to the real human/robot agent, all in real time. New cross-domain synthetic environments are created in RealTHASC using Unreal Engine™, bridging the simulation-to-reality gap and allowing for the inclusion of underwater/ground/aerial autonomous vehicles, each equipped with a multi-modal sensor suite. The experimental capabilities offered by RealTHASC are demonstrated through three case studies showcasing mixed real/virtual human/robot interactions in diverse domains, leveraging and complementing the benefits of experimentation in simulation and in the real world.

Rearfoot Strike Run Retraining forAchilles Tendon Pain: A Two-patient Case Series.

Running-related injuries are prevalent in the military and are often related to physical fitness test training. Non-rearfoot striking while running is known to increase the risk of Achilles tendon injuries because of the high eccentric energy absorption by the elastic components of the planarflexor muscle-tendon complex. However, there is limited evidence to suggest benefits of converting runners with Achilles tendon pain to use a rearfoot strike. This is a case series of two active-duty Service members with chronic, running-related Achilles tendon pain that utilized a natural non-rearfoot strike pattern. Both patients were trained to utilize a rearfoot strike while running through the use of real-time visual feedback from wearable sensors. The trained rearfoot strike pattern was retained for over one month after the intervention, and both patients reported improvements in pain and self-reported function. This case series demonstrated the clinical utility of converting two non-rearfoot strike runners to a rearfoot strike pattern to decrease eccentric demands on the plantarflexors and reduce Achilles tendon pain while running.

41 Role of real-time visual feedback device on cardiopulmonary resuscitation training among school children – A randomized controlled trial

41 Role of real-time visual feedback device on cardiopulmonary resuscitation training among school children – A randomized controlled trial

Multimodal feedback systems for smart laser osteotomy: Depth control and tissue differentiation.

The study aimed to improve the safety and accuracy of laser osteotomy (bone surgery) by integrating optical feedback systems with an Er:YAG laser. Optical feedback consists of a real-time visual feedback system that monitors and controls the depth of laser-induced cuts and a tissue sensor differentiating tissue types based on their chemical composition. The developed multimodal feedback systems demonstrated the potential to enhance the safety and accuracy of laser surgery. The proposed method utilizes a laser-induced breakdown spectroscopy (LIBS) system and long-range Bessel-like beam optical coherence tomography (OCT) for tissue-specific laser surgery. The LIBS system detects tissue types by analyzing the plasma generated on the tissue by a nanosecond Nd:YAG laser, while OCT provides real-time monitoring and control of the laser-induced cut depth. The OCT system operates at a wavelength of 1288 ± 30 nm and has an A-scan rate of 104.17 kHz, enabling accurate depth control. Optical shutters are used to facilitate the integration of these multimodal feedback systems. The proposed system was tested on five specimens of pig femur bone to evaluate its functionality. Tissue differentiation and visual depth feedback were used to achieve high precision both on the surface and in-depth. The results showed successful real-time tissue differentiation and visualization without any visible thermal damage or carbonization. The accuracy of the tissue differentiation was evaluated, with a mean absolute error of 330.4 μm and a standard deviation of ±248.9 μm, indicating that bone ablation was typically stopped before reaching the bone marrow. The depth control of the laser cut had a mean accuracy of 65.9 μm with a standard deviation of ±45 μm, demonstrating the system's ability to achieve the pre-planned cutting depth. The integrated approach of combining an ablative laser, visual feedback (OCT), and tissue sensor (LIBS) has significant potential for enhancing minimally invasive surgery and warrants further investigation and development.

Results From a Multi-Institutional Pilot Study of iContour, an Interactive Online Platform with Real-Time Feedback to Improve Contouring Education for Radiation Oncology Residents

Numerous studies have shown that variability in contouring by radiation oncologists is common and associated with poor clinical outcomes. Contouring is taught via an apprenticeship model during residency with inconsistent results. Currently, there is no standardized contouring curriculum. We hypothesized that an interactive online educational platform for learners to practice contouring and receive real-time visual feedback would be a useful curricular tool. The iContour platform displays anonymized DICOM data and allows for input and analysis of user contours in a web-based interface. Nine cases are available from the Head/Neck (H+N), Gynecologic (Gyn), and Gastrointestinal (GI) disease sites. The system presents users with a case and asks them to contour representative slices from 2-3 target volumes or OARs. Upon submission, users are shown several forms of feedback. These include immediate visual comparison with expert contours on the same dataset (all cases), customized feedback based on overlap with prespecified "avoidance" and "inclusion" structures that highlight common mistakes (Gyn cases), and tumor control probability models to estimate the clinical impact of a user's contour variations on patient outcomes (H+N cases). Some cases include short videos outlining anatomy and contouring principles which are shown before the user contours. A pilot study was performed to evaluate technical performance and educational utility. Pre- and post-surveys with Likert-type questions (1-5 scale from strongly disagree to strongly agree) were used to assess user satisfaction and preferences regarding feedback. A total of 9 residents participated (median PGY3, range PGY2-5) from 5 institutions in 2 countries. Each participant completed 2 cases from a single disease site (n = 3 each for H+N, Gyn and GI), one with educational videos and one without. 67% of users had completed a prior clinical rotation in their disease site. Overall, residents felt the system was a useful educational tool (mean Likert score 4.67 +/- 0.47) and were interested in using it during clinical rotations (4.89 +/- 0.31). Most participants (7/9) felt iContour was more useful than existing resources for contouring education. Residents unanimously (9/9) found direct visual comparison with expert contours the most useful type of feedback, and that cases with videos before contouring were more educational. The iContour platform is a useful educational tool for radiation oncology residents. Participants felt receiving immediate visual feedback on contours was a valuable learning experience. Short instructional videos before contouring can be utilized to provide "just in time" teaching. A randomized study to formally assess the platform's impact on contouring skills is planned.

362 Role of Real-Time Visual Feedback Device on Cardiopulmonary Resuscitation Training Among School Children: A Randomized Controlled Trial

362 Role of Real-Time Visual Feedback Device on Cardiopulmonary Resuscitation Training Among School Children: A Randomized Controlled Trial