Visual Guidance Research Articles

Visual Impairment as a Learning Disability in Children with Cerebral Palsy: A Narrative Review

Cerebral Palsy (CP) is a group of non-progressive brain disorders causing movement, posture, and motor function impairments. Cerebral Visual Impairment (CVI) affects visual pathways, affecting clarity of vision, visual fields, and occulomotor incoordination. Symptoms vary in nature and severity due to the complexity of the visual cortex. Physiotherapy (PT) is essential for treating CP, promoting physical, mental, and social well-being. Physical therapists also teach parents, how to care for their children at home, including feeding, bathing, dressing, and other activities. CVI is often linked to CP and can lead to various visual problems, including reduced visual guidance of movement. To effectively manage this condition, a multidisciplinary approach is required, as children with CVI often have neurological deficits in addition to their visual problems. Visual function assessment involves measuring thresholds or limits for each eye, while functional vision assessment is conducted in the child’s environment with both eyes open. Levels of vision can be classified into light perception, intermittent fixation, reliable focus, constant attention to small objects, and reliable visual acuity. Children with Visual Difficulties (CVI) exhibit distinct visual behaviours and require proper training. Sensory room training improves gross and fine motor abilities, and early intervention reduces issues associated with these diseases. Physical therapy interventions are very important for this population’s health and wellness, which can be focused on acquiring and improving independence in motor abilities, visual function, coordination, balance and general health improvement. Children with CVI are at danger of developing neurological dysfunction. An early intervention and adequate training for young CP children with CVI can improve motor abilities and reduce issues associated with these diseases. The following review will help to understand briefly about visual impairments in patients with CP, mainly focusing on new therapeutic interventions following visual impairments that are effective and easily administered.

Insertion of a decompression tube to the proximal colon under direct colonoscopic visual guidance in acute colonic pseudo-obstruction

Insertion of a decompression tube to the proximal colon under direct colonoscopic visual guidance in acute colonic pseudo-obstruction

TSFNet: Triple-Steam Image Captioning

Image captioning is a challenging task that generates a natural language description based on the visual understanding of the given image. Significant region representation is a milestone in image captioning. Despite the great success of existing region-based works, they only focus on the salient objects and encode these objects independently, still plagued by the lack of global contextual information and visual relationships. In fact, the global contextual information and structured visual relationships are exactly the merits of traditional grid features and emerging scene graph features. In this paper, we present a Triple-Steam Feature Fusion Network (TSFNet) to leverage the complementary advantages of the grid, region, and scene graph triple-steam visual representations in image captioning. Concretely, in our TSFNet, a novel Dual-level Attention (DA) mechanism is proposed to simultaneously explore visual intrinsic properties and word-related attributes uniformly of different features. Then attention enhanced features of different modalities are mapped into a joint representation to guide the caption generation. Moreover, we design a new global-aware decoder, which leverages the concatenated representation of triple-steam features and the joint attention representation to obtain global visual guidance information, further refine the complex multimodal reasoning. To verify the effectiveness of our feature fusion model, we perform extensive experiments on the highly competitive MSCOCO dataset to evaluate the model quantitatively and qualitatively. The results illustrate that the proposed framework outperforms many state-of-the-art image captioning approaches in various evaluation metrics, and generates more accurate and abundant captions.

Automated Medical Visualization Application of Supervised Learning to Clinical Diagnosis, Disease and Therapy Management.docx

The rapid advancement and development in high performance computing, ultrafast computing, autonomous technologies and complexity of biomedical data for visualization and image guidance play a significant role in modern surgery to help surgeons perform their surgical procedures. Brain tumour diagnosis requires an enhanced, effective as well as accurate 3-D visualization system for navigation, reference, diagnosis as well as documentation. The automatic and effective 3-D high performance artificial intelligence-enabled medical visualization framework was designed and implemented using automated machine learning (AutoML) to take the advantage of complexity in the underlying datasets to help specialists in identifying the most appropriate regions of interest and their associated hyper parameters for optimizing performance, whilst simultaneously attempting to maximize the reliability of resulting predictions. C# and Compute Unified Device Architecture (CUDA) in Microsoft.NET environment in comparison side by side with visual basic studio was used for the implementation. The framework was evaluated for rendering processing speed with brain datasets obtained from the department of surgery, University of North Carolina, United States. Interestingly, our framework achieves 3-D visualization of the human brain, reliable enough to detect and locate possible brain tumor with high interactive speed and accuracy.

Towards 2D/3D Registration of the Preoperative MRI to Intraoperative Fluoroscopic Images for Visualisation of Bone Defects

ABSTRACT Magnetic Resonance Imaging (MRI) is a medical imaging modality that allows for the evaluation of soft-tissue diseases and the assessment of bone quality. Preoperative MRI volumes are used by surgeons to identify defected bones, perform the segmentation of lesions, and generate surgical plans before the surgery. Nevertheless, conventional intraoperative imaging modalities such as fluoroscopy are less sensitive in detecting potential lesions. In this work, we propose a 2D/3D registration pipeline that aims to register preoperative MRI with intraoperative 2D fluoroscopic images. To showcase the feasibility of our approach, we use the core decompression procedure as a surgical example to perform 2D/3D femur registration. The proposed registration pipeline is evaluated using digitally reconstructed radiographs (DRRs) to simulate the intraoperative fluoroscopic images. The resulting transformation from the registration is later used to create overlays of preoperative MRI annotations and planning data to provide intraoperative visual guidance to surgeons. Our results suggest that the proposed registration pipeline is capable of achieving reasonable transformation between MRI and digitally reconstructed fluoroscopic images for intraoperative visualisation applications.

Usability, acceptability and feasibility of a novel technology with visual guidance with video and audio recording during newborn resuscitation: a pilot study

ObjectiveInadequate adherence to resuscitation for non-crying infants will have poor outcome and thus rationalise a need for real-time guidance and quality improvement technology. This study assessed the usability, feasibility and...

Fast Recognition and Pose Estimation Algorithm for Space Cooperative Target via Mono-vision

Visual guidance based on cooperative target is one of the most commonly used approaches in the field of space in-orbit service, such as spacecraft rendezvous and docking and satellite maintenance. Considering the limited space hardware resources, it is necessary to design an embedded image processing algorithm to meet the special needs of space recognition, such as small memory consumption and fast recognition speed. Therefore, aiming at the classic three-point cooperative target, recognition, and pose estimation algorithm is designed in this paper based on mono-vision. In this method, a fast inflection point detection method is designed, and the contour is divided into several edges with a single feature, which improves the accuracy of circle recognition. Finally, the relative pose of the target coordinate system can be obtained quickly and accurately by P3P. The algorithm has low complexity and can realize fast recognition with small memory, even with limited space-grade hardware resources. The effectiveness of the proposed method is verified by experiments.

A novel visual guidance framework for robotic welding based on binocular cooperation

• An initial welding position guidance framework based on uncalibrated visual servoing and binocular cooperation is studied. The proposed method combines the advantages of passive and active vision sensors, and improves the flexibility of the vision system. • The global camera first controls the robot to approach the workpiece through the proposed EKF based uncalibrated visual servoing. Then, the derived cooperative motion equation will lead the robot towards the IWP. • 2 The features extraction algorithms for realizing UVS are studied. The method to estimate the desired position of welding torch is discussed. And a compression strategy for Yolo-v4 is designed to realize real-time welding torch detection, which can be an optional solution for target detection tasks on computational-resource-limited devices. • The simulation experiments demonstrate the feasibility and stability of the proposed guidance framework. The comparison experiments indicate that the framework can increase the flexibility of the vision system while maintaining satisfactory accuracy. Visual sensor based welding guidance technology has played an essential part in intelligentized robotic welding. However, scanning trajectory dependence of the laser vision sensor (LVS) and low accuracy of the passive vision sensor limit the application of single visual sensor based guidance methods in practical welding guidance. Aiming at this problem, a global camera is introduced to LVS system, and a novel initial welding point (IWP) guidance framework based on binocular cooperation is put forward. The robot is first controlled by the global camera to approach the workpiece through the proposed extended Kalman filter based uncalibrated visual servoing (UVS). Then, the derived cooperative equation of the LVS and the global camera will lead the robot toward the IWP. Subsequently, the feature extraction algorithms for UVS are designed. A compression strategy for Yolo-v4 is present to realize real-time welding torch detection in small computing platforms, and the estimation method of desired features is analyzed. Finally, the feasibility of the proposed algorithm is verified through experiments. The proposed framework combines the advantages of rich information for global camera and high precision for LVS, and could be a feasible solution to realize fully autonomous welding guidance.

Stable visually guided reaching does not require an internal feedforward model to compensate for internal delay: Data and model

Visually guided reaches are performed in ≈1s. Given unstable feedback control with neural transmission delay, stable visually guided reaching is assumed to require internal feedforward models that generate simulated feedback without delay that combines with actual feedback for stability. We investigated whether stable visually guided reaching requires internal models to handle such delay. Participants performed rapid targeted reaches in a virtual environment with different mappings between speeds of the hand and hand avatar. First, participants reached with visual guidance and constant mapping. Second, feedforward reaches were performed with constant mapping and hand avatar only visible at reach start and end. Reaches were accurate. Third, participants performed reaches with visual guidance and different mappings every trial. We expected performance as in the first condition. Finally, feedforward reaches with variable mapping yielded large errors showing visual guidance in the previous condition was successful despite an ineffective internal model. We simulated reaches using a proportional rate model with disparity Tau controlling the virtual Equilibrium Point in an Equilibrium Point (EP) model. The time dimensioned information and dynamic remained stable with delayed feedback. Finally, we fit movement times using the proportional rate EP model with 0msec, 50msec, and 100msec delay. With the fitted model parameters, we compared the model reach trajectories with the behavioral trajectories. Stable visually guided reaching did not require an internal feedforward model.

Characterisation of visual guidance of steering to intercept targets following curving trajectories using Qualitative Inconsistency Detection

This study explored the informational variables guiding steering behaviour in a locomotor interception task with targets moving along circular trajectories. Using a new method of analysis focussing on the temporal co-evolution of steering behaviour and the potential information sources driving it, we set out to invalidate reliance on plausible informational candidates. Applied to individual trials rather than ensemble averages, this Qualitative Inconsistency Detection (QuID) method revealed that steering behaviour was not compatible with reliance on information grounded in any type of change in the agent-centred target-heading angle. First-order changes in the environment-centred target’s bearing angle could also not adequately account for the variations in behaviour observed under the different experimental conditions. Capturing the observed timing of unfolding steering behaviour ultimately required a combination of (velocity-based) first-order and (acceleration-based) second-order changes in bearing angle. While this result may point to reliance on fractional-order based changes in bearing angle, the overall importance of the present findings resides in the demonstration of the necessity to break away from the existing practice of trying to fit behaviour into a priori postulated functional strategies based on categorical differences between operative heuristic rules or control laws.

Development of a force-field-based control strategy for an upper-limb rehabilitation robot

Abstract. Robot-assisted rehabilitation has proven to be effective for improving the motor performance of patients with neuromuscular injuries. The effectiveness of robot-assisted training directly depends on the control strategy applied in the therapy training. This paper presents an end-effector upper-limb rehabilitation robot for the functional recovery training of disabled patients. A force-field-based rehabilitation control strategy is then developed to induce active patient participation during training tasks. The proposed control strategy divides the 3D space around the rehabilitation training path into a human-dominated area and a robot-dominated area. It encodes the space around the training path and endows the corresponding normal and tangential force; the tangential component assists with movement along the target path, and the normal component pushes the patient's hand towards the target path using a real-time adjustable controller. Compared with a common force-field controller, the human–robot interaction in this strategy is easy and can be quickly adjusted by changing the force field's range or the variation characteristics of two forces, and the intervention in two directions can change continuously and smoothly despite the patient's hand crossing the two areas. Visual guidance based on the Unity-3D environment is introduced to provide visual training instructions. Finally, the feasibility of the proposed control scheme is validated via training experiments using five healthy subjects.

Investigating features in augmented visual feedback for virtual reality rehabilitation of upper-extremity function through isometric muscle control

Previous studies have demonstrated how augmented feedback can accelerate motor learning. Still, how specific feedback features of complexity and intermittency can influence learning a challenging, force-driven motor task remains largely unknown. This study is an initial investigation of how variations in the complexity and intermittency of augmented visual guidance affect the performance of an isometric muscle control task with a computerized platform. This novel platform has been developed to rehabilitate upper-extremity function after neuromuscular dysfunction (e.g., spinal cord injury, stroke) while utilizing: 1) a position-adjustable arm brace for gravity support; 2) a myoelectric command interface; 3) virtual reality (VR) for motor training. Results from this study elucidate new motor control principles and suggest how augmented guidance may be leveraged in designing VR motor rehabilitation programs, which are highly flexible and customizable to individual users. This study demonstrated that simpler and more intermittent feedback typically resulted in better performance (i.e., shorter computerized motion pathlengths). Supplementary results suggested these feedback modes also reduced cognitive loading (i.e., alpha/beta band magnitudes in electroencephalography) but increased physical arousal (i.e., higher skin conductance). In sum, this study indicates that for complex, force-driven tasks, augmented guidance must be presented selectively to accelerate gains in motor performance. This study suggests that simple and intermittent feedback avoids cognitively overwhelming the user while encouraging physical engagement that supports better performance.

Design, Sensing, and Control of a Magnetic Compliant Continuum Manipulator

Continuum manipulators coupled with magnetic actuation have great potential as steerable instruments for diverse surgical applications. They can be maneuvered inside the human body to reach difficult-to-access surgical sites with contactless actuation. This paper presents a new design of a compliant continuum manipulator of diameter 3 mm and length 70 mm, capable of spatial bending under magnetic actuation. A quasi-static model is developed to estimate the 3D motion of the manipulator. Experiments report an overall mean error in whole shape estimation of the manipulator between the model and the ground truth of 1.7 mm and 4.8 mm, when suspended vertically and horizontally from its base, respectively. Furthermore, fiber Bragg grating (FBG) sensors are integrated with the manipulator to enable shape sensing. Closed-loop control is demonstrated to trace different trajectories with the tip of the manipulator. A square trajectory and a straight line trajectory are generated with an average error in tip position of 4.1 mm between the desired and estimated positions. The potential of the manipulator as a steerable instrument is validated by maneuvering it inside phantoms of a bifurcating arterial system and a heart with visual guidance from a miniature camera.

Integrated approach to AUV docking based on nonlinear offset-free model predictive control

This study proposes an integrated docking approach for autonomous underwater vehicle (AUV), which directly generate control force and torque from visual guidance features to drive precise AUV docking. Accurate docking control has been constructed as nonlinear offset-free model predictive control (OFNMPC) problem to effectively deal with external disturbances, model mismatch, and various other constraints. For the problem of state and disturbance estimation required by OFNMPC, observers have been presented based on the extended Kalman filter (EKF) and moving horizon estimation (MHE). A visual observation model is proposed so that the observer can directly use visual measurements. OFNMPC was integrated with EKF and MHE, and the proposed OFNMPC-EKF and OFNMPC-MHE docking approaches were applied to an actual AUV model. The simulation results show that the proposed approaches could drive the AUV to complete accurate docking along a reasonable trajectory based on visual measurements. The navigation, guidance, and control functions of docking were effectively integrated. It was observed that in the presence of external disturbances and measurement noise, OFNMPC-EKF has better robustness.

Exploring the Visual Guidance of Motor Imagery in Sustainable Brain–Computer Interfaces

Motor imagery brain–computer interface (MI-BCI) systems hold the possibility of restoring motor function and also offer the possibility of sustainable autonomous living for individuals with various motor and sensory impairments. When utilizing the MI-BCI, the user’s performance impacts the system’s overall accuracy, and concentrating on the user’s mental load enables a better evaluation of the system’s overall performance. The impacts of various levels of abstraction on visual guidance of mental training in motor imagery (MI) may be comprehended. We proposed hypotheses about the effects of visually guided abstraction on brain activity, mental load, and MI-BCI performance, then used the event-related desynchronization (ERD) value to measure the user’s brain activity, extracted the brain power spectral density (PSD) to measure the brain load, and finally classified the left- and right-handed MI through a support vector machine (SVM) classifier. The results showed that visual guidance with a low level of abstraction could help users to achieve the highest brain activity and the lowest mental load, and the highest accuracy rate of MI classification was 97.14%. The findings imply that to improve brain–computer interaction and enable those less capable to regain their mobility, visual guidance with a low level of abstraction should be employed when training brain–computer interface users. We anticipate that the results of this study will have considerable implications for human-computer interaction research in BCI.

Implementation of Data-Driven Timelines and Visual Guidance Tools to Improve Radiation Oncology Task Completion: “Plan Readiness”

<h3>Purpose/Objective(s)</h3> The time between physician treatment plan approval and patient treatment initiation is often too short in our current Radiation Oncology practice. This reflects inconsistencies in the timeframe of task completion between CT simulation and treatment plan approval, the first and last steps in a complex, multi-step planning workflow in Radiation Oncology. This poses challenges in terms of the QA and plan checks needed prior to treatment initiation, with potential implications for patient safety. Our goal was to implement a sustainable and clear departmental process that would increase the percentage of plans approved by the attending Radiation Oncologist 1 full business day prior to treatment start for all non-urgent patients. Secondary goals included a reduction in last-minute patient delays and improvement in team member engagement and satisfaction. <h3>Materials/Methods</h3> We established realistic timelines for each task from simulation to treatment initiation, while setting a cutoff time for implementing patient appointment delay when expectations were not met. Treatment site, technique, and complexity-specific timelines were built by our team based on 4 major factors: 1) Clinical Urgency, 2) Task Complexity, 3) Evaluation of "task completion" data from a large cohort of previously treated patients, and 4) Interviews with all team members involved in the treatment planning process. These timelines were generated in a visual task management tool, and were linked as serial or parallel to keep the entire team on schedule for each patient. Prior to implementation, appropriate departmental meetings were held to discuss these changes, with extensive team member feedback utilized to generate the new policy. Timelines were generated that ranged from 5 to 12 working days, with clear goals and guidelines for each involved team member regarding their task(s). Our overarching goal, termed "Plan Readiness", was to achieve an electronically approved treatment plan 1 full working day prior to treatment start for > 90% of our patients. Appropriate exclusions were made for emergencies and special/new procedures in the department. <h3>Results</h3> For the 5 months prior to implementation, our baseline data demonstrated that 71% of plans were electronically approved in our treatment planning system greater than 24 hours prior to treatment start. After implementation of our new "Plan Readiness" process in March 2021, we have been able to stay above 90% pass rate, with a cumulative average of 94.6% over an ongoing 10-month timeframe. <h3>Conclusion</h3> The design and execution of data-driven timelines and visual guidance tools, combined with a team-based approach of communication and open feedback, has resulted in preliminary favorable outcomes in achieving timely "Plan Readiness" in our relatively large, hospital-based Department of Radiation Oncology. Future directions include the use of analytics software and automated data collection to perform more robust assessments.

Digital single-operator cholangioscopy to guide selective cannulation of complex biliary strictures

BackgroundSelective cannulation and stenting of complex, tight, and/or angulated biliary strictures under endoscopic retrograde cholangiopancreaticography (ERCP) can be challenging. Digital single-operator cholangioscopy (SOC) may facilitate guidewire advancement through the stricture with endoscopic visual guidance. We aimed to describe a case series on clinical outcomes of this technique for selective cannulation, when used after failed conventional ERCP attempts.MethodsConsecutive patients who underwent therapeutic digital SOC for selective cannulation of biliary strictures after failed conventional ERCP were retrospectively included.ResultsTen patients with a malignant (n = 6) or benign (n = 4) biliary stricture were included. Digital SOC-assisted selective guidewire insertion and stent placement across the biliary stricture were technically successful in five (50%) patients. Bilirubin levels improved in all patients with technical success. One (10%) patient developed a post-ERCP pancreatitis.ConclusionsTechnically successful cannulation of biliary strictures with digital SOC was achieved in half of patients in whom cannulation with conventional ERCP failed, sparing them more invasive interventions. Stricture opacification during the failed ERCP was seen in all technically successful SOCs.

Deep scene-scale material estimation from multi-view indoor captures

The movie and video game industries have adopted photogrammetry as a way to create digital 3D assets from multiple photographs of a real-world scene. But photogrammetry algorithms typically output an RGB texture atlas of the scene that only serves as visual guidance for skilled artists to create material maps suitable for physically-based rendering. We present a learning-based approach that automatically produces digital assets ready for physically-based rendering, by estimating approximate material maps from multi-view captures of indoor scenes that are used with retopologized geometry. We base our approach on a material estimation Convolutional Neural Network (CNN) that we execute on each input image. We leverage the view-dependent visual cues provided by the multiple observations of the scene by gathering, for each pixel of a given image, the color of the corresponding point in other images. This image-space CNN provides us with an ensemble of predictions, which we merge in texture space as the last step of our approach. Our results demonstrate that the recovered assets can be directly used for physically-based rendering and editing of real indoor scenes from any viewpoint and novel lighting. Our method generates approximate material maps in a fraction of time compared to the closest previous solutions.

Development and feasibility evaluation of an AR-assisted radiotherapy positioning system.

PurposeThe aim of this study is to develop an augmented reality (AR)–assisted radiotherapy positioning system based on HoloLens 2 and to evaluate the feasibility and accuracy of this method in the clinical environment.MethodsThe obtained simulated computed tomography (CT) images of an “ISO cube”, a cube phantom, and an anthropomorphic phantom were reconstructed into three-dimensional models and imported into the HoloLens 2. On the basis of the Vuforia marker attached to the “ISO cube” placed at the isocentric position of the linear accelerator, the correlation between the virtual and real space was established. First, the optimal conditions to minimize the deviation between virtual and real objects were explored under different conditions with a cube phantom. Then, the anthropomorphic phantom–based positioning was tested under the optimal conditions, and the positioning errors were evaluated with cone-beam CT.ResultsUnder the normal light intensity, the registration and tracking angles are 0°, the distance is 40 cm, and the deviation reached a minimum of 1.4 ± 0.3 mm. The program would not run without light. The hologram drift caused by the light change, camera occlusion, and head movement were 0.9 ± 0.7 mm, 1.0 ± 0.6 mm, and 1.5 ± 0.9 mm, respectively. The anthropomorphic phantom–based positioning errors were 3.1 ± 1.9 mm, 2.4 ± 2.5 mm, and 4.6 ± 2.8 mm in the X (lateral), Y (vertical), and Z (longitudinal) axes, respectively, and the angle deviation of Rtn was 0.26 ± 0.14°.ConclusionThe AR-assisted radiotherapy positioning based on HoloLens 2 is a feasible method with certain advantages, such as intuitive visual guidance, radiation-free position verification, and intelligent interaction. Hardware and software upgrades are expected to further improve accuracy and meet clinicalbrendaannmae requirements.

Probing an intelligent predictive maintenance approach with deep learning and augmented reality for machine tools in IoT-enabled manufacturing

• We solved the fault prediction problem with the combination of CNN and LSTM . • We solved the maintenance decision problem with the deep reinforcement learning. • The enhanced visual guidance of the maintenance process was realized by AR , which could integrate the invisible maintenance information into the machine tools. • The proposed predictive maintenance approach outperforms others in terms of accuracy and effectiveness. In the Industry 4.0 era, the number and complexity of machine tools are both increased, which is prone to cause malfunctions and downtime in the manufacturing process. Predictive Maintenance (PdM), as a pivotal part of Prognostics and Health Management (PHM), plays a vital role in enhancing the reliability of machine tools in the Internet of Things (IoT)-enabled manufacturing. In order to realize a highly reliable maintenance plan integrated with the fault prediction, the maintenance decision-making, and the Augmented Reality (AR)-enabled auxiliary maintenance, an intelligent predictive maintenance approach for machine tools is proposed in this paper via multiple services cooperating within a single framework. The fault prediction service is supported by the combination of Convolutional Neural Network and Long Short-Term Memory (CNN-LSTM). Specified features from massive production data acquired by IoT can be comprehensively extracted using CNN, and their nonlinear relationship can be fitted by LSTM. Based on the fault prediction result, deep reinforcement learning is adopted to achieve the production control and schedule maintenance personnel if a fault code appears. On top of this, the guidance information from the maintenance experience database can be integrated into the faulty machine tools in the form of visibility through AR, which can guide the maintenance personnel to complete maintenance tasks more efficiently. Moreover, the remote expert service is also integrated in the AR-supported auxiliary maintenance, which is activated to solve unexpected faults that are not stored in the maintenance experience database. Comparative experiments are conducted in the IoT-enabled manufacturing workshop with real-world case studies, and the results demonstrate that the proposed predictive maintenance approach is both effective and practical.