Focus on the experts: Co-designing an augmented reality eye-gaze tracking system with surgical trainees to improve endoscopic instruction

Several decades of eye related research has shown how valuable eye gaze data are for applications that are essential to human daily life. Eye gaze data in a broad sense has been used in research and systems for eye movements, eye tracking, and eye gaze tracking. Since early 2000, eye gaze tracking systems have emerged as interactive gaze-based systems that could be remotely deployed and operated, known as remote eye gaze tracking (REGT) systems. The drop point of visual attention known as point of gaze (PoG), and the direction of visual attention known as line of sight (LoS), are important tasks of REGT systems. In this paper, we present a comparative evaluation of REGT systems intended for the PoG and LoS estimation tasks regarding past to recent progress. Our literature evaluation presents promising insights on key concepts and changes recorded over time in hardware setup, software process, application, and deployment of REGT systems. In addition, we present current issues in REGT research for future attempts.

Solar energy and wind energy are two main renewable energy sources. Concentration of solar energy onto a smaller area may reduce the cost of PV modules and achieve high efficiency. On the other hand, wind augmentation allows lower trigger wind speed besides the reduced wind turbine size. For both solar concentration and wind augmentation, their tracking and support system may be the most costly factor. It will therefore be attractive to extend the function of a solar parabolic concentrator to encompass wind augmentation, thereby maximizing the operation time and cost-effectiveness of the costly tracking system. This study presents a novel combined solar concentration/wind augmentation (CSCWA) system. The CSCWA system has the advantage of operating in the daytime and night time whenever it is sunny or windy. The feasibility of using a dish parabolic concentrator for wind augmentation is investigated in CFD simulation and wind tunnel model testing. The increase in monthly power output of the CSCWA system against the independent non-augmented wind turbine and concentrating PV systems is also discussed to identify the suitable climatic conditions. Copyright The Author 2010. Published by Oxford University Press. All rights reserved. For Permissions, please email: journals.permissions@oup.com, Oxford University Press.

Real-time eye detection is important for many HCI applications, including eye gaze tracking, auto stereoscopic displays, video conferencing, face detection, and recognition. Currently, the eye gaze tracking systems are usually implemented on general purpose processors. As eye gaze tracking algorithms move from research labs to the real world, power consumption and cost become critical issues. This motivates searching for implementations using a digital signal processor (DSP). Our goal in this paper is to explore the feasibility of implementing DSP-based eye gaze tracking systems. To achieve this goal, we implement a fully automatic eye gaze tracking system on Texas Instruments’ TMS320DM642 DSP, profile performance, and analyze opportunities for optimization. Our experiments demonstrate that well-optimized eye gaze tracking implementations on DSP can be an effective choice for embedded eye gaze tracking products.

The central aim of this study is to develop an adaptive real-time eye-gaze tracking (EGT) system that serves as an assistive tool for persons with motor challenges access computers with optimal practicality. The novelty of the proposed method is that it adapts to the different and changing jitter characteristics of each specific user, through the configuration and training of an artificial neural network (ANN). A profile, generated for each user by the ANN through a one-time short training session, comes to reinforce the stability of mouse-cursor movements. The user profile, which can be fine-tuned with additional training, and the methods for training are embedded in the proposed system. The results using 9 subjects show an average jitter reduction of 36% in test 1 which is to follow a moving target, and 53% for following the contour of a square, which resulted in eye-gaze displacements that are significantly smoother.

This article develops a cooperative-critic learning-based secure tracking control (CLSTC) method for unknown nonlinear systems in the presence of multisensor faults. By introducing a low-pass filter, the sensor faults are transformed into "pseudo" actuator faults, and an augmented system that integrates the system state and the filter output is constructed. To reduce design costs, a joint neural network Luenberger observer (NNLO) structure is established by using neural network and input/output data of the system to identify unknown system dynamics and sensor faults online. To achieve the optimal secure tracking control, an augmented tracking system is formed by integrating the dynamics of tracking error, reference trajectory, and filter output. Then, a novel cost function is designed for the augmented tracking system, which employs the fault estimation and the discount factor. The Hamilton-Jacobi-Bellman equation is solved to obtain the CLSTC strategy through an adaptive critic structure with cooperative tuning laws. Besides, the Lyapunov stability theorem is utilized to prove that all signals of the closed-loop system converge to a small neighborhood of the equilibrium point. Simulation results demonstrate that the proposed control method has good fault tolerance performance and is suitable for solving secure control problems of nonlinear systems with various sensor faults.

The previous chapter compared gaze controlled and finger tracking based pointing modalities and we found users found it difficult to home on target using the gaze controlled interface and even with finger tracking system in automotive environment. In this chapter, we have proposed an algorithm that can activate a target even before the pointer reaches on top of it. For a gaze controlled interface, the target will be activated as soon as the saccade launches near the target reducing the fixation duration required to activate a target. In the latter half of the chapter we discussed different fusion strategies to combine eye gaze and finger tracking systems together. The previous chapter already introduced multimodal eye gaze tracking system by combining eye gaze tracking with joystick and LeapMotion controller, this chapter takes forward the concept with more sophisticated fusion models.

While Eye Gaze Tracking (EGT) systems have demonstrated their potential as computer cursor control devices, their application outside the laboratory environment has been less prominent than originally expected. This may be due to limitations in the stability of the cursor controlled by EGT devices and in the potential for unintended selections ("clicks"). To address these EGT limitations, we have developed a cursor control system that integrates electromyogram-based control with EGT-based control. This integrated system allows a user to perform small cursor displacements and selection operations ("left click") by detecting distinct changes in electromyogram (EMG) signals generated during specific facial movements. Specifically, the system monitors and processes, in real time, EMG signals from four electrodes attached to the head of the subject. We compared the performance of this EMG/EGT integrated system, with the performance of a stand-alone EGT system and a hand-held mouse, in point-and-click trials. Statistical analysis of the results indicates that the EMG/EGT system is slower than the EGT system and the mouse, but it also revealed that it is significantly less error-prone than the EGT-only system. The analysis also revealed that the performance of the EMG/EGT system does not seem to be well matched to Fitts' model.

The paper proposes the mosaic sustainable marker detection method for augmented reality markers. The existing markers of augmented reality systems have been analyzed. The advantages and disadvantages of existing markers of augmented reality systems have been analyzed. The mosaic sustainable marker detection method for augmented reality systems has been developed. The block diagram of mosaic sustainable marker detection method for augmented reality systems is presented. The stages of the mosaic sustainable marker detection method for augmented reality systems are considered. Experimental studies of the mosaic sustainable marker detection method for augmented reality systems has been carried out.

The traditional navigation system used in spinal puncture is not able to monitor the surgical process in real time and the accuracy of navigation is unsatisfactory. In this study, an augmented reality surgical navigation system based on multi-view virtual and real registration is proposed to solve these problems. The theory of virtual and real registration in augmented reality technology is analyzed, and the methods of single-view and multi-view virtual and real registration are compared. The principle of coordinate transformation in the surgical navigation module is analyzed. The platform of augmented reality surgical navigation system based on multi-view virtual and real registration technology is designed. The experiments of the spinal model are used to verify the accuracy of virtual and real registration. The accuracy of the proposed navigation system is verified by the experiment of simulating puncture operation with robotic control of surgical tool. The experimental results show that the accuracy of single-view and multi-view virtual and real registration is 9.85±0.80mm and 1.62±0.22mm respectively. The accuracy of the augmented reality surgical navigation system added with multi-view virtual and real registration technology is 1.70±0.25mm, which is 35% higher than that of the augmented reality surgical navigation system not previously used. The augmented reality surgical navigation system for spinal puncture based on multi-view virtual and real registration proposed in this study can meet the requirements of physician on the accuracy of surgery. It can also help physician to monitor the surgical process in real time and improve the success rate of surgery.

The COVID-19 pandemic has accelerated the growing global interest in the role of augmented and virtual reality in surgical training. While this technology grows at a rapid rate, its efficacy remains unclear. To that end, we offer a systematic review of the literature summarizing the role of virtual and augmented reality on spine surgery training. A systematic review of the literature was conducted on May 13th, 2022. PubMed, Web of Science, Medline, and Embase were reviewed for relevant studies. Studies from both orthopedic and neurosurgical spine programs were considered. There were no restrictions placed on the type of study, virtual/augmented reality modality, nor type of procedure. Qualitative data analysis was performed, and all studies were assigned a Medical Education Research Study Quality Instrument (MERSQI) score. The initial review identified 6752 studies, of which 16 were deemed relevant and included in the final review, examining a total of nine unique augmented/virtual reality systems. These studies had a moderate methodological quality with a MERSQI score of 12.1 + 1.8; most studies were conducted at single-center institutions, and unclear response rates. Statistical pooling of the data was limited by the heterogeneity of the study designs. This review examined the applications of augmented and virtual reality systems for training residents in various spine procedures. As this technology continues to advance, higher-quality, multi-center, and long-term studies are required to further the adaptation of VR/AR technologies in spine surgery training programs.

25 - Annotating Real-World Objects Using Augmented Reality

Augmented Reality (AR) is a novel interaction way in many fields and applications, and it provides more direct and sensory operation mode for human-machine interaction.Virtual training is becoming an effective approach for knowledge transfers in various fields including agriculture.Augmented Reality can promote virtual training in agriculture.In this paper, we presented an interactive virtual training method and developed a system based on Augmented Reality for agriculture.By use of D'fusion framework-a developing platform based on Augmented Reality, we implemented the presented algorithm and combined relevant agricultural training knowledge.The system gave interactive operations like games with an example of strawberry.The virtual models of strawberry were constructed by real three-dimensional data to ensure the realistic training results.The system provided a convenient interactive training-learning software tool.Finally, the future works in deeply applications are discussed.

For the problems the hardware configuration of multi-camera and multi-light source is complicated and the eye image of little auxiliary light source is too dark in the eye gaze tracking system, A new eye gaze tracking system based on two-dimensional mapping model is proposed, which achieved the transformation from two-dimensional coordinates to three-dimensional coordinates combined attitude angle of virtual reality headset. In this paper, the mapping relationship between visual features and eye gaze points is established by improving the method of pupil image recognition and eye gaze estimation based pupil-corneal, which the accuracy of eye gaze estimation is further improved. The experimental results show that the error of eye gaze tracking system is less than 1.1 °, which achieved the good effect of eye gaze tracking.

BACKGROUND: Augmented reality is a potential alternative for surgical navigation, as it can provide visualization of various anatomical structures of the maxillofacial region. We have developed a specific platform for three-dimensional model management and intraoperative augmented reality navigation for surgical procedures. AIM: To evaluate surgeons’ perceptions and the usability of augmented reality navigation for the most common surgical procedures in oral surgery. MATERIALS AND METHODS: We performed a phantom study to determine the accuracy of the augmented reality system in the maxillofacial region. Registration errors typical in image-guided surgery were measured: fiducial registration error, target registration error, and fiducial localization error. Thereafter, a clinical trial with several surgeons performing surgical procedures on jaws using augmented reality technology was conducted. Surgeons filled out a dedicated questionnaire after their experience with augmented reality navigation. RESULTS: The mean fiducial registration error was 0.9 mm (standard deviation 0.7 mm; 95% confidence interval 0.4–1.3 mm). The mean target registration error was 1.3 mm (standard deviation 0.5 mm; 95% confidence interval 1.1–1.5 mm). The fiducial localization error was the most significant with 2.2 mm (standard deviation 0.9; 95% confidence interval 1.9–2.5 mm). The higher rankings in the user experience questionnaire were related to the novelty and excitement of using augmented reality navigation in maxillofacial surgery. The pragmatic quality aspect explains the technical focus of perception to achieve goals in a product, system, or service design. Efficiency was expected to be slightly higher; however, in our opinion, this is due to the technical difficulties of the system for novel augmented reality technology. CONCLUSION: The results revealed the satisfactory accuracy of the augmented reality system in the maxillofacial region and the user experience of the augmented reality navigation system for oral surgery.

Computer-assisted systems obtained an increased interest in orthopaedic surgery over the last years, as they enhance precision compared to conventional hardware. The expansion of computer assistance is evolving with the employment of augmented reality. Yet, the accuracy of augmented reality navigation systems has not been determined. To examine the accuracy of component alignment and restoration of the affected limb's mechanical axis in primary total knee arthroplasty (TKA), utilizing an augmented reality navigation system and to assess whether such systems are conspicuously fruitful for an accomplished knee surgeon. From May 2021 to December 2021, 30 patients, 25 women and five men, underwent a primary unilateral TKA. Revision cases were excluded. A preoperative radiographic procedure was performed to evaluate the limb's axial alignment. All patients were operated on by the same team, without a tourniquet, utilizing three distinct prostheses with the assistance of the Knee+™ augmented reality navigation system in every operation. Postoperatively, the same radiographic exam protocol was executed to evaluate the implants' position, orientation and coronal plane alignment. We recorded measurements in 3 stages regarding femoral varus and flexion, tibial varus and posterior slope. Firstly, the expected values from the Augmented Reality system were documented. Then we calculated the same values after each cut and finally, the same measurements were recorded radiologically after the operations. Concerning statistical analysis, Lin's concordance correlation coefficient was estimated, while Wilcoxon Signed Rank Test was performed when needed. A statistically significant difference was observed regarding mean expected values and radiographic measurements for femoral flexion measurements only (Z score = 2.67, P value = 0.01). Nonetheless, this difference was statistically significantly lower than 1 degree (Z score = -4.21, P value < 0.01). In terms of discrepancies in the calculations of expected values and controlled measurements, a statistically significant difference between tibial varus values was detected (Z score = -2.33, P value = 0.02), which was also statistically significantly lower than 1 degree (Z score = -4.99, P value < 0.01). The results indicate satisfactory postoperative coronal alignment without outliers across all three different implants utilized. Augmented reality navigation systems can bolster orthopaedic surgeons' accuracy in achieving precise axial alignment. However, further research is required to further evaluate their efficacy and potential.