Evaluating the Perceptual Impact of Virtual and Physical Lighting on Medical Volume Rendering in AR

We introduce virtual blue noise lighting, a rendering pipeline for estimating indirect illumination with a blue noise distribution of virtual lights. Our pipeline is designed for virtual lights with non-uniform emission profiles that are more expensive to store, but required for properly and efficiently handling specular transport. Unlike the typical virtual light placement approaches that traverse light paths from the original light sources, we generate them starting from the camera. This avoids two important problems: wasted memory and computation with fully-occluded virtual lights, and excessive virtual light density around high-probability light paths. In addition, we introduce a parallel and adaptive sample elimination strategy to achieve a blue noise distribution of virtual lights with varying density. This addresses the third problem of virtual light placement by ensuring that they are not placed too close to each other, providing better coverage of the (indirectly) visible surfaces and further improving the quality of the final lighting estimation. For computing the virtual light emission profiles, we present a photon splitting technique that allows efficiently using a large number of photons, as it does not require storing them. During lighting estimation, our method allows using both global power-based and local BSDF important sampling techniques, combined via multiple importance sampling. In addition, we present an adaptive path extension method that avoids sampling nearby virtual lights for reducing the estimation error. We show that our method significantly outperforms path tracing and prior work in virtual lights in terms of both performance and image quality, producing a fast but biased estimate of global illumination.

This paper focuses on how virtual objects' shadows as well as differences in alignment between virtual and real lighting influence distance perception in optical see‐through (OST) augmented reality (AR). Four hypotheses are proposed: (H1) Participants underestimate distances in OST AR; (H2) Virtual objects' shadows improve distance judgment accuracy in OST AR; (H3) Shadows with different realism levels have different influence on distance perception in OST AR; (H4) Different levels of lighting misalignment between real and virtual lights have different influence on distance perception in OST AR scenes. Two experiments were designed with an OST head mounted display (HMD), the Microsoft HoloLens. Participants had to match the position of a virtual object displayed in the OST‐HMD with a real target. Distance judgment accuracy was recorded under the different shadows and lighting conditions. The results validate hypotheses H2 and H4 but surprisingly showed no impact of the shape of virtual shadows on distance judgment accuracy thus rejecting hypothesis H3. Regarding hypothesis H1, we detected a trend toward underestimation; given the high variance of the data, more experiments are needed to confirm this result. Moreover, the study also reveals that perceived distance errors and completion time of trials increase along with targets' distance.

A comprehensive understanding of microsurgical neuroanatomy, familiarity with the operating room environment, patient positioning in relation to the surgery, and knowledge of surgical approaches is crucial in neurosurgical education. However, challenges such as limited patient exposure, heightened patient safety concerns, a decreased availability of surgical cases during training, and difficulties in accessing cadavers and laboratories have adversely impacted this education. Three-dimensional (3D) models and augmented reality (AR) applications can be utilized to depict the cortical and white matter anatomy of the brain, create virtual models of patient surgical positions, and simulate the operating room and neuroanatomy laboratory environment. Herein, the authors, who used a single application, aimed to demonstrate the creation of 3D models of anatomical cadaver dissections, surgical approaches, patient surgical positions, and operating room and laboratory designs as alternative educational materials for neurosurgical training. A 3D modeling application (Scaniverse) was employed to generate 3D models of cadaveric brain specimens and surgical approaches using photogrammetry. It was also used to create virtual representations of the operating room and laboratory environment, as well as the surgical positions of patients, by utilizing light detection and ranging (LiDAR) sensor technology for accurate spatial mapping. These virtual models were then presented in AR for educational purposes. Virtual representations in three dimensions were created to depict cadaver specimens, surgical approaches, patient surgical positions, and the operating room and laboratory environment. These models offer the flexibility of rotation and movement in various planes for improved visualization and understanding. The operating room and laboratory environment were rendered in three dimensions to create a simulation that could be navigated using AR and mixed reality technology. Realistic cadaveric models with intricate details were showcased on internet-based platforms and AR platforms for enhanced visualization and learning. The utilization of this cost-effective, straightforward, and readily available approach to generate 3D models has the potential to enhance neuroanatomical and neurosurgical education. These digital models can be easily stored and shared via the internet, making them accessible to neurosurgeons worldwide for educational purposes.

This study is designed to determine whether differences in eye fatigue and visual performance can be shown under varying virtual industrial lighting conditions. It is based on the results of studies of more traditional video display terminal (VDT) tasks reported in the literature. One experiment was designed to determine if the effects of virtual lighting on eye fatigue and visual performance in a simulated virtual industrial environment are similar to some other VDT tasks with varying luminance contrast. Results of a test of 20 participants in a vigilance task show that there is a significant difference in performance and eye fatigue in the virtual environment with varying virtual light conditions. These results may help designers see that performance in some virtual “lighting” conditions, for some tasks, is consistent with that in the real. However, due to the difficulties of determining the appropriate virtual objects to be considered for the luminance measures, additional research is needed to be able to generalize the results to other industrial training scenarios. A second experiment was intended to test for the luminance decrement in a VDT that was shown in recent literature. The results would have potential implications for the experiment that included a vigilance task. However, the results showed that the luminance decrement demonstrated in recent literature did not occur. These results suggest that the equipment used in the present experiments should not cause difficulty in interpreting the results of the vigilance task. © 2002 Wiley Periodicals, Inc.

Computer augmented reality (CAR) is a rapidly emerging field which enables users to mix real and virtual worlds. Our goal is to provide interactive tools to perform common illumination, i.e., light interactions between real and virtual objects, including shadows and relighting (real and virtual light source modification). In particular, we concentrate on virtually modifying real light source intensities and inserting virtual lights and objects into a real scene; such changes can be very useful for virtual lighting design and prototyping. To achieve this, we present a three-step method. We first reconstruct a simplified representation of real scene geometry using semiautomatic vision-based techniques. With the simplified geometry, and by adapting recent hierarchical radiosity algorithms, we construct an approximation of real scene light exchanges. We next perform a preprocessing step, based on the radiosity system, to create unoccluded illumination textures. These replace the original scene textures which contained real light effects such as shadows from real lights. This texture is then modulated by a ratio of the radiosity (which can be changed) over a display factor which corresponds to the radiosity for which occlusion has been ignored. Since our goal is to achieve a convincing relighting effect, rather than an accurate solution, we present a heuristic correction process which results in visually plausible renderings. Finally, we perform an interactive process to compute new illumination with modified real and virtual light intensities.

The Operating Room and Learning Environment for US-Based Muslim Women in Medicine.

In the field of surgery, major changes that have occurred include the advent of minimally invasive surgery and the realization of the importance of the ‘systems’ in the surgical care of the patient (Pierorazio & Allaf, 2009). Challenges in surgical training are two-fold: (i) to train the surgical residents to manage a patient clinically (ii) to train them in operative skills (Singh & Darzi,2013). In Pakistan, another issue with surgical training is
 that we have the shortest duration of surgical training in general surgery of four years only, compared to six to eight years in Europe and America (Zafar & Rana, 2013). Along with it, the smaller number of patients to surgical residents’ ratio is also an issue in surgical training. This warrants formal training outside the operation room. It has been reported by many authors that changes are required in the current surgical training system due to the significant deficiencies in the graduating surgeon (Carlsen et al., 2014; Jarman et al., 2009; Parsons, Blencowe, Hollowood, & Grant, 2011). Considering surgical training, it is imperative that a surgeon is competent in clinical management and operative skills at the end of the surgical training. To achieve this outcome in this challenging scenario, a resident surgeon should be provided with the opportunities of training outside the operation theatre, before s/he can perform procedures on a real patient. The need for this training was felt more when the Institute of Medicine in the USA published a report, ‘To Err is Human’ (Stelfox, Palmisani, Scurlock, Orav, & Bates, 2006), with an aim to reduce medical errors. This is required for better training and objective assessment of the surgical residents. The options for this training include but are not limited to the use of mannequins, virtual patients, virtual simulators, virtual reality, augmented reality, and mixed reality. Simulation is a technique to substitute or add to real experiences with guided ones, often immersive in nature, that reproduce substantial aspects of the real world in a fully interactive way. Mannequins, virtual simulators are in use for a long time now. They are available in low fidelity to high fidelity mannequins and virtual simulators and help residents understand the surgical anatomy, operative site and practice their skills. Virtual patients can be discussed with students in a simple format of the text, pictures, and videos as case files available online, or in the form of customized software applications based on algorithms. In a study done by Courtielle et al, they reported that knowledge retention is increased in residents when it is delivered through virtual patients as compared to lecturing (Courteille et al., 2018).But learning the skills component requires hands-on practice. This gap can be bridged with virtual, augmented, or mixed reality. There are three types of virtual reality (VR) technologies: (i) non-immersive, (ii) semi-immersive, and (iii) fully immersive. Non-immersive (VR) involves the use of software and computers. In semi-immersive and immersive VR, the virtual image is presented through the head-mounted display(HMD), the difference being that in the fully immersive type, the virtual image is completely obscured from the actual world. Using handheld devices with haptic feedback the trainee can perform a procedure in the virtual environment (Douglas, Wilke, Gibson, Petricoin, & Liotta, 2017). Augmented reality (AR) can be divided into complete AR or mixed reality (MR). Through AR and MR, a trainee can see a
 virtual and a real-world image at the same time, making it easy for the supervisor to explain the steps of the surgery. Similar to VR, in AR and MR the user wears an HMD that shows both images. In AR, the virtual image is transparent whereas, in MR, it appears solid (Douglas et al., 2017). Virtual augmented and mixed reality has more potential to train surgeons as they provide fidelity very close to the real situation and require fewer physical resources and space compared to the simulators. But they are costlier, and affordability is an issue. To overcome this, low-cost solutions to virtual reality have been developed. It is high time that we also start thinking on the same lines and develop this means of training our surgeons at an affordable cost.

Augmented reality (AR) solely based on image features is achievable in operating room conditions and its precision is compatible with otological surgery. The objective of this work was to evaluate the performance of a vision-based AR system for middle ear surgery in the operating room conditions. Nine adult patients undergoing ossicular procedures were included in this prospective study. AR was obtained by combining real-time video from the operating microscope with the virtual image obtained from the preoperative computed tomography (CT)-scan. Initial registration between the video and the virtual CT image was achieved using manual selection of six points on the tympanic sulcus. Patient-microscope movements during the procedure were tracked using image-feature matching algorithm. The microscope was randomly moved at an approximated speed of 5 mm/s in the three axes of space and rotation for 180 seconds. The accuracy of the system was assessed by calculating the distance between each fiducial point selected on the video image and its corresponding point on the scanner. AR could be obtained for at least 3 minutes in seven out of nine patients. The overlay fiducial and target registration errors were 0.38 ± 0.23 mm (n = 7) and 0.36 ± 0.15 mm (n = 5) respectively, with a drift error of 1.2 ± 0.5 μm/s. The system was stable throughout the procedure and achieved a refresh rate of 12 fps. Moderate bleeding and introduction of surgical instruments did not compromise the performance of the system. The AR system yielded sub-millimetric accuracy and remained stable throughout the experimental study despite patient-microscope movements and field of view obtrusions.

Light source estimation and virtual lighting must be believable in terms of appearance and correctness in augmented reality scenes. As a result of illumination complexity in an outdoor scene, realistic lighting for augmented reality is still a challenging problem. In this paper, we propose a framework based on an estimation of environmental lighting from well-defined objects, specifically human faces. The method is tuned for outdoor use, and the algorithm is further enhanced to illuminate virtual objects exposed to direct sunlight. Our model can be integrated into existing mobile augmented reality frameworks to enhance visual perception.

Light source estimation and virtual lighting must be believable in terms of appearance and correctness in augmented reality scenes. As a result of illumination complexity in an outdoor scene, realistic lighting for augmented reality is still a challenging problem. In this paper, we propose a framework based on an estimation of environmental lighting from well-defined objects, specifically human faces. The method is tuned for outdoor use, and the algorithm is further enhanced to illuminate virtual objects exposed to direct sunlight. Our model can be integrated into existing mobile augmented reality frameworks to enhance visual perception.

Game theory: Applications for surgeons and the operating room environment

376 The impact of after-hours simulator practice on performance of vascular anastomosis during surgical training: A randomized trial

Digitalization in surgery is gaining attention in the surgical community, with robotics and augmented reality as key issues. The term surgical robot is basically not adequate to describe currently available telesupport and manipulation systems. These are passive tools which have to be activated by the surgeon and only provide relatively low levels of active support. Accordingly, justification of use is currently difficult with respect to the cost-benefit relationship. Areal breakthrough will be achieved by upgrading them into genuine intelligent and collaborative support systems and justify the term as the true meaning of robotics. Augmented or enriched reality improves or facilitates normal sensory perception by the integration of additional information of adifferent nature. Intuitive perception of the surgical site would have the potential to revolutionize surgery, but prior to clinical use, the matching of the real and the virtual world still has to be optimized (referencing); however, AR is now already avaluable tool for training and simulation as well as workflow support in the operating room (OR). The promising new technological development towards the future cooperative surgical OR environment, including both robotic and AR modules, will have asignificant impact on surgery, even in the mid-term. Decisive for this is that surgeons actively take part in the evaluation of this process to ensure that future "intelligent" tools will remain mere assistant or supporting systems.

PurposeThis study evaluated the influence of illuminance and color temperature of ambient light on the trueness, precision, and scanning time of a digital impression. MethodsMaster data were acquired with a high-accuracy coordinate-measuring machine. The illuminance of ambient light was set at 0lux, 500lux, and 2500lux with a light-emitting diode (LED). Using a conversion filter, the color temperature was set at 3900 Kelvin (K) (yellow), 4100K (orange), 7500K (white), and 19,000K (blue). There were thus a total of 12 possible lighting conditions. The reference model was scanned five times under each condition by an intraoral scanner. Trueness was calculated as the mean difference between the master data and experimental data. Precision was calculated as the mean difference between the repeated scans in each test group. Statistical analysis was performed with two-way analysis of variance (ANOVA) and post hoc Tukey’s multiple comparison test. The significance level was 0.05. ResultsFor trueness, the mean deviation was significantly lower at 500lux than at 0lux and 2500lux. At 500lux, the mean deviation was significantly lower at 3900K than at other temperatures. Regardless of the color temperature, the scanning time was significantly longer at 2500lux than at other illuminance levels. ConclusionsThe 3900K and 500lux condition is the most appropriate lighting condition for taking a digital impression. This condition is typical of clinical settings. High illuminance ambient light increased the scanning time.

The virtual overlay of patient-specific anatomies onto a surgical site through Augmented Reality (AR) technologies has been thought to be a potentially ideal neuronavigational system for use in neurosurgery. Although impressive and futuristic, there are many design considerations that must be taken into account, including surgeon reception, perceived utility, intuitive control and manipulation design, and overall system accuracy during surgery. To implement AR into the neurosurgical Operating Room (OR), a gradual approach of evolutionary design to ensure widespread adoption may be considered. This thesis presents a potential pathway for the introduction of AR technologies into the neurosurgical OR. The thesis is divided into three parts: incorporation of AR features into existing platforms for improved functionality and introduction of AR concepts to surgical environments, observation and evaluation of surgeon perception of AR overlays and AR headsets to inform display methods and designs, and quantification of virtual object placement accuracy in a clinical environment. The findings presented show that AR integrated systems improve OR workflow when conventional tracked tools are unavailable, user preference of AR overlays onto the surgical site change depending on operator experience level, and the placement accuracy of state-of-the-art AR head mounted displays are suitable for presurgical planning and very close to accuracy needed for surgical guidance. These three elements are key to developing a pathway for adoption of AR technologies in the OR, and help to inform designs for future headsets to assist surgeons and improve patient care.