Augmented Video Research Articles

A novel augmented reality for hidden organs visualisation in surgery: enhanced super-pixel with sub sampling and variance adaptive algorithm

In recent years, Augmented Reality (AR) has gained more attention as an effective tool in medical surgeries. The potentials of using AR in the medical field can change conventional medical procedures. However, the technology still facing fundamental challenges, especially hidden organs, for example, the organs behind the bowel and liver. The surgeries in these areas lack accuracy in the visualization of the soft tissues behind the bowel and liver like the uterus and gall bladder. This research aims to improve the accuracy of visualisation and the processing time of the augmented video. The proposed system consists of an enhanced super-pixel algorithm with variance weight adaptation and subsampling method. The simulation studies show significant improvements in visualization accuracy and a reduction in processing time. The results show reduced visualisation error by 0.23 mm. It provides better accuracy of the video in terms of visualization error from 1.58 ~ 1.83 mm to 1.35 ~ 1.60 mm, and the processing time decreases from 50 ~ 58 ms/frames to 40 ~ 48 ms/frames. The proposed system \ focused on the pixel refinement for the 3d reconstruction of the soft tissue, which helps solve the issue of visualising the bowel and liver in an augmented video.

A novel solution of using mixed reality in bowel and oral and maxillofacial surgical telepresence: 3D mean value cloning algorithm.

Most of the mixed reality models used in the surgical telepresence are suffering from the discrepancies in the boundary area and spatial-temporal inconsistency due to the illumination variation in the video frames. The aim behind this work is to propose a new solution that helps produce the composite video by merging the augmented video of the surgery site and virtual hand of the remote expertise surgeon. The purpose of the proposed solution is to decrease the processing time and enhance the accuracy of merged video by decreasing the overlay and visualization error and removing occlusion and artefacts. The proposed system enhanced mean-value cloning algorithm that helps to maintain the spatial-temporal consistency of the final composite video. The enhanced algorithm includes the three-dimensional mean-value coordinates and improvised mean-value interpolant in the image cloning process, which helps to reduce the sawtooth, smudging and discolouration artefacts around the blending region. The accuracy in terms of overlay error of the proposed solution is improved from 1.01 to 0.80mm, whereas the accuracy in terms of visualization error is improved from 98.8% to 99.4%. The processing time is reduced to 0.173s from 0.211s. The processing time and the accuracy of the proposed solution are enhanced as compared to the state-of-art solution. Our solution helps make the object of interest consistent with the light intensity of the target image by adding the space distance that helps maintain the spatial consistency in the final merged video.

Application of DenTeach in Remote Dentistry Teaching and Learning During the COVID-19 Pandemic: A Case Study.

In December 2019, an outbreak of novel coronavirus pneumonia occurred, and subsequently attracted worldwide attention when it bloomed into the COVID-19 pandemic. To limit the spread and transmission of the novel coronavirus, governments, regulatory bodies, and health authorities across the globe strongly enforced shut down of educational institutions including medical and dental schools. The adverse effects of COVID-19 on dental education have been tremendous, including difficulties in the delivery of practical courses such as restorative dentistry. As a solution to help dental schools adapt to the pandemic, we have developed a compact and portable teaching-learning platform called DenTeach. This platform is intended for remote teaching and learning pertaining to dental schools at these unprecedented times. This device can facilitate fully remote and physical-distancing-aware teaching and learning in dentistry. DenTeach platform consists of an instructor workstation (DT-Performer), a student workstation (DT-Student), advanced wireless networking technology, and cloud-based data storage and retrieval. The platform procedurally synchronizes the instructor and the student with real-time video, audio, feel, and posture (VAFP). To provide quantitative feedback to instructors and students, the DT-Student workstation quantifies key performance indices (KPIs) related to a given task to assess and improve various aspects of the dental skills of the students. DenTeach has been developed for use in teaching, shadowing, and practice modes. In the teaching mode, the device provides each student with tactile feedback by processing the data measured and/or obtained from the instructor's workstation, which helps the student enhance their dental skills while inherently learning from the instructor. In the shadowing mode, the student can download the augmented videos and start watching, feeling, and repeating the tasks before entering the practice mode. In the practice mode, students use the system to perform dental tasks and have their dental performance skills automatically evaluated in terms of KPIs such that both the student and the instructor are able to monitor student’s work. Most importantly, as DenTeach is packaged in a small portable suitcase, it can be used anywhere by connecting to the cloud-based data storage network to retrieve procedures and performance metrics. This paper also discusses the feasibility of the DenTeach device in the form of a case study. It is demonstrated that a combination of the KPIs, video views, and graphical reports in both teaching and shadowing modes effectively help the student understand which aspects of their work needs further improvement. Moreover, the results of the practice mode over 10 trials have shown significant improvement in terms of tool handling, smoothness of motion, and steadiness of the operation.

Enhancing Tourist Memories with Augmented Videography Technology

The development of Internet technology has witnessed much interpersonal communication change from paper-based into digital form. Although the advent of Internet has led to the decline of paper-¬based communication, it also brings opportunities to revive traditional forms of communication. To revive traditional postcard sharing, I present here FilmLocker, a mobile application allows users to link a digital video to customize physical postcard. This project aims to enhance user's interpersonal communication, keep their travel memories and improve emotional expression by combining permanent physical postcard with impermanent digital augmented videos.

Coarse‐to‐fine 3D road model registration for traffic video augmentation

This study addresses the problem of non-perspective pose estimation from line correspondences in the traffic scenarios. A coarse-to-fine 3D road registration method is proposed for this problem in two stages. Firstly, the iterative closest point algorithm is exploited to estimate the pose coarsely. An objective function is then established to incorporate the feature correspondences for refining the coarse pose. Besides, the framework including road registration is employed for traffic video augmentation. The framework begins with the inputs of traffic videos, road information from Geographic Information Systems and 3D models of traffic elements (e.g. vehicles, pedestrians). Subsequently, 3D road model generation and point-to-line correspondence establishment are achieved in the preprossessing stage. After road and viewpoint registration, the 3D graphic engine is employed to simulate the traffic scene with the road, viewpoints and traffic elements. The augmented videos are generated by fusing the original frames and newly projected traffic elements. The authors demonstrate the superiority of the proposed registration method by the comparison to state-of-the-arts in both quantitative and qualitative experiments. In addition, the frames of the augmented videos validate the proposed method in the application.

A Novel Solution of Using Mixed Reality in Bowel and Oral and Maxillofacial Surgical Telepresence: 3D Mean Value Cloning algorithm.

Most of the Mixed Reality models used in the surgical telepresence are suffering from the discrepancies in the boundary area and spatial-temporal inconsistency due to the illumination variation in the video frames. The aim behind this work is to propose a new solution that helps produce the composite video by merging the augmented video of the surgery site and virtual hand of the remote expertise surgeon. The purpose of the proposed solution is to decrease the processing time and enhance the accuracy of merged video by decreasing the overlay and visualization error and removing occlusion and artefacts. The proposed system enhanced the mean value cloning algorithm that helps to maintain the spatial-temporal consistency of the final composite video. The enhanced algorithm includes the 3D mean value coordinates and improvised mean value interpolant in the image cloning process, which helps to reduce the sawtooth, smudging and discoloration artefacts around the blending region RESULTS: As compared to the state of art solution, the accuracy in terms of overlay error of the proposed solution is improved from 1.01mm to 0.80mm whereas the accuracy in terms of visualization error is improved from 98.8% to 99.4%. The processing time is reduced to 0.173 seconds from 0.211 seconds CONCLUSION: Our solution helps make the object of interest consistent with the light intensity of the target image by adding the space distance that helps maintain the spatial consistency in the final merged video. This article is protected by copyright. All rights reserved.

A real time expert system for anomaly detection of aerators based on computer vision and surveillance cameras

Aerators are essential and crucial auxiliary devices in intensive culture, especially in industrial culture in China. In this paper, we propose a real-time expert system for anomaly detection of aerators based on computer vision technology and existing surveillance cameras. The expert system includes two modules, i.e., object region detection and working state detection. First, we present a small object region detection method based on the region proposal idea. Moreover, we propose a novel algorithm called reference frame Kanade-Lucas-Tomasi (RF-KLT) algorithm for motion feature extraction in fixed regions. Then, we describe a dimension reduction method of time series for establishing a feature dataset with obvious boundaries between classes. Finally, we use machine learning algorithms to build the feature classifier. The proposed expert system can realize real-time, robust and cost-free anomaly detection of aerators in both the actual video dataset and the augmented video dataset. Demo is available at https://youtu.be/xThHRwu_cnI.

A novel augmented reality to visualize the hidden organs and internal structure in surgeries.

Augmented reality (AR) is still a primarily theoretical concept in areas such as bowel, liver, gallbladder, and jaw surgeries because of the limitation of visualization accuracy of hidden organs and internal structures. This paper aims to improve the cutting accuracy, visualizing accuracy, and processing time of the augmented video. The proposed system consists of an enhanced block-matching algorithm (BMA) with ghosting map technique. Results proved that proposed system reduced the visualization error, which ranges from 1.48 to 1.83 mm against the existing system visualization error 1.67 to 2.0. Similarly, the processing time also improved 59 to 72 ms/frame over the 50 to 58 ms/frame. This study showed the improvement and solved the problem soft tissue reconstruction and visualization on the AR video that used in bowel and gallbladder surgeries.

Artificial Intelligence to Design a Mask Insensible to the Distance From the Camera to the Scene Objects

The sharpness of an image depends on the spatial frequency response of the photographic imaging system and the sensor characteristics. In conventional digital cameras, only those objects within a distance range are in focus, while other objects are captured with different amounts of blurring depending on their distance to the focal plane. This can be desired for some applications; however, this can also be undesired because some objects in the scene may be blurred and impossible to recover. In the field of augmented reality, simulating this natural effect of showing sharp objects in combination with blurred objects increases the visual realism of augmented video. In order to simulate this effect, it is very important to capture all objects in the scene with high quality so that it could be possible to dynamically blur different objects in the scene at runtime. In this paper, we present an algorithm to find a set of possible complex-amplitude transmittance masks capable of considerably reducing the impact of focus errors in the scene objects. Computer simulations are used to compare the masks found in this paper with a classic mask in the state of the art. The main contribution of this paper is the use of Chebyshev polynomials to model an optical mask, and then, use artificial intelligence to establish some properties of this mask, such as the depth of field, the resolution, and the amount of gathered light.

Quantifying Distraction Potential of Augmented Reality Head-Up Displays for Vehicle Drivers

Background: A recent National Highway Traffic & Safety Administration (NHTSA) report states that 10% of fatal crashes and 18% of injury crashes were reported as distraction-affected crashes. In that same year, 3,179 people were killed and an estimated 431,000 injured in motor vehicle crashes involving distracted drivers, many of which involved secondary visual displays (NHTSA, 2016). Augmented reality (AR) head-up displays (HUD) promise to be less distractive than traditional in-vehicle displays since they do not take drivers’ eyes off the road (Gabbard, Fitch, & Kim, 2014). However, empirical studies have reported possible negative consequences of AR HUDs, in part, due to AR graphics’ salience (Sharfi & Shinar, 2014), frequent changes (Wolffsohn, McBrien, Edgar, & Stout, 1998), and visual clutter (Burnett & Donkor, 2012). Moreover, current in-vehicle display assessment methods which are based on eye-off-road time measures (NHTSA, 2012), cannot capture this unique challenge. Objective: This work aims to propose a new method for the assessment of AR HUDs by quantifying both positive (informing drivers) and negative (distracting drivers) consequences of AR HUDs which might not be captured by current in-vehicle display assessment methods. Method: We proposed a new way of quantifying the distraction potential of AR HUDs by measuring driver situation awareness with operational improvements on the situation awareness global assessment technique (Endsley, 2012) to suit AR usability evaluations. A human-subject experiment was conducted in a driving simulator to apply the proposed method and to evaluate two AR HUD interfaces for pedestrian collision warning. The AR warning interfaces were prototyped by the augmented video technique (Soro, Rakotonirainy, Schroeter, & Wollstdter, 2014). Twenty-four participants drove while interacting with different types of AR pedestrian collision warning interfaces (no warning, bounding box, and virtual shadow). Drivers’ situation awareness, confidence, and workload were measured and compared to the no warning condition. Results: Only one of the warning interface designs, the virtual shadow (Kim, Isleib, & Gabbard, 2016), improved driver situation awareness about pedestrians which were cued by the AR HUD, not affecting situation awareness about other environmental elements which were not augmented by the HUD. The experiment also showed drivers’ overconfidence bias while interacting with the bounding box which is another warning interface design. The empirical user study did not provide any evidence for reduced driver workload when AR warnings were given. Conclusion: Our initial human-subject study demonstrated a potential of the proposed method in quantifying both positive and negative consequences of AR HUDs on driver cognitive processes. More importantly, the experiment showed that AR interfaces can have both positive and negative consequences on driver situation awareness depending upon how we design perceptual forms of graphical elements. Application: The proposed assessment methods for AR HUDs can inform not only comparative evaluation among design alternatives but also assist in incrementally improving design iterations to better support drivers’ information needs, situation awareness, and in turn, performance, and safety.

A Novel Weighted Integral Energy Functional (WIEF) Algorithm: Augmented Reality (AR) for Visualising the Blood Vessels in Breast Implant Surgeries

The use of Augmented Reality (AR) for visualising blood vessels in surgery is still at the experimental stage and has not been implemented due to limitations in terms of accuracy and processing time. The AR also hasn't applied in breast surgeries yet. As there is a need for a plastic surgeon to see the blood vessels before he cuts the breast and before putting the implant, this paper aims to improve the accuracy of augmented videos in visualising blood vessels during Breast Implant Surgery. The proposed system consists of a Weighted Integral Energy Functional (WIFE) algorithm to increase the accuracy of the augmented view in visualising the occluded blood vessels that covered by fat in the operating room. The results on breast area shows that the proposed algorithm improves video accuracy in terms of registration error to 0.32 mm and processing time to 23 sec compared to the state-of-the-art method. The proposed system focuses on increasing the accuracy in augmented view in visualising blood vessels during Breast Implant Surgery as it reduces the registration error. Thus, this study concentrates on looking at the feasibility of the use of Augmented Reality technology in Breast Augmentation surgeries.

Augmented reality system for muscle activity biofeedback

Augmented Reality (AR) has been proved successful in several applications from surgical training to balance rehabilitation. This work aims to develop an AR system for real-time visualization of an index of muscle activity superimposed to the investigated muscle. The system includes: (1) a video camera, (2) one or more surface EMG (sEMG) detection/acquisition systems, (3) one processing and visualization unit ( Fig. 1 ). The system integrates the information from the video camera and from the sEMG systems (the ARV of the sEMG signal epoch corresponding to the current video frame) and creates an “augmented video frame” by coloring the detection systems, identified within the video frame, with a color representative of the muscle activity (blue for low and red for high sEMG activity). The patient or the clinical operator can see the real-time augmented video on a display. The software can run (1) on a PC using a webcam for video capture and showing the augmented video on a monitor, (2) on a tablet using the integrated camera or (3) on the Epson Moverio BT-300 smartglasses using the see-through modality. Fig. 2 shows the AR feedback during a leg extension exercise; vastus medialis, vastus lateralis and rectus femoralis muscles are monitored using a sEMG bipolar system (Due, OTBioelettronica and LISiN). Fig. 3 shows the AR system used for the monitoring of sEMG activity distribution in the lumbar muscles using a multi-channel sEMG systems (32 channel amplifier by LISiN). An AR system for the visualization of sEMG activity over the muscles has been developed. The system is currently under validation for augmented biofeedback in sport and rehabilitation in order to verify advantages with respect to standard biofeedback.

Augmented reality for history education

Augmented Reality is live, direct or indirect view of a physical real world environment whose elements are augmented by personal computers (PC) that produces the information such as sound, video, designs or GPS data. This paper shows an instructive mobile application based system model on Augmented Reality which is used to learn subjects like history through augmented videos. The objective of development of this system model is to make the learning interesting for the young generation. Unity 3D and Vuforia Augmented Reality Software Development Kit (SDK) is used for the development of this model. The prime purpose of this application model is to enhance the learning process with digital technologies. This paper has step by step implementation instructions for the development of augmented reality modeling that can supplement the current teaching-learning environment to generate interest among young generation in less interesting subjects such as History, Geography, etc.

Mobile augmented reality applications for construction projects

PurposeThe purpose of this paper is to facilitate the process of monitoring construction projects. Classic practice for construction progress tracking relies on paper reports, which entails a serious amount of manual data collection as well as the effort of imagining the actual progress from the paperwork.Design/methodology/approachThis paper presents a new methodology for monitoring construction progress using smartphones. This is done by proposing a new system consisting of a newly-developed application named “BIM-U” and a mobile augmented reality (AR) channel named “BIM-Phase”. “BIM-U” is an Android application that allows the end-user to update the progress of activities onsite. These data are used to update the project’s 4D model enhanced with different cost parameters such as earned value, actual cost and planned value. The “BIM-Phase” application is a mobile AR channel that is used during construction phase through implementing a 4D “as-planned” phased model integrated with an augmented video showing real or planned progress.FindingsThe results from the project are then analysed and assessed to anticipate the potential of these and similar techniques for tracking time and cost on construction projects.Originality/valueThe proposed system through “BIM-U” and “BIM Phase” exploits the potential of mobile applications and AR in construction through the use of handheld mobile devices to offer new possibilities for measuring and monitoring work progress using building information modelling.

Mobile augmented communication for remote collaboration in a physical work context

Informal learning in a physical work context requires communication and collaboration that build on a common ground and an active awareness of a situation. We explored whether mobile video conversations augmented with on-screen drawing features were beneficial for improving communication and remote collaboration practices in the construction and facility maintenance services sectors. We used field studies in actual work contexts to map how participants solve physical tasks with remote help powered by augmented video calls, and examined how the drawing feature was used in these contexts. The research data were collected from interviews and job shadowing. The results suggest that augmented video calls enhance remote collaboration by allowing workers to point at task objects and locations, thus potentially improving informal workplace learning.

Introduction to the Special Section on Augmented Video

Merging computer-generated content with real-world visual data is one of the main challenges in fields like augmented reality or visual effects and is increasingly important in broadcasting, gaming, medical, automotive, maintenance, and learning applications. Although augmented reality (AR) has been investigated for a long time, it has recently emerged as a hot topic, with significant commercial interest. One reason for that is the availability of new camera-equipped devices like smart phones or tablets that enable see-through capabilities. Combined with powerful graphics capabilities, sensors, and tracking methods, AR now becomes available to everyone. In addition, glasses-based systems, like Microsoft’s HoloLens, allow for hands-free visualization, enabling many new applications.

Design of educational environments rased on interactive applications using augmented reality and video games for the active learning of archeology

Esta investigacion describe el diseno de un entorno educativo virtual mediante videojuegos de realidad aumentada denominado Cultiaventura. El entorno se ha desarrollado como prototipo de apoyo al aprendizaje activo de la Arqueologia en Educacion Primaria. La metodologia empleada se basa en la union del modelo educativo, fundamentado en competencias y el modelo de desarrollo incremental de software. Los resultados indican que el diseno ejecutado ha permitido crear recursos digitales interactivos, alineados a las rutas de aprendizaje del Ministerio de Educacion del Peru.

3D printed haptic models triggered augmented operative videos

3D printed haptic models triggered augmented operative videos

Video-based weathering gallery

We present a simple and effective method for constructing a gallery that consists of weathering effect elements called time-dependent appearance manifolds (TDAMs). Since TDAMs are computed from sample video clips showing dynamic weathering phenomena, they represent very smooth changes in the appearance of weathered pixels over time. Once a gallery with a variety of weathering effects is prepared, users can interactively choose and apply the predefined effects onto the surface of 3D graphic models and then finally assign the most appropriate one. This video-based weathering method can be implemented with very simple algorithms and it supports predictable, intuitive, and natural effects. Our system allows users to produce photorealistic augmented videos that include 3D graphic models weathered by our method. Moreover, users can easily enhance the realism of the augmented videos by manipulating rendering parameters such as the degree of weathering, texturing, lighting, and shadowing through user-friendly graphical user interfaces (GUIs).

Image-guided transorbital procedures with endoscopic video augmentation.

Surgical interventions to the orbital space behind the eyeball are limited to highly invasive procedures due to the confined nature of the region along with the presence of several intricate soft tissue structures. A minimally invasive approach to orbital surgery would enable several therapeutic options, particularly new treatment protocols for optic neuropathies such as glaucoma. The authors have developed an image-guided system for the purpose of navigating a thin flexible endoscope to a specified target region behind the eyeball. Navigation within the orbit is particularly challenging despite its small volume, as the presence of fat tissue occludes the endoscopic visual field while the surgeon must constantly be aware of optic nerve position. This research investigates the impact of endoscopic video augmentation to targeted image-guided navigation in a series of anthropomorphic phantom experiments. A group of 16 surgeons performed a target identification task within the orbits of four skull phantoms. The task consisted of identifying the correct target, indicated by the augmented video and the preoperative imaging frames, out of four possibilities. For each skull, one orbital intervention was performed with video augmentation, while the other was done with the standard image guidance technique, in random order. The authors measured a target identification accuracy of 95.3% and 85.9% for the augmented and standard cases, respectively, with statistically significant improvement in procedure time (Z=-2.044, p=0.041) and intraoperator mean procedure time (Z=2.456, p=0.014) when augmentation was used. Improvements in both target identification accuracy and interventional procedure time suggest that endoscopic video augmentation provides valuable additional orientation and trajectory information in an image-guided procedure. Utilization of video augmentation in transorbital interventions could further minimize complication risk and enhance surgeon comfort and confidence in the procedure.