Viewpoint Tracking Research Articles

Whole‐body motion planning and tracking of a mobile robot with a gimbal RGB‐D camera for outdoor 3D exploration

AbstractAutonomous exploration is a fundamental problem for certain applications of unmanned ground vehicles (UGVs). Depth cameras are widely used in exploring indoor and specific outdoor environments. However, existing methods suffer from inefficient and unstable performance of exploration in open outdoor environments. This study develops a framework to achieve fast and robust three‐dimensional exploration by UGVs with a gimbal camera in challenging outdoor environments. A hierarchical reactive planning approach based on a multilayered map is proposed to improve exploration efficiency. Additionally, an adaptive semantic mapping algorithm is proposed to more accurately and stably represent the environmental localization uncertainty. Localization uncertainty is integrated into the planning approach to decrease odometry drift and prevent localization failure, further enhancing the exploration robustness. A viewpoint tracking controller based on terrain‐aware model predictive control is proposed to guarantee the accuracy of the planned viewpoint tracking and to further improve the robustness of localization and exploration on rough terrains. Finally, several physical‐engine simulations and experiments in outdoor environments are performed. The comparisons to existing state‐of‐the‐art approaches have verified the effectiveness of the proposed approach. The code of our system will be available at https://github.com/HITSZ-NRSL/Open3DExplorer.git.

Perceptual Assessment of Image and Depth Quality of Dynamically Depth-Compressed Scene for Automultiscopic 3D Display.

This article discusses the depth range which automultiscopic 3D (A3D) displays should reproduce for ensuring an adequate perceptual quality of substantially deep scenes. These displays usually need sufficient depth reconstruction capabilities covering the whole scene depth, but due to the inherent hardware restriction of these displays this is often difficult, particularly for showing deep scenes. Previous studies have addressed this limitation by introducing depth compression that contracts the scene depth into a smaller depth range by modifying the scene geometry, assuming that the scenes were represented as CG data. The previous results showed that reconstructing only a physical depth of 1 m is needed to show scenes with much deeper depth and without large perceptual quality degradation. However, reconstructing a depth of 1 m is still challenging for actual A3D displays. In this study, focusing on a personal viewing situation, we introduce a dynamic depth compression that combines viewpoint tracking with the previous approach and examines the extent to which scene depths can be compressed while keeping the original perceptual quality. Taking into account the viewer's viewpoint movements, which were considered a cause of unnaturalness in the previous approach, we performed an experiment with an A3D display simulator and found that a depth of just 10 cm was sufficient for showing deep scenes without inducing a feeling of unnaturalness. Next, we investigated whether the simulation results were valid even on a real A3D display and found that the dynamic approach induced better perceptual quality than the static one even on the real A3D display and that it had a depth enhancing effect without any hardware updates. These results suggest that providing a physical depth of 10 cm on personalized A3D displays is general enough for showing any deeper 3D scenes with appealing subjective quality.

An accurate 3D augmented reality navigation system with enhanced autostereoscopic display for oral and maxillofacial surgery.

Oral and maxillofacial surgery demands high precision navigation to achieve optimal surgical outcomes. Augmented reality (AR) has been a promising solution for intuitive and accurate guidance, whereas existing systems have room for improvement. We propose a high quality and accurate in situ AR navigation system. Enhanced image quality is achieved by deploying eye tracking in lenticular-based autostereoscopic display. Viewpoint tracking and optical tracking are integrated to assure accurate in situ images. The proposed system can provide in situ AR images in real-time, with a viewing angle of 59.5°×49.1°, 3D image resolution of 0.35×0.21mm, and image accuracy of 0.99±0.70mm. A maxillary drilling experiment obtains average position error of 1.12±0.30mm and localization time of 8.7±3.9s, significantly better than conventional 2D navigation system. The proposed system can display high-quality AR images and therefore reduce positioning error and operating time.

Motion parallax and lossless resolution autostereoscopic 3D display based on a binocular viewpoint tracking liquid crystal dynamic grating adaptive screen.

The autostereoscopic 3D display has two important indicators, both the number of viewpoints and display resolution. However, it's a challenge to improve both the viewpoint and the resolution. Here, we develop a fixed-position multiview and lossless resolution autostereoscopic 3D display system that includes the dynamic liquid crystal (LC) grating screen. This display system consists of an LC display panel and an LC grating screen. The synchronization of the frame switching of the LC display panel and the LC grating screen shutter enables the preserved resolution. The "eye space" design makes the viewpoint dense enough and determines the LC grating screen's parameters. We use binocular viewpoint tracking technology to realize the LC grating screen's adaptive control based on the above work. Different binocular views are rendered in real-time according to the different positions of a single pair of stereoscopic viewpoints in the eye space, making the motion parallax possible. We present the working principle and mathematical analysis. We implement a prototype for verifying the principle. According to the experiment results analysis, this prototype can achieve viewpoint tracking and motion parallax based on resolution lossless and viewpoint dense enough.

Crosstalk Reduction for Parallax Barrier Stereoscopic Display Based on High-speed Viewpoint Tracking and Projection

Crosstalk Reduction for Parallax Barrier Stereoscopic Display Based on High-speed Viewpoint Tracking and Projection

Projection-based augmented reality system for assembly guidance and monitoring

PurposeThis study aims to monitor and guide the assembly process. The operators need to change the assembly process according to the products’ specifications during manual assembly of mass customized production. Traditional information inquiry and display methods, such as manual lookup of assembly drawings or electronic manuals, are inefficient and error-prone.Design/methodology/approachThis paper proposes a projection-based augmented reality system (PBARS) for assembly guidance and monitoring. The system includes a projection method based on viewpoint tracking, in which the position of the operator’s head is tracked and the projection images are changed correspondingly. The assembly monitoring phase applies a method for parts recognition. First, the pixel local binary pattern (PX-LBP) operator is achieved by merging the classical LBP operator with the pixel classification process. Afterward, the PX-LBP features of the depth images are extracted and the randomized decision forests classifier is used to get the pixel classification prediction image (PCPI). Parts recognition and assembly monitoring is performed by PCPI analysis.FindingsThe projection image changes with the viewpoint of the human body, hence the operators always perceive the three-dimensional guiding scene from different viewpoints, improving the human-computer interaction. Part recognition and assembly monitoring were achieved by comparing the PCPIs, in which missing and erroneous assembly can be detected online.Originality/valueThis paper designed the PBARS to monitor and guide the assembly process simultaneously, with potential applications in mass customized production. The parts recognition and assembly monitoring based on pixels classification provides a novel method for assembly monitoring.

Analysing student engagement with 360-degree videos through multimodal data analytics and user annotations

ABSTRACT This paper presents the findings from a case study which explores how students engage with learning materials mediated through 360-degree panoramic videos. The case study extends findings from a project which developed an online platform for viewing and annotating 360-degree videos, and for providing data analytics (heatmaps, viewpoint tracking and area of interest displays) to map student engagement with video content. In this case, information obtained from the data analytics forms the basis for the analysis which, taking a social semiotic perspective, explores (a) what multimodal resources in the video-recorded classroom activity function to attract and hold students’ attention, and (b) what multimodal resources students employ when interacting with the 360-degree videos and for which purpose, taking the context into account. The findings indicate that for students to engage meaningfully with educational content requires familiarity with the demands of 360 video technology, which for some users can prove distracting, or even disabling.

Virtual Mirror Rendering With Stationary RGB-D Cameras and Stored 3-D Background

Mirrors are indispensable objects in our lives. The capability of simulating a mirror on a computer display, augmented with virtual scenes and objects, opens the door to many interesting and useful applications from fashion design to medical interventions. Realistic simulation of a mirror is challenging as it requires accurate viewpoint tracking and rendering, wide-angle viewing of the environment, as well as real-time performance to provide immediate visual feedback. In this paper, we propose a virtual mirror rendering system using a network of commodity structured-light RGB-D cameras. The depth information provided by the RGB-D cameras can be used to track the viewpoint and render the scene from different prospectives. Missing and erroneous depth measurements are common problems with structured-light cameras. A novel depth denoising and completion algorithm is proposed in which the noise removal and interpolation procedures are guided by the foreground/background label at each pixel. The foreground/background label is estimated using a probabilistic graphical model that considers color, depth, background modeling, depth noise modeling, and spatial constraints. The wide viewing angle of the mirror system is realized by combining the dynamic scene, captured by the static camera network with a 3-D background model created off-line, using a color-depth sequence captured by a movable RGB-D camera. To ensure a real-time response, a scalable client-and-server architecture is used with the 3-D point cloud processing, the viewpoint estimate, and the mirror image rendering are all done on the client side. The mirror image and the viewpoint estimate are then sent to the server for final mirror view synthesis and viewpoint refinement. Experimental results are presented to show the accuracy and effectiveness of each component and the entire system.

Real‐time interactive fusion of physical/electronic auscultation simulation and 3D CT/MRI‐based anatomical modeling applied to training in lung sounds detection, discrimination and classification

Objective: To test the instructional effectiveness of simultaneous coupling of the spatial detection, discrimination and classification of simulated lung sounds to real‐time viewpoint tracking of displayed digital 3D anatomical models over the current method of auscultation alone. Methods: An integration of a computer‐based clinical training system comprising two complementary functions: 3D anatomical visualizations, and physiological diagnostic sounds simulation using a physical manikin. These functions are linked in real‐time by the student stethoscope's 3D position/orientation in space on the surface of the manikin. Thus, during any given simulation drill using as input a pre‐defined diagnostic lung sound, a student will sample signs from the manikin and simultaneously view the case's prescribed condition or pathology volumetrically reconstructed and visualized from de‐identified patient MRI/CT image sets carefully selected to represent the given abnormality. Expected results: Active student tracking, perception of diagnostic sounds and visualizations of the corresponding anatomical structures will significantly increase student learning over methods using auscultation without visualization.This research is supported by the Department of Anatomy and the Dean's Office, Des Moines University.