Dynamic 3D Reconstruction Research Articles

Multi-view high-dynamic-range 3D reconstruction and point cloud quality evaluation based on dual-frame difference images

A high dynamic range can easily lead to image saturation, making it a challenge for structured light 3D reconstruction. The article proposes a multi-view 3D topography measurement system, which consists of dual projectors, a single camera, and a high-precision rotary platform. The system utilizes single-frame images to achieve high-dynamic-range surface adaptive exposure. Subsequently, it proposes a method that combines two-frame differential images with multi-view imaging to identify highly reflective regions and complete point cloud holes. This approach addresses the issue of visual blind spots caused by shadows and local high reflectivity due to the occlusion of the measured object’s geometric features. Finally, the system employs deep learning to evaluate the quality of the high-dynamic-range point cloud data. Comparative experiments show that optimal exposure in single-frame images can achieve better imaging quality and shorten capture time. The dual-frame difference image algorithm can identify high-reflection areas and complete the point cloud data. The point cloud quality evaluation model based on IT-PCQA demonstrates the effectiveness of the proposed method for high-dynamic-range 3D reconstruction.

3D reconstruction with single-frame two-step phase-shift method based on orthogonal composite fringe pattern projection

Abstract In order to realize single-frame three-dimensional (3D) reconstruction, a single-frame two-step phase-shift method based on orthogonal composite pattern projection is proposed to solve the problem that the traditional N-step phase-shift profilometry needs multiple projections for 3D reconstruction. The orthogonal composite pattern uses only two carrier channels to reduce the spectrum overlapping influence on the demodulation accuracy of carrier and modulated fringes. A two-dimensional variational mode decomposition method is adopted to remove the background DC component of the sinusoidal fringe to overcome the mode overlap problem by controlling the size of the bandwidth. Thus, the two-step phase-shift method is applied to calculate the phases for 3D reconstruction. The experimental results show that, compared with the typical Fourier transform profilometry method, 3-step composite method and 2 + 1 composite method, the 3D reconstruction accuracy of the proposed method is improved by 49.1%,31.4% and 23.2% respectively according to mean absolute error, and by 73.0%, 58.4% and 56.8% respectively according to mean squared error as the evaluation index. Finally, the dynamic 3D reconstruction experiment demonstrates the good adaptability of dynamic 3D reconstruction.

Dual-stage hybrid network for single-shot fringe projection profilometry based on a phase-height model.

Single-shot fringe projection profilometry (FPP) is widely used in the field of dynamic optical 3D reconstruction because of its high accuracy and efficiency. However, the traditional single-shot FPP methods are not satisfactory in reconstructing complex scenes with noise and discontinuous objects. Therefore, this paper proposes a Deformable Convolution-Based HINet with Attention Connection (DCAHINet), which is a dual-stage hybrid network with a deformation extraction stage and depth mapping stage. Specifically, the deformable convolution module and attention gate are introduced into DCAHINet respectively to enhance the ability of feature extraction and fusion. In addition, to solve the long-standing problem of the insufficient generalization ability of deep learning-based single-shot FPP methods on different hardware devices, DCAHINet outputs phase difference, which can be converted into 3D shapes by simple multiplication operations, rather than directly outputting 3D shapes. To the best of the author's knowledge, DCAHINet is the first network that can be applied to different hardware devices. Experiments on virtual and real datasets show that the proposed method is superior to other deep learning or traditional methods and can be used in practical application scenarios.

Super-Resolved Dynamic 3D Reconstruction of the Vocal Tract during Natural Speech.

MRI is the gold standard modality for speech imaging. However, it remains relatively slow, which complicates imaging of fast movements. Thus, an MRI of the vocal tract is often performed in 2D. While 3D MRI provides more information, the quality of such images is often insufficient. The goal of this study was to test the applicability of super-resolution algorithms for dynamic vocal tract MRI. In total, 25 sagittal slices of 8 mm with an in-plane resolution of 1.6 × 1.6 mm2 were acquired consecutively using a highly-undersampled radial 2D FLASH sequence. The volunteers were reading a text in French with two different protocols. The slices were aligned using the simultaneously recorded sound. The super-resolution strategy was used to reconstruct 1.6 × 1.6 × 1.6 mm3 isotropic volumes. The resulting images were less sharp than the native 2D images but demonstrated a higher signal-to-noise ratio. It was also shown that the super-resolution allows for eliminating inconsistencies leading to regular transitions between the slices. Additionally, it was demonstrated that using visual stimuli and shorter text fragments improves the inter-slice consistency and the super-resolved image sharpness. Therefore, with a correct speech task choice, the proposed method allows for the reconstruction of high-quality dynamic 3D volumes of the vocal tract during natural speech.

Evaluating visual encoding quality of a mixed reality user interface for human–machine co-assembly in complex operational terrain

During human–machine collaboration in manufacturing activities, it is important to provide real-time annotations in the three-dimensional workspace for local workers who may lack relevant experience and knowledge. For example, in MR assembly, workers need to be alerted to avoid entering hazardous areas when manually replacing components. Recently, many researchers have explored various visual cues for expressing physical task progress information in the MR interface of intelligent systems. However, the relationship between the implantation of visual cues and the balance of interface cognition has not been well revealed, especially in tasks that require annotating hazardous areas in complex operational terrains. In this study, we developed a novel MR interface for an intelligent assembly system that supports local scene sharing based on dynamic 3D reconstruction, remote expert behavior intention recognition based on deep learning, and local personnel operational behavior visual feedback based on external bounding box. We compared the encoding results of the proposed MR interface with 3D annotations combined with 3D sketch cues (3DS), which combines 3D spatial cues (3DSC) and 3DS combined with adaptive cues (AVC), through a case study. We found that for physical tasks that require specific area annotations, 3D annotations with context (3DAC) can better improve the quality of manual work and regulate the cognitive load distribution of the MR interface more reasonably.

Environment Modeling Method for Intelligent Robots Based on Binocular Vision

With the development of technology, the demand for intelligent mobile robots is growing steadily. Environment perception and modeling, are the key elements to achieve robot intelligence. Therefore, binocular stereo vision sensor is used as the core device to perform operations such as binocular camera calibration, obtaining binocular camera internal/external parameters, and distortion correction. On this basis, a multi-dimensional and all-round environment perception system is constructed, and the dynamic target detection results and 3D scene map reconstruction results are displayed dynamically in real time. The experimental results of binocular camera calibration show that the re-projection error lies in the sub-pixel interval, and the image distortion caused by the camera is basically eliminated to meet the experimental expectation of distortion correction; the experimental results of environment modeling show that the mean value of deviation between the object representation value of 3D scene map and the actual object real value is -0.041m, among which the minimum deviation value is 0.001m and the maximum deviation value is 0.532m, in The experimental results are excellent in the tasks of 3D scene map reconstruction and feature point matching. In addition, the storage space of 3D scene map is significantly reduced, and the real-time performance of the system is significantly better than the traditional method.

High dynamic reflection surface 3D reconstruction with sharing phase demodulation mechanism and multi-indicators guided phase domain fusion.

Accurate and complete 3D measurement of complex high dynamic range (HDR) surfaces has been challenging for structured light projection technique. The behavior of spraying a layer of diffuse reflection material, which will inevitably incur additional thickness. Existing methods based on additional facilities will increase the cost of hardware system. The algorithms-based methods are cost-effective and nondestructive, but they generally require redundant patterns for image fusion and model training, which fail to be suitable for practicing automated 3D measurement for complex HDR surfaces. In this paper, a HDR surface 3D reconstruction method based on sharing demodulation phase unwrapping mechanism and multi-indicators guided phase fusion strategy is proposed. The division of the exposure interval is optimized via the image entropy to generate an optimal exposure sequence. The combination of temporal-spatial binary (TSB) encoding fringe patterns with time-integration strategy and the variable exposure mode of digital mirror device (DMD)-based projector with a minimum projection exposure time of 233μs enables the proposed approach to broadly adapt complex HDR surfaces. We propose an efficient phase analysis solution called sharing mechanism that wrapped phase sequences from captured different intensity fringe images are unwrapped through sharing the same group of misaligned Gray code (MGC) decoding result. Finally, a phase sequences fusion model guided by multi-indicators, including exposure quality, phase gradient smoothness and pixel effectiveness, is established to obtain an optimum phase map for final 3D reconstruction. Comparative experiments indicate that the proposed method can completely restore the 3D topography of HDR surfaces with the images reduction of at least 65% and the measurement integrity is maintained at over 98% while preserving the measurement accuracy and excluding the outliers.

Embedded 3D reconstruction of dynamic objects in real time for maritime situational awareness pictures

Assessing the security status of maritime infrastructures is a key factor for maritime safety and security. Facilities such as ports and harbors are highly active traffic zones with many different agents and infrastructures present, like containers, trucks or vessels. Conveying security-related information in a concise and easily understandable format can support the decision-making process of stakeholders, such as port authorities, law enforcement agencies and emergency services. In this work, we propose a novel real-time 3D reconstruction framework for enhancing maritime situational awareness pictures by joining temporal 2D video data into a single consistent display. We introduce and verify a pipeline prototype for dynamic 3D reconstruction of maritime objects using a static observer and stereoscopic cameras on an GPU-accelerated embedded device. A simulated dataset of a harbor basin was created and used for real-time processing. Usage of a simulated setup allowed verification against synthetic ground-truth data. The presented pipeline runs entirely on a remote, low-power embedded system with ∼\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sim $$\\end{document}6 Hz. A Nvidia Jetson Xavier AGX module was used, featuring 512 CUDA-cores, 16 GB memory and an ARMv8 64-bit octa-core CPU.

Reconstructing dynamic human shapes from sparse silhouettes via latent space optimization of Parametric shape models

The problem of dynamic 3D reconstruction has gained popularity over the last few years with most approaches relying on data driven learning and optimization methods. However this is quite a challenging task because of the need for tracking different features in both space and time?that too of deformable objects-where such robust tracking may not always be possible. A common way to better ground the problem is by using some forms of regularizations primarily on the shape representations. Over the years, mesh-based linear blend skinning models have been the standard for fitting templates of humans to the observed time series data of human deformation. However, this approach suffers from optimization difficulties arising from maintaining a consistent mesh topology. In this paper, a novel algorithm for reconstructing dynamic human shapes has been proposed, which uses only sparse silhouette information. This is achieved by first creating shape models based on the signed distance neural fields which are subsequently optimized via volumetric differentiable rendering to best match the observed data. Several experiments have been carried out in this work to test the robustness of this method and the results show it to be quite robust, outperforming prior state of the art on dynamic human shape reconstruction by 45%.

Visual Three-Dimensional Reconstruction Based on Spatiotemporal Analysis Method

To accurately reconstruct the three-dimensional (3D) surface of dynamic objects, we proposed a wrapped phase extraction method for spatiotemporal analysis based on 3D wavelet transform (WT). Our proposed method uses a 2D spatial fringe image combined with the time dimension and forms a 3D image sequence. The encoded fringe image sequence’s wrapped phase information was extracted by 3D WT and complex Morlet wavelet, and we improved the wrapped phase extraction’s accuracy by using the characteristics of spatiotemporal analysis and a multi-scale analysis of 3D WT, then we reconstructed the measured object by wrapped phase unwrapping and phase height transformation. Our simulation experiment results show that our proposed method can further filter the noise in the time dimension, and its accuracy is better than that of the one- (1D) and two-dimensional (2D) WT wrapped phase extraction method and the 3D Fourier transform wrapped phase extraction method because the reconstructed spherical crown’s RMSE value does not exceed 0.25 and the PVE value is less than 0.95. Our results show that the proposed method can be applied to the dynamic 3D reconstruction of a real human thoracic and abdominal surface, which fluctuates slowly with respiration movement, further verifying its effectiveness.

Single frame color fringe projection profilometry based on empirical mode decomposition imbalance correction algorithm for fast three-dimensional reconstruction

Phase-shifting profilometry (PSP) is considered to be the most accurate technique for phase retrieval with fringe projection profilometry (FPP) systems. However, PSP requires that multiple phase-shifted fringe patterns be acquired, usually sequentially, which limits PSP to static or quasistatic imaging. We introduce color FPP (CFPP), which provides three-dimensional (3D) imaging using a single acquisition, for the application of real-time profiling. A single frame acquisition provides all three phase-shifted fringe patterns needed for the PSP phase retrieval algorithm. However, some problems must be overcome in CFPP, namely, color coupling and imbalance. Following up on the study, we propose a fixed coefficient decoupling method based on point-to-point correction and then introduce the empirical mode decomposition algorithm to decompose a color fringe pattern into three high-frequency components and three low-frequency components for imbalance correction, which improves measurement accuracy and realizes a single frame measurement for complex and discontinuous objects. After correction by the proposed method, the shape of the tested object can be recovered by one a single frame color fringe pattern, which meets the need for dynamic measurement, and the accuracy is not limited by the movement speed of object. The simulations and experimental results show that the proposed method can significantly reduce the influence of color coupling and imbalance in CFPP, and thus realize dynamic and high-precision 3D reconstruction.

Emerging Role of Ultrasound in Dysphagia Assessment and Intervention: A Narrative Review.

Ultrasonography has gained increasing attention as a non-invasive and radiation-free instrument for the assessment of swallowing function. In the past decades, an extensive repertoire of ultrasonographic techniques, such as, B-mode dynamic scanning, pixel analysis, M-mode, Doppler, 3D reconstruction, and sonoelastography, has been applied in the evaluation of oropharyngeal structures and movement. Yet, a universal consensus on the examination protocols and clinical implications remains to be established. This review aimed to provide a brief introduction of the application of ultrasound in dysphagia assessment and intervention, encompassing the ultrasonography of swallowing-related muscles, tongue movement, and hyolaryngeal excursion, as well as ultrasound-guided interventions in the management of dysphagia. In addition to non-invasiveness, ultrasonography, a portable, easy to use, and low-cost technique, could compliment videofluoroscopic swallowing study as a first-line screening and follow-up tool for the evaluation of swallowing function, although further study is warranted to provide quantitative diagnostic and prognostic values. Finally, ultrasonography aids in the precisely targeted injection of botulinum toxin in patients exhibiting oropharyngeal muscle spasticity.

Using facial landmarks to assist the stereo matching in fringe projection based 3D face profilometry

A three-dimensional (3D) face profilometry based on facial landmarks and a multi-view system is proposed. Theoretically, at least three fringe patterns are required in phase-shifting profilometry, but the stereo relative phase pair obtained by a spatial unwrapping algorithm cannot be used for stereo matching directly because the starting points of phase unwrapping are not on the same fringe period order, so it will result in incorrect disparity. We proposed an innovative method that utilizes facial landmarks as prior knowledge to adjust the relative phase pair to the same phase reference for accurate stereo matching through a phase adjustment algorithm to determine the accurate fringe order difference. The three-step phase-shifting patterns are projected in a looping sequence, and each three-adjacent fringe stereo pair is treated as a group to obtain one frame of a 3D face, which means that the 3D face output of our proposed method can achieve the same frame rate as the camera. The accuracy evaluation and dynamic 3D facial expressions reconstruction verify the success of the proposed method.

Mean Shift Fusion Color Histogram Algorithm for Nonrigid Complex Target Tracking in Sports Video

We analyze and study the tracking of nonrigid complex targets of sports video based on mean shift fusion color histogram algorithm. A simple and controllable 3D template generation method based on monocular video sequences is constructed, which is used as a preprocessing stage of dynamic target 3D reconstruction algorithm to achieve the construction of templates for a variety of complex objects, such as human faces and human hands, broadening the use of the reconstruction method. This stage requires video sequences of rigid moving target objects or sets of target images taken from different angles as input. First, the standard rigid body method of Visuals is used to obtain the external camera parameters of the sequence frames as well as the sparse feature point reconstruction data, and the algorithm has high accuracy and robustness. Then, a dense depth map is computed for each input image frame by the Multi-View Stereo algorithm. The depth reconstruction with a too high resolution not only increases the processing time significantly but also generates more noise, so the resolution of the depth map is controlled by parameters. The multiple hypothesis target tracking algorithms are used to track multiple targets, while the chunking feature is used to solve the problem of mutual occlusion and adhesion between targets. After finishing the matching, the target and background models are updated online separately to ensure the validity of the target and background models. Our results of nonrigid complex target tracking by mean shift fusion color histogram algorithm for sports video improve the accuracy by about 8% compared to other studies. The proposed tracking method based on the mean shift algorithm and color histogram algorithm can not only estimate the position of the target effectively but also depict the shape of the target well, which solves the problem that the nonrigid targets in sports video have complicated shapes and are not easy to track. An example is given to demonstrate the effectiveness and adaptiveness of the applied method.

Multi-view data capture for dynamic object reconstruction using handheld augmented reality mobiles

We propose a system to capture nearly-synchronous frame streams from multiple and moving handheld mobiles that is suitable for dynamic object 3D reconstruction. Each mobile executes Simultaneous Localisation and Mapping on-board to estimate its pose, and uses a wireless communication channel to send or receive synchronisation triggers. Our system can harvest frames and mobile poses in real time using a decentralised triggering strategy and a data-relay architecture that can be deployed either at the Edge or in the Cloud. We show the effectiveness of our system by employing it for 3D skeleton and volumetric reconstructions. Our triggering strategy achieves equal performance to that of an NTP-based synchronisation approach, but offers higher flexibility, as it can be adjusted online based on application needs. We created a challenging new dataset, namely 4DM, that involves six handheld augmented reality mobiles recording an actor performing sports actions outdoors. We validate our system on 4DM, analyse its strengths and limitations, and compare its modules with alternative ones.

Hierarchical, Dense and Dynamic 3D Reconstruction Based on VDB Data Structure for Robotic Manipulation Tasks.

This paper presents a novel approach to implement hierarchical, dense and dynamic reconstruction of 3D objects based on the VDB (Variational Dynamic B + Trees) data structure for robotic applications. The scene reconstruction is done by the integration of depth-images using the Truncated Signed Distance Field (TSDF). The proposed reconstruction method is based on dynamic trees in order to provide similar reconstruction results to the current state-of-the-art methods (i.e., complete volumes, hashing voxels and hierarchical volumes) in terms of execution time but with a direct multi-level representation that remains real-time. This representation provides two major advantages: it is a hierarchical and unbounded space representation. The proposed method is optimally implemented to be used on a GPU architecture, exploiting the parallelism skills of this hardware. A series of experiments will be presented to prove the performance of this approach in a robot arm platform.

Dynamic Obstacles Detection and 3D Reconstruction by a Monocular Camera and a 3D-map

Dynamic Obstacles Detection and 3D Reconstruction by a Monocular Camera and a 3D-map

Accurate Dynamic SLAM Using CRF-Based Long-Term Consistency.

Accurate camera pose estimation is essential and challenging for real world dynamic 3D reconstruction and augmented reality applications. In this article, we present a novel RGB-D SLAM approach for accurate camera pose tracking in dynamic environments. Previous methods detect dynamic components only across a short time-span of consecutive frames. Instead, we provide a more accurate dynamic 3D landmark detection method, followed by the use of long-term consistency via conditional random fields, which leverages long-term observations from multiple frames. Specifically, we first introduce an efficient initial camera pose estimation method based on distinguishing dynamic from static points using graph-cut RANSAC. These static/dynamic labels are used as priors for the unary potential in the conditional random fields, which further improves the accuracy of dynamic 3D landmark detection. Evaluation using the TUM and Bonn RGB-D dynamic datasets shows that our approach significantly outperforms state-of-the-art methods, providing much more accurate camera trajectory estimation in a variety of highly dynamic environments. We also show that dynamic 3D reconstruction can benefit from the camera poses estimated by our RGB-D SLAM approach.

Modified three-wavelength phase unwrapping algorithm for dynamic three-dimensional shape measurement

Digital fringe projection (DFP) method has been widely used in three-dimensional (3D) shape measurement. Lots of studies have been conducted to enhance the measurement accuracy and speed by modulating the patterns. However, it is still hard to acquire enough number of 3D results to ensure the continuity of dynamic 3D shape measurement when phase shifting profilometry and temporal phase unwrapping algorithm is adopted. Besides, object motion can introduce phase error and thus measurement error for phase shifting profilometry. To this end, a dynamic 3D reconstruction framework based on modified three-wavelength phase unwrapping algorithm and phase error compensation method is proposed to solve this problem. By redefining the order of fringe patterns projected by the digital light processing (DLP) projector, the number of required patterns for 3D reconstruction can be reduced and the speed of dynamic 3D shape measurement can be improved. And using phase error compensation method based on Hilbert transform, the accuracy of 3D reconstruction can also be improved. Experiments were carried out to demonstrated the effectiveness of the proposed framework. And the modified algorithm successfully improved the speed of dynamic 3D shape measurement by 1∕3.

A Comprehensive Review Towards Segmentation and Detection of Cancer Cell and Tumor for Dynamic 3D Reconstruction

Automated cancer cell and tumor segmentation and detection for 3D modeling are still an unsolved research problem in computer vision, image processing and pattern recognition research domains. Human body is complex three-dimensional structure where numerous types of cancer and tumor may exist regardless of shape or position. A three-dimensional (3D) reconstruction of cancer cell and tumor from body parts does not lead to loss of information like 2D shape visualization. Various research methodologies for segmentation and detection for 3D reconstruction of cancer cell and tumor were performed by previous research. However, the pursuit for better methodology for segmentation and detection for 3D reconstruction of cancer cell and tumor are still unsolved research problem due to lack of efficient feature extraction for details surface information, misclassification during training phases and low tissue contrast which causes low detection and precision rate with high computational complexity during detection and segmentation. This research addresses comprehensive and critical review of various segmentation and detection research methodologies for cancer affected cell and tumor in human body in the basis of three-dimensional reconstruction from MRI or CT images. At first, core research background is illustrated highlighting various aspects addressed by this research. After that, various previous methods with advantages and disadvantages followed by various phases used as frameworks exist in the previous research demonstrated by this research. Then, extensive experimental evaluations done by previous research are demonstrated by this research with various performance metrics. At last, this research summarized overall observation on previous research categorized into two aspects, i.e. observation on common research methodologies and recommended area for further research. Overall reviews proposed in this paper have been extensively studied in various research papers which can significantly contribute to computer vision research and can be potential for future development and direction for future research.