Multi-view Reconstruction Techniques Research Articles

Defect Detection and 3D Reconstruction of Complex Urban Underground Pipeline Scenes for Sewer Robots

Detecting defects in complex urban sewer scenes is crucial for urban underground structure health monitoring. However, most image-based sewer defect detection models are complex, have high resource consumption, and fail to provide detailed damage information. To increase defect detection efficiency, visualize pipelines, and enable deployment on edge devices, this paper proposes a computer vision-based robotic defect detection framework for sewers. The framework encompasses positioning, defect detection, model deployment, 3D reconstruction, and the measurement of realistic pipelines. A lightweight Sewer-YOLO-Slim model is introduced, which reconstructs the YOLOv7-tiny network by adjusting its backbone, neck, and head. Channel pruning is applied to further reduce the model’s complexity. Additionally, a multiview reconstruction technique is employed to build a 3D model of the pipeline from images captured by the sewer robot, allowing for accurate measurements. The Sewer-YOLO-Slim model achieves reductions of 60.2%, 60.0%, and 65.9% in model size, parameters, and floating-point operations (FLOPs), respectively, while improving the mean average precision (mAP) by 1.5%, reaching 93.5%. Notably, the pruned model is only 4.9 MB in size. Comprehensive comparisons and analyses are conducted with 12 mainstream detection algorithms to validate the superiority of the proposed model. The model is deployed on edge devices with the aid of TensorRT for acceleration, and the detection speed reaches 15.3 ms per image. For a real section of the pipeline, the maximum measurement error of the 3D reconstruction model is 0.57 m. These results indicate that the proposed sewer inspection framework is effective, with the detection model exhibiting advanced performance in terms of accuracy, low computational demand, and real-time capability. The 3D modeling approach offers valuable insights for underground pipeline data visualization and defect measurement.

Fragmentation Guided Human Shape Reconstruction

We present a novel semantic-driven multi-view reconstruction technique for producing realistic 3D human models. Our approach borrows the fragmentation concept in Cubism style painting, where the human body is decomposed into semantically meaningful fragments for conveying space and movement. We first employ deep learning-based skeleton estimation for warping a proxy human model under the canonical pose to the target multi-view input. It also conducts 3D fragment labeling on the warped model to separate different human body parts. Finally, we utilize the normal, depth, and fragment label of the proxy model as priors in the multi-view stereo reconstruction process. The comprehensive experiments have shown that our reconstruction technique outperforms the state-of-the-art methods in robustness and accuracy, especially near occlusion boundaries and on textureless regions. In particular, it manages to significantly reduce the “adhesive” artifacts commonly observed in MVS that incorrectly stitches different body parts.

SATELLITE STEREO BASED DIGITAL SURFACE MODEL GENERATION USING SEMI GLOBAL MATCHING IN OBJECT AND IMAGE SPACE

This paper presents methodology and evaluation of Digital Surface Models (DSM) generated from satellite stereo imagery using Semi Global Matching (SGM) applied in image space and georeferenced voxel space. SGM is a well known algorithm, used widely for DSM generation from airborne and satellite imagery. SGM is typically applied in the image space to compute disparity map corresponding to a stereo image pair. As a different approach, SGM can be applied directly to the georeferenced voxel space similar to the approach of volumetric multi-view reconstruction techniques. The matching in voxel space simplifies the DSM generation pipeline because the stereo rectification and triangulation steps are not required. For a comparison, the complete pipeline for generation of DSM from satellite pushbroom sensors is also presented. The results on the ISPRS satellite stereo benchmark using Worldview stereo imagery of 0.5m resolution shows that the SGM applied in image space produce slightly better results than its object space counterpart. Furthermore, a qualitative analysis of the results on Worldview-3 stereo and Pleiades tri-stereo images are presented.

SATELLITE STEREO BASED DIGITAL SURFACE MODEL GENERATION USING SEMI GLOBAL MATCHING IN OBJECT AND IMAGE SPACE

Abstract. This paper presents methodology and evaluation of Digital Surface Models (DSM) generated from satellite stereo imagery using Semi Global Matching (SGM) applied in image space and georeferenced voxel space. SGM is a well known algorithm, used widely for DSM generation from airborne and satellite imagery. SGM is typically applied in the image space to compute disparity map corresponding to a stereo image pair. As a different approach, SGM can be applied directly to the georeferenced voxel space similar to the approach of volumetric multi-view reconstruction techniques. The matching in voxel space simplifies the DSM generation pipeline because the stereo rectification and triangulation steps are not required. For a comparison, the complete pipeline for generation of DSM from satellite pushbroom sensors is also presented. The results on the ISPRS satellite stereo benchmark using Worldview stereo imagery of 0.5m resolution shows that the SGM applied in image space produce slightly better results than its object space counterpart. Furthermore, a qualitative analysis of the results on Worldview-3 stereo and Pleiades tri-stereo images are presented.

Articulated Billboards for Video‐based Rendering

AbstractWe present a novel representation and rendering method for free‐viewpoint video of human characters based on multiple input video streams. The basic idea is to approximate the articulated 3D shape of the human body using a subdivision into textured billboards along the skeleton structure. Billboards are clustered to fans such that each skeleton bone contains one billboard per source camera. We call this representation articulated billboards.In the paper we describe a semi‐automatic, data‐driven algorithm to construct and render this representation, which robustly handles even challenging acquisition scenarios characterized by sparse camera positioning, inaccurate camera calibration, low video resolution, or occlusions in the scene. First, for each input view, a 2D pose estimation based on image silhouettes, motion capture data, and temporal video coherence is used to create a segmentation mask for each body part. Then, from the 2D poses and the segmentation, the actual articulated billboard model is constructed by a 3D joint optimization and compensation for camera calibration errors. The rendering method includes a novel way of blending the textural contributions of each billboard and features an adaptive seam correction to eliminate visible discontinuities between adjacent billboards textures.Our articulated billboards do not only minimize ghosting artifacts known from conventional billboard rendering, but also alleviate restrictions to the setup and sensitivities to errors of more complex 3D representations and multiview reconstruction techniques. Our results demonstrate the flexibility and the robustness of our approach with high quality free‐viewpoint video generated from broadcast footage of challenging, uncontrolled environments.

WE‐C‐330A‐04: Effect of Projection Angles Used in Multi‐View Reconstruction (MVR) Using Images From a Microangiographic (MA) Detector and An Image‐Intensifier (II) System

Purpose: The sensitivity of a new 3D Multi‐View Reconstruction (MVR) angiography technique to the projection angles used is evaluated by comparing 3D centerlines calculated from combinations of three projections acquired from two imaging systems with that from micro‐Cone Beam CT (μCBCT), which is taken as truth. Method and Materials: A 3D centerline of a contrast‐filled carotid vessel phantom was reconstructed from image data acquired using a custom‐made μCBCT system with a microangiographic (MA) detector (45 μm pixels, 4.5 cm field‐of‐view (FOV)). Projection images of the same phantom were also acquired using the MA and an image intensifier (II) detector system (120 μm pixels, 4.5 in FOV) on a C‐arm x‐ray unit. The MVR technique was used to compute 3D centerlines for 12 combinations of projection angles. Each 3D MVR centerline was aligned with the μCBCT “true” 3D centerline using a Procrustes technique, and a root‐mean‐square (RMS) deviation was calculated. Results: The average RMS deviation for the MA‐MVR centerlines is 25 μm with a standard deviation of 3 μm over the 12 different projection‐angle combinations, whereas the average RMS deviation for the II‐MVR centerlines is 41 μm with a standard deviation of 4 μm over these same combinations. The RMS deviation as a percent of the internal vessel diameter, 0.75 mm, is 3.3% for the MA and 5.5% for the II and appears to be independent of view selection. Conclusion: For the MVR technique, the improved resolution of the MA resulted in improved centerline determination compared to the II system. For both detectors, the selection of a particular projection set had little effect on the RMS centerline deviation. The low RMS deviations for both detectors indicate that the MVR technique can provide accurate 3D centerlines.(Partial support from NIH Grants R01‐NS43924, R01‐EB02873, R01‐HL52567, R01‐EB02916, and Toshiba Medical Systems Corporation).