Multi-view Constraints Research Articles

SC-NeuS: Consistent Neural Surface Reconstruction from Sparse and Noisy Views

The recent neural surface reconstruction approaches using volume rendering have made much progress by achieving impressive surface reconstruction quality, but are still limited to dense and highly accurate posed views. To overcome such drawbacks, this paper pays special attention on the consistent surface reconstruction from sparse views with noisy camera poses. Unlike previous approaches, the key difference of this paper is to exploit the multi-view constraints directly from the explicit geometry of the neural surface, which can be used as effective regularization to jointly learn the neural surface and refine the camera poses. To build effective multi-view constraints, we introduce a fast differentiable on-surface intersection to generate on-surface points, and propose view-consistent losses on such differentiable points to regularize the neural surface learning. Based on this point, we propose a joint learning strategy, named SC-NeuS, to perform geometry-consistent surface reconstruction in an end-to-end manner. With extensive evaluation on public datasets, our SC-NeuS can achieve consistently better surface reconstruction results with fine-grained details than previous approaches, especially from sparse and noisy camera views. The source code is available at https://github.com/zouzx/sc-neus.git.

Decoupling Dynamic Monocular Videos for Dynamic View Synthesis.

The challenge of dynamic view synthesis from dynamic monocular videos, i.e., synthesizing novel views for free viewpoints given a monocular video of a dynamic scene captured by a moving camera, mainly lies in accurately modeling the dynamic objects of a scene using limited 2D frames, each with a varying timestamp and viewpoint. Existing methods usually require pre-processed 2D optical flow and depth maps by off-the-shelf methods to supervise the network, making them suffer from the inaccuracy of the pre-processed supervision and the ambiguity when lifting the 2D information to 3D. In this paper, we tackle this challenge in an unsupervised fashion. Specifically, we decouple the motion of the dynamic objects into object motion and camera motion, respectively regularized by proposed unsupervised surface consistency and patch-based multi-view constraints. The former enforces the 3D geometric surfaces of moving objects to be consistent over time, while the latter regularizes their appearances to be consistent across different viewpoints. Such a fine-grained motion formulation can alleviate the learning difficulty for the network, thus enabling it to produce not only novel views with higher quality but also more accurate scene flows and depth than existing methods requiring extra supervision. We will make the code publicly available at https://github.com/mengyou2/DecoulpingNeRF.

A novel intrarenal multimodal 2D/3D registration algorithm and preliminary phantom study.

Retrograde intrarenal surgery (RIRS) is a widely utilized diagnostic and therapeutic tool for multiple upper urinary tract pathologies. The image-guided navigation system can assist the surgeon to perform precise surgery by providing the relative position between the lesion and the instrument after the intraoperative image is registered with the preoperative model. However, due to the structural complexity and diversity of multi-branched organs such as kidneys, bronchi, etc., the consistency of the intensity distribution of virtual and real images will be challenged, which makes the classical pure intensity registration method prone to bias and random results in a wide search domain. In this paper, we propose a structural feature similarity-based method combined with a semantic style transfer network, which significantly improves the registration accuracy when the initial state deviation is obvious. Furthermore, multi-view constraints are introduced to compensate for the collapse of spatial depth information and improve the robustness of the algorithm. Experimental studies were conducted on two models generated from patient data to evaluate the performance of the method and competing algorithms. The proposed method obtains mean target error (mTRE) of 0.971±0.585mm and 1.266±0.416mm respectively, with better accuracy and robustness overall. Experimental results demonstrate that the proposed method has the potential to be applied to RIRS and extended to other organs with similar structures.

Semi-Supervised Multi-View Fusion for Identifying CAP and COVID-19 With Unlabeled CT Images

Recently, under the condition of reducing nucleic acid testing for COVID-19 in large population, the computer-aided diagnosis with the chest computed tomography (CT) image has become increasingly important in differential diagnosis of community-acquired pneumonia (CAP) and COVID-19. In practice, there usually exist a mass of unlabeled CT images, especially in regions without adequate medical resources, and the existing diagnosis methods cannot take advantage of the useful information among them. Therefore, it is practical and urgent need to develop a computer-aided diagnosis model that can effectively exploit both labeled and unlabeled samples. To this end, in this paper, we propose a semi-supervised multi-view fusion method for the diagnosis of COVID-19. It explores both the discriminative features from labeled samples and the structure information from unlabeled samples and fuses multi-view features extracted from CT images, including image feature, statistical feature, and lesions specific feature, for improving the diagnostic performance. Specifically, in the proposed model, we utilize semi-supervised learning technique with pairwise constraint regularization to learn the model with both labeled samples and unlabeled samples. Simultaneously, we employ low-rank multi-view constraint to capture latent complementary information among different features from CT images. Experimental results show that the proposed method outperforms the state-of-the-art methods in differential diagnosis of CAP vs. COVID-19.

Glass façade segmentation and repair for aerial photogrammetric 3D building models with multiple constraints

Aerial photogrammetric three-dimensional (3D) city models provide photorealistic virtual scenes for various applications, such as urban planning, autonomous driving, emergency management, etc. With the rapid progress of urbanization, glass façades commonly exist on buildings, which cannot be well reconstructed with geometric deformation and distortion due to the limitation of the image-based Multi-View Stereo (MVS) modeling pipeline. The distorted glass façades make building models visually unreal and messy. This paper proposes a workflow to segment and repair glass façades in aerial photogrammetric 3D building models with multiple constraints to resolve the problem, mainly including two aspects: 1) projection-based glass façade segmentation with geometric information and multi-view constraints, and 2) glass façade model repair via mesh smoothing and texture mapping. First, an efficient approach for façade information extraction is proposed, which utilizes the density of the reprojected point clouds and elevation differences because the typical structure of building façades is usually vertically aligned to the ground surface. Second, we project the façade mesh onto the multi-view oblique aerial images. The glass façades are predicted by a deep learning-based semantic segmentation method to distinguish the glass façades. Finally, based on the extracted glass façades, we remedy the geometric deformation using a mesh denoising method by grouping strategy with edge constraints. After smoothing the mesh structure of glass façades, we remap the texture to generate the completed 3D models with optimized glass façades. Our method leverages multiple constraints, including multi-view constraints, façade structure constraints, grouping strategy, and boundary constraints. The multi-view and façade structure constraints help decrease the adverse effects caused by false segmentation; the grouping strategy and boundary constraints can avoid disorders in some specific areas and preserve the sharp feature in the adjacent regions. Thus, our approach is robust and does not worsen the 3D model. Experiments demonstrate that our workflow is feasible for glass façade segmentation and repair for aerial photogrammetric 3D building models.

Vid2CAD: CAD Model Alignment Using Multi-View Constraints From Videos.

We address the task of aligning CAD models to a video sequence of a complex scene containing multiple objects. Our method can process arbitrary videos and fully automatically recover the 9 DoF pose for each object appearing in it, thus aligning them in a common 3D coordinate frame. The core idea of our method is to integrate neural network predictions from individual frames with a temporally global, multi-view constraint optimization formulation. This integration process resolves the scale and depth ambiguities in the per-frame predictions, and generally improves the estimate of all pose parameters. By leveraging multi-view constraints, our method also resolves occlusions and handles objects that are out of view in individual frames, thus reconstructing all objects into a single globally consistent CAD representation of the scene. In comparison to the state-of-the-art single-frame method Mask2CAD that we build on, we achieve substantial improvements on the Scan2CAD dataset (from 11.6% to 30.7% class average accuracy).

EgoFish3D: Egocentric 3D Pose Estimation From a Fisheye Camera via Self-Supervised Learning

Egocentric vision has gained increasing popularity recently, opening new avenues for human-centric applications. However, the use of the egocentric fisheye cameras allows wide angle coverage but image distortion is introduced along with strong human body self-occlusion imposing significant challenges in data processing and model reconstruction. Unlike previous work only leveraging synthetic data for model training, this paper presents a new real-world EgoCentric Human Pose (ECHP) dataset. To tackle the difficulty of collecting 3D ground truth using motion capture systems, we simultaneously collect images from a head-mounted egocentric fisheye camera as well as from two third-person-view cameras, circumventing the environmental restrictions. By using self-supervised learning under multi-view constraints, we propose a simple yet effective framework, namely EgoFish3D, for egocentric 3D pose estimation from a single image in different real-world scenarios. The proposed EgoFish3D incorporates three main modules. 1) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">The third-person-view module</i> takes two exocentric images as input and estimates the 3D pose represented in the third-person camera frame; 2) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">the egocentric module</i> predicts the 3D pose in the egocentric camera frame; and 3) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">the interactive module</i> estimates the rotation matrix between the third-person and the egocentric views. Experimental results on our ECHP dataset and existing benchmark datasets demonstrate the effectiveness of the proposed EgoFish3D, which can achieve superior performance to existing methods.

WS-OPE: Weakly Supervised 6-D Object Pose Regression Using Relative Multi-Camera Pose Constraints

Precise annotation of 6-D poses in real data is intricate and time-consuming, however, an essential requirement to train pose estimation pipelines. We propose a way for scalable, end-to-end 6-D pose regression with weak supervision to avoid this problem. Our method requires neither 3-D models nor 6-D object poses as ground truth. Instead, we use 2-D bounding boxes and object sizes as the only labels and constrain the problem with multiple images of known relative poses during training. A novel Rotated-IoU loss brings together a pose prediction from an image with labeled 2-D bounding boxes of the corresponding object in other views. Our rotation estimation combines an initial coarse pose classification with an offset regression using a continuous rotation parametrization that allows for direct pose estimation. At test time, the model still uses only a single image to predict a 6-D pose. We observe that multi-view constraints and our rotation representation used during training lead to better learning of 6-D pose embeddings in comparison to fully supervised methods. Experiments on several datasets show that the proposed method is capable of predicting poses of good quality, in spite being trained with only weak labels. Direct pose regression without the need for a consecutive refinement stage thereby ensures real-time performance.

Unsupervised Meta Learning With Multiview Constraints for Hyperspectral Image Small Sample set Classification.

The difficulties of obtaining sufficient labeled samples have always been one of the factors hindering deep learning models from obtaining high accuracy in hyperspectral image (HSI) classification. To reduce the dependence of deep learning models on training samples, meta learning methods have been introduced, effectively improving the classification accuracy in small sample set scenarios. However, the existing methods based on meta learning still need to construct a labeled source data set with several pre-collected HSIs, and must utilize a large number of labeled samples for meta-training, which is actually time-consuming and labor-intensive. To solve this problem, this paper proposes a novel unsupervised meta learning method with multiview constraints for HSI small sample set classification. Specifically, the proposed method first builds an unlabeled source data set using unlabeled HSIs. Then, multiple spatial-spectral multiview features of each unlabeled sample are generated to construct tasks for unsupervised meta learning. Finally, the designed residual relation network is used for meta-training and small sample set classification based on the voting strategy. Compared with existing supervised meta learning methods for HSI classification, our method can only utilize HSIs without any label for unsupervised meta learning, which significantly reduces the number of requisite labeled samples in the whole classification process. To verify the effectiveness of the proposed method, extensive experiments are carried out on 8 public HSIs in the cross-domain and in-domain classification scenarios. The statistical results demonstrate that, compared with existing supervised meta learning methods and other advanced classification models, the proposed method can achieve competitive or better classification performance in small sample set scenarios.

Unsupervised Video Action Clustering via Motion-Scene Interaction Constraint

In the past few years, scene contextual information has been increasingly used for action understanding with promising results. However, unsupervised video action clustering using context has been less explored, and existing clustering methods cannot achieve satisfactory performances. In this paper, we propose a novel unsupervised video action clustering method by using the motion-scene interaction constraint (MSIC). The proposed method takes the unique static scene and dynamic motion characteristics of video action into account, and develops a contextual interaction constraint model under a self-representation subspace clustering framework. First, the complementarity of multi-view subspace representation in each context is explored by single-view and multi-view constraints. Afterward, the context-constrained affinity matrix is calculated and the MSIC is introduced to mutually regularize the disagreement of subspace representation in scene and motion. Finally, by jointly constraining the complementarity of multi-views and the consistency of multi-contexts, an overall objective function is constructed to guarantee the video action clustering result. The experiments on four video benchmark datasets (Weizmann, KTH, UCFsports, and Olympic) demonstrate that the proposed method outperforms the state-of-the-art methods.

Multi-view Object Extraction with Fractional Boundaries.

This paper presents an automatic method to extract a multi-view object in a natural environment. We assume that the target object is bounded by the convex volume of interest defined by the overlapping space of camera viewing frustums. There are two key contributions of our approach. First, we present an automatic method to identify a target object across different images for multi-view binary co-segmentation. The extracted target object shares the same geometric representation in space with a distinctive color and texture model from the background. Second, we present an algorithm to detect color ambiguous regions along the object boundary for matting refinement. Our matting region detection algorithm is based on information theory, which measures the Kullback-Leibler (KL) divergence of local color distribution of different pixel-bands. The local pixel-band with the largest entropy is selected for matte refinement, subject to the multi-view consistent constraint. Our results are highquality alpha mattes consistent across all different viewpoints. We demonstrate the effectiveness of the proposed method using various examples.

Accurate and occlusion-robust multi-view stereo

This paper proposes an accurate multi-view stereo method for image-based 3D reconstruction that features robustness in the presence of occlusions. The new method offers improvements in dealing with two fundamental image matching problems. The first concerns the selection of the support window model, while the second centers upon accurate visibility estimation for each pixel. The support window model is based on an approximate 3D support plane described by a depth and two per-pixel depth offsets. For the visibility estimation, the multi-view constraint is initially relaxed by generating separate support plane maps for each support image using a modified PatchMatch algorithm. Then the most likely visible support image, which represents the minimum visibility of each pixel, is extracted via a discrete Markov Random Field model and it is further augmented by parameter clustering. Once the visibility is estimated, multi-view optimization taking into account all redundant observations is conducted to achieve optimal accuracy in the 3D surface generation for both depth and surface normal estimates. Finally, multi-view consistency is utilized to eliminate any remaining observational outliers. The proposed method is experimentally evaluated using well-known Middlebury datasets, and results obtained demonstrate that it is amongst the most accurate of the methods thus far reported via the Middlebury MVS website. Moreover, the new method exhibits a high completeness rate.

Multiple-Layer Visibility Propagation-Based Synthetic Aperture Imaging through Occlusion.

Heavy occlusions in cluttered scenes impose significant challenges to many computer vision applications. Recent light field imaging systems provide new see-through capabilities through synthetic aperture imaging (SAI) to overcome the occlusion problem. Existing synthetic aperture imaging methods, however, emulate focusing at a specific depth layer, but are incapable of producing an all-in-focus see-through image. Alternative in-painting algorithms can generate visually-plausible results, but cannot guarantee the correctness of the results. In this paper, we present a novel depth-free all-in-focus SAI technique based on light field visibility analysis. Specifically, we partition the scene into multiple visibility layers to directly deal with layer-wise occlusion and apply an optimization framework to propagate the visibility information between multiple layers. On each layer, visibility and optimal focus depth estimation is formulated as a multiple-label energy minimization problem. The layer-wise energy integrates all of the visibility masks from its previous layers, multi-view intensity consistency and depth smoothness constraint together. We compare our method with state-of-the-art solutions, and extensive experimental results demonstrate the effectiveness and superiority of our approach.

A Novel DSM Filtering Algorithm for Landslide Monitoring Based on Multiconstraints

With the aim of addressing the problems of vegetation interference and noise points existing in the process of digital surface model (DSM) filtering for landslide monitoring, this paper presents a point cloud filtering algorithm based on multiconstraints, which include vegetation detection, a multiview constraint, and local terrain continuity. The proposed algorithm is implemented via the following steps. First, we carry out the vegetation detection procedure and then remove the corresponding vegetation regions from the original images. Next, we utilize the multiview and local terrain continuity constraints to eliminate noise points from the generated point clouds (obtained through close-range photogrammetric processing). Finally, we apply the filtered point clouds to the landslide monitoring test to acquire reliable deformation information. The experimental results confirm that the vegetation and noise points can be effectively removed from the original point clouds by the use of the filtering algorithm proposed in this paper. Furthermore, accurate deformation areas can be obtained by comparing the DSMs from the different periods, and the achieved result is very close to the actual situation, which proves the effectiveness of the proposed algorithm.

Auto-calibrating photometric stereo using ring light constraints

We propose an auto-calibration method for photometric stereo. Our method exploits constraints placed on light sources to recover their positions and the surface normals of an object up to a scaling and planar rotation ambiguity. The ambiguity is resolved with multi-view consistency constraints leading to a Euclidean reconstruction of the geometric shape of the object. Auto-calibration methods are helpful to overcome the limitations imposed by calibrating objects. We evaluate our algorithm with experiments on real world scenes.

SCAPE-based human performance reconstruction

This paper presents an automatic approach to reconstruct human motion using noisy depth data from multiple views. Although multi-view constraints are provided by this setup, it still exhibits great challenges to robustly reconstruct dynamic human performances due to inherent complexity and self-occlusion of human motion. In the insight that the semantics of human motion will supply strong prior in motion reconstruction, we therefore propose a SCAPE-based motion reconstruction algorithm. As the building blocks of this main algorithm, we (1) re-train a SCAPE model based on an expanded human pose database containing human poses collected from different databases to enlarge the tracking space, (2) develop a correspondence estimation method based on articulated ICP to improve the robustness of SCAPE tracking. We conduct experiments to demonstrate the effectiveness of our method, and show that our system is able to capture and reconstruct accurate human motion.

Multiview phase shifting: a full-resolution and high-speed 3D measurement framework for arbitrary shape dynamic objects

This Letter presents a multiview phase shifting (MPS) framework for full-resolution and high-speed reconstruction of arbitrary shape dynamic objects. Unlike conventional methods, this framework can directly find the corresponding points from the wrapped phase-maps. Therefore, only a minimum number of images are required for phase shifting to measure arbitrary shape objects, including discontinuous surfaces. Benefit from phase shifting MPS can achieve full spatial resolution and high, accurate 3D reconstruction. Benefit from multiview constraint MPS is also robust to discontinuities. Experimental results are presented to verify the performance of the proposed technique.

On Averaging Multiview Relations for 3D Scan Registration

In this paper, we present an extension of the iterative closest point (ICP) algorithm that simultaneously registers multiple 3D scans. While ICP fails to utilize the multiview constraints available, our method exploits the information redundancy in a set of 3D scans by using the averaging of relative motions. This averaging method utilizes the Lie group structure of motions, resulting in a 3D registration method that is both efficient and accurate. In addition, we present two variants of our approach, i.e., a method that solves for multiview 3D registration while obeying causality and a transitive correspondence variant that efficiently solves the correspondence problem across multiple scans. We present experimental results to characterize our method and explain its behavior as well as those of some other multiview registration methods in the literature. We establish the superior accuracy of our method in comparison to these multiview methods with registration results on a set of well-known real datasets of 3D scans.

Surrounding Moving Obstacle Detection for Autonomous Driving Using Stereo Vision

Detection and tracking surrounding moving obstacles such as vehicles and pedestrians are crucial for the safety of mobile robotics and autonomous vehicles. This is especially the case in urban driving scenarios. This paper presents a novel framework for surrounding moving obstacles detection using binocular stereo vision. The contributions of our work are threefold. Firstly, a multiview feature matching scheme is presented for simultaneous stereo correspondence and motion correspondence searching. Secondly, the multiview geometry constraint derived from the relative camera positions in pairs of consecutive stereo views is exploited for surrounding moving obstacles detection. Thirdly, an adaptive particle filter is proposed for tracking of multiple moving obstacles in surrounding areas. Experimental results from real-world driving sequences demonstrate the effectiveness and robustness of the proposed framework.

Convenient calibration method for unsynchronized camera networks using an inaccurate small reference object

In this paper, a new and convenient calibration algorithm is proposed for unsynchronized camera networks with a large capture volume. The proposed method provides a simple and accurate means of calibration using a small 3D reference object. Moreover, since the inaccuracy of the object is also compensated simultaneously, the manufacturing cost can be decreased. The extrinsic and intrinsic parameters are recovered simultaneously by capturing an object placed arbitrarily in different locations in the capture volume. The proposed method first resolves the problem linearly by factorizing projection matrices into the camera and the object pose parameters. Due to the multi-view constraints imposed on factorization, consistency of the rigid transformations among cameras and objects can be imposed. These consistent estimation results can be further refined using a non-linear optimization process. The proposed algorithm is evaluated via simulated and real experiments in order to verify that it is more efficient than previous methods.