Geometric Priors Research Articles

Leveraging Feature Extraction and Context Information for Image Relighting

Example-based image relighting aims to relight an input image to follow the lighting settings of another target example image. Deep learning-based methods for such tasks have become highly popular. However, they are often limited by the geometric priors or suffer from shadow reconstruction and lack of texture details. In this paper, we propose an image-to-image translation network called DGATRN to tackle this problem by enhancing feature extraction and unveiling context information to achieve visually plausible example-based image relighting. Specifically, the proposed DGATRN consists of a scene extraction, a shadow calibration, and a rendering network, and our key contribution lies in the first two networks. We propose an up- and downsampling approach to improve the feature extraction capability to capture scene and texture details better. We also introduce a feature attention downsampling block and a knowledge transfer to utilize the attention impact and underlying knowledge connection between scene and shadow. Experiments were conducted to evaluate the usefulness and effectiveness of the proposed method.

Objformer: Boosting 3D object detection via instance-wise interaction

Deep learning on point clouds drives 3D object detection. Despite rapid progress, point-based methods still suffer from the problems such as incompletion and occlusion, which are caused by the material properties of objects and cluttered scenes. These difficult targets increase the difficulty of identification or even lead to misidentification, severely weakening the performance of point-based methods on 3D object detection. To alleviate the above problems, we propose the Objformer to boost point-based 3D object detection via instance-wise interaction. We design an instance feature encoder to encode clean instance features, which contain key geometric priors and holistic semantic information. Further, an instance interaction module is devised to aggregate the complementary features across instances with label-guided interaction, boosting the performance of the 3D object detection. Experiments show that Objformer outperforms previous point-based state-of-the-arts on two popular benchmarks, ScanNet V2 and SUN RGB-D. Especially, our single-modal Objformer even outperforms the competing advanced multi-modal fusion method on both SUN RGB-D and ScanNet V2.

Neural Radiation Fields in a Tidal Flat Environment

Tidal flats are critical ecosystems, playing a vital role in biodiversity conservation and ecological balance. Collecting tidal flat environmental information using unmanned aerial vehicles (UAVs) and subsequently utilizing 3D reconstruction techniques for their detection and protection holds significance in providing comprehensive and detailed tidal flat information, including terrain, slope, and other parameters. It also enables scientific decision-making for the preservation of tidal flat ecosystems and the monitoring of factors such as rising sea levels. Moreover, the latest advancements in neural radiance fields (Nerf) have provided valuable insights and novel perspectives for our work. We face the following challenges: (1) the performance of a single network is limited due to the vast area to cover; (2) regions far from the camera center may exhibit suboptimal rendering results; and (3) changes in lighting conditions present challenges for the achievement of precise reconstruction. To tackle these challenges, we partitioned the tidal flat scene into distinct submodules, carefully preserving overlapping regions between each submodule for collaborative optimization. The luminance of each image is quantified by the appearance embedding vector produced by every captured image. Subsequently, this corresponding vector serves as an input to the model, enhancing its performance across varying lighting conditions. We also introduce an ellipsoidal sphere transformation that brings distant image elements into the sphere’s interior, enhancing the algorithm’s capacity to represent remote image information. Our algorithm is validated using tidal plane images collected from UAVs and compared with traditional Nerf based on two metrics: peak signal-to-noise ratio (PSNR) and learned perceptual image patch similarity (LPIPS). Our method enhances the PSNR value by 2.28 and reduces the LPIPS value by 0.11. The results further demonstrate that our approach significantly enhances Nerf’s performance in tidal flat environments. Utilizing Nerf for the 3D reconstruction of tidal flats, we bypass the need for explicit representation and geometric priors. This innovative approach yields superior novel view synthesis and enhances geometric perception, resulting in high-quality reconstructions. Our method not only provides valuable data but also offers profound insights for environmental monitoring and management.

Single-image piecewise planar reconstruction of urban buildings based on geometric priors

Single-image 3D urban building reconstruction is an important and challenging topic in computer vision. However, it is a highly ill-posed problem due to the intrinsic geometrical ambiguity in a 3D space. To address the problem, this paper presents a novel single-image piecewise planar reconstruction method based on geometric priors. The proposed method utilizes geometric priors (e.g., plane orientation and intersection angles between planes) learned through convolutional neural networks to handle three challenging subtasks sequentially: (1) detecting planar regions with structure information (e.g., each component edge is associated with two planes intersecting at a specified angle); (2) inferring the planes corresponding to the planar regions progressively by taking full advantage of diverse constraints (e.g., structure and proximity constraints) based on geometric priors; and (3) globally optimizing the resulting planes under a unified framework that incorporates image cues and geometric priors. Experimental results on two public datasets released by CASIA and our own dataset demonstrate that the proposed method significantly outperforms five state-of-the-art methods (including two geometry-based methods and three learning-based methods) by an approximately 20 % margin in terms of accuracy. Code and data are available at https://github.com/three-dimensional-computer-vision/single-image-reconstruction.

Remote Sensing Neural Radiance Fields for Multi-View Satellite Photogrammetry

Neural radiance fields (NeRFs) combining machine learning with differentiable rendering have arisen as one of the most promising approaches for novel view synthesis and depth estimates. However, NeRFs only applies to close-range static imagery and it takes several hours to train the model. The satellites are hundreds of kilometers from the earth. Satellite multi-view images are usually captured over several years, and the scene of images is dynamic in the wild. Therefore, multi-view satellite photogrammetry is far beyond the capabilities of NeRFs. In this paper, we present a new method for multi-view satellite photogrammetry of Earth observation called remote sensing neural radiance fields (RS-NeRFs). It aims to generate novel view images and accurate elevation predictions quickly. For each scene, we train an RS-NeRF using high-resolution optical images without labels or geometric priors and apply image reconstruction losses for self-supervised learning. Multi-date images exhibit significant changes in appearance, mainly due to cars and varying shadows, which brings challenges to satellite photogrammetry. Robustness to these changes is achieved by the input of solar ray direction and the vehicle removal method. NeRFs make it intolerable by requiring a very long time to train an easy scene. In order to significantly reduce the training time of RS-NeRFs, we build a tiny network with HashEncoder and adopted a new sampling technique with our custom CUDA kernels. Compared with previous work, our method performs better on novel view synthesis and elevation estimates, taking several minutes.

Interactive piecewise planar building reconstruction from a single image based on geometric priors

The development of methods to reconstruct the piecewise planar structures of buildings from a single image is an important and challenging problem in the field of computer vision. Although most prior works have considered fully automatic reconstruction solutions, these methods typically perform poorly for complex building structures. To address this problem, we propose an interactive single-image 3D building reconstruction method to integrate automatic plane inference with interactive plane refinement based on geometric priors. In this method, based on geometric priors (e.g., the orientation of the plane and the intersection angle between planes) learned using convolutional neural networks, building regions are first partitioned into multiple polygonal regions using click interactions. Then, the planes corresponding to these polygonal regions are automatically and progressively inferred by taking full advantage of the constraints constructed using the geometric priors. Finally, under the guidance of interactive plane refinement based on click correlations, the resulting planes are globally optimized by jointly utilizing image cues and geometric priors. The results of an experimental evaluation on three datasets demonstrate that the proposed method achieved better performance than existing methods in terms of both qualitative and quantitative measures.

GeoGlue: feature matching with self-supervised geometric priors for high-resolution UAV images

ABSTRACT We present GeoGlue, a novel method using high-resolution UAV imagery for accurate feature matching, which is normally challenging due to the complicated scenes. Current feature detection methods are performed without guidance of geometric priors (e.g., geometric lines), lacking enough attention given to salient geometric features which are indispensable for accurate matching due to their stable existence across views. In this work, geometric lines are firstly detected by a CNN-based geometry detector (GD) which is pre-trained in a self-supervised manner through automatically generated images. Then, geometric lines are naturally vectorized based on GD and thus non-significant features can be disregarded as judged by their disordered geometric morphology. A graph attention network (GAT) is utilized for final feature matching, spanning across the image pair with geometric priors informed by GD. Comprehensive experiments show that GeoGlue outperforms other state-of-the-art methods in feature-matching accuracy and performance stability, achieving pose estimation with maximum rotation and translation errors under 1% in challenging scenes from benchmark datasets, Tanks & Temples and ETH3D. This study also proposes the first self-supervised deep-learning model for curved line detection, generating geometric priors for matching so that more attention is put on prominent features and improving the visual effect of 3D reconstruction.

ASPPR: active single-image piecewise planar 3D reconstruction based on geometric priors

ASPPR: active single-image piecewise planar 3D reconstruction based on geometric priors

3D Deformation Capture via a Configurable Self-Sensing IMU Sensor Network

Motion capture technologies reconstruct human movements and have wide-ranging applications. Mainstream research on motion capture can be divided into vision-based methods and inertial measurement unit (IMU)-based methods. The vision-based methods capture complex 3D geometrical deformations with high accuracy, but they rely on expensive optical equipment and suffer from the line-of-sight occlusion problem. IMU-based methods are lightweight but hard to capture fine-grained 3D deformations. In this work, we present a configurable self-sensing IMU sensor network to bridge the gap between the vision-based and IMU-based methods. To achieve this, we propose a novel kinematic chain model based on the four-bar linkage to describe the minimum deformation process of 3D deformations. We also introduce three geometric priors, obtained from the initial shape, material properties and motion features, to assist the kinematic chain model in reconstructing deformations and overcome the data sparsity problem. Additionally, to further enhance the accuracy of deformation capture, we propose a fabrication method to customize 3D sensor networks for different objects. We introduce origami-inspired thinking to achieve the customization process, which constructs 3D sensor networks through a 3D-2D-3D digital-physical transition. The experimental results demonstrate that our method achieves comparable performance with state-of-the-art methods.

Neural 3D reconstruction from sparse views using geometric priors

Sparse view 3D reconstruction has attracted increasing attention with the development of neural implicit 3D representation. Existing methods usually only make use of 2D views, requiring a dense set of input views for accurate 3D reconstruction. In this paper, we show that accurate 3D reconstruction can be achieved by incorporating geometric priors into neural implicit 3D reconstruction. Our method adopts the signed distance function as the 3D representation, and learns a generalizable 3D surface reconstruction model from sparse views. Specifically, we build a more effective and sparse feature volume from the input views by using corresponding depth maps, which can be provided by depth sensors or directly predicted from the input views. We recover better geometric details by imposing both depth and surface normal constraints in addition to the color loss when training the neural implicit 3D representation. Experiments demonstrate that our method both outperforms state-of-the-art approaches, and achieves good generalizability.

Reconstructing Floorplans from Point Clouds Using GAN

This paper proposed a method for reconstructing floorplans from indoor point clouds. Unlike existing corner and line primitive detection algorithms, this method uses a generative adversarial network to learn the complex distribution of indoor layout graphics, and repairs incomplete room masks into more regular segmentation areas. Automatic learning of the structure information of layout graphics can reduce the dependence on geometric priors, and replacing complex optimization algorithms with Deep Neural Networks (DNN) can improve the efficiency of data processing. The proposed method can retain more shape information from the original data and improve the accuracy of the overall structure details. On this basis, the method further used an edge optimization algorithm to eliminate pixel-level edge artifacts that neural networks cannot perceive. Finally, combined with the constraint information of the overall layout, the method can generate compact floorplans with rich semantic information. Experimental results indicated that the algorithm has robustness and accuracy in complex 3D indoor datasets; its performance is competitive with those of existing methods.

Geometric deep learning reveals the spatiotemporal features of microscopic motion

The characterization of dynamical processes in living systems provides important clues for their mechanistic interpretation and link to biological functions. Owing to recent advances in microscopy techniques, it is now possible to routinely record the motion of cells, organelles and individual molecules at multiple spatiotemporal scales in physiological conditions. However, the automated analysis of dynamics occurring in crowded and complex environments still lags behind the acquisition of microscopic image sequences. Here we present a framework based on geometric deep learning that achieves the accurate estimation of dynamical properties in various biologically relevant scenarios. This deep-learning approach relies on a graph neural network enhanced by attention-based components. By processing object features with geometric priors, the network is capable of performing multiple tasks, from linking coordinates into trajectories to inferring local and global dynamic properties. We demonstrate the flexibility and reliability of this approach by applying it to real and simulated data corresponding to a broad range of biological experiments.

HiDAnet: RGB-D Salient Object Detection via Hierarchical Depth Awareness

RGB-D saliency detection aims to fuse multi-modal cues to accurately localize salient regions. Existing works often adopt attention modules for feature modeling, with few methods explicitly leveraging fine-grained details to merge with semantic cues. Thus, despite the auxiliary depth information, it is still challenging for existing models to distinguish objects with similar appearances but at distinct camera distances. In this paper, from a new perspective, we propose a novel Hierarchical Depth Awareness network (HiDAnet) for RGB-D saliency detection. Our motivation comes from the observation that the multi-granularity properties of geometric priors correlate well with the neural network hierarchies. To realize multi-modal and multi-level fusion, we first use a granularity-based attention scheme to strengthen the discriminatory power of RGB and depth features separately. Then we introduce a unified cross dual-attention module for multi-modal and multi-level fusion in a coarse-to-fine manner. The encoded multi-modal features are gradually aggregated into a shared decoder. Further, we exploit a multi-scale loss to take full advantage of the hierarchical information. Extensive experiments on challenging benchmark datasets demonstrate that our HiDAnet performs favorably over the state-of-the-art methods by large margins. The source code can be found in https://github.com/Zongwei97/HIDANet/.

RSG-GCN: Predicting Semantic Relationships in Urban Traffic Scene With Map Geometric Prior

Automated identification of the relationships between traffic actors and surrounding objects, in order to describe their behavior and predict their intentions, has become the focus of increasing attention in the field of autonomous driving. Therefore, in this work, we propose a Road Scene Graphs-Graph Convolutional Network (RSG-GCN) as a novel, graph-based model for predicting the topological graph structure of a given traffic scene. The status of the actors and HD map information are integrated as prior knowledge, allowing the edges linking the actor nodes to capture potential semantic relationships, such as “vehicle approaching pedestrian” and “pedestrian waiting at intersection”. To train this model, we created our own RSG dataset, as well as a relational dataset and benchmark derived from nuScenes. Our extensive range of experiments demonstrate that our model can more accurately predict semantic relationships and behavior in a given traffic scene than other popular traffic scene prediction models. In particular, regarding the use of HD map prior knowledge, we found that the resulting increase in accuracy significantly outweighs performance loss caused by the increase in graph size. The downstream applications of RSG include traffic scene retrieval and synthetic traffic scene generation, which are briefly described.

NeuralRoom

We present a novel neural surface reconstruction method called NeuralRoom for reconstructing room-sized indoor scenes directly from a set of 2D images. Recently, implicit neural representations have become a promising way to reconstruct surfaces from multiview images due to their high-quality results and simplicity. However, implicit neural representations usually cannot reconstruct indoor scenes well because they suffer severe shape-radiance ambiguity. We assume that the indoor scene consists of texture-rich and flat texture-less regions. In texture-rich regions, the multiview stereo can obtain accurate results. In the flat area, normal estimation networks usually obtain a good normal estimation. Based on the above observations, we reduce the possible spatial variation range of implicit neural surfaces by reliable geometric priors to alleviate shape-radiance ambiguity. Specifically, we use multiview stereo results to limit the NeuralRoom optimization space and then use reliable geometric priors to guide NeuralRoom training. Then the NeuralRoom would produce a neural scene representation that can render an image consistent with the input training images. In addition, we propose a smoothing method called perturbation-residual restrictions to improve the accuracy and completeness of the flat region, which assumes that the sampling points in a local surface should have the same normal and similar distance to the observation center. Experiments on the ScanNet dataset show that our method can reconstruct the texture-less area of indoor scenes while maintaining the accuracy of detail. We also apply NeuralRoom to more advanced multiview reconstruction algorithms and significantly improve their reconstruction quality.

Visual Attention-Based Self-Supervised Absolute Depth Estimation Using Geometric Priors in Autonomous Driving

Although existing monocular depth estimation methods have made great progress, predicting an accurate absolute depth map from a single image is still challenging due to the limited modeling capacity of networks and the scale ambiguity issue. In this paper, we introduce a fully Visual Attention-based Depth (VADepth) network, where spatial attention and channel attention are applied to all stages. By continuously extracting the dependencies of features along the spatial and channel dimensions over a long distance, VADepth network can effectively preserve important details and suppress interfering features to better perceive the scene structure for more accurate depth estimates. In addition, we utilize geometric priors to form scale constraints for scale-aware model training. Specifically, we construct a novel scale-aware loss using the distance between the camera and a plane fitted by the ground points corresponding to the pixels of the rectangular area in the bottom middle of the image. Experimental results on the KITTI dataset show that this architecture achieves the state-of-the-art performance and our method can directly output absolute depth without post-processing. Moreover, our experiments on the SeasonDepth dataset also demonstrate the robustness of our model to multiple unseen environments.

Neural Scene Representation for Locomotion on Structured Terrain

We propose a learning-based method to reconstruct the local terrain for locomotion with a mobile robot traversing urban environments. Using a stream of depth measurements from the onboard cameras and the robot’s trajectory, the algorithm estimates the topography in the robot’s vicinity. The raw measurements from these cameras are noisy and only provide partial and occluded observations that in many cases do not show the terrain the robot stands on. Therefore, we propose a 3D reconstruction model that faithfully reconstructs the scene, despite the noisy measurements and large amounts of missing data coming from the blind spots of the camera arrangement. The model consists of a 4D fully convolutional network on point clouds that learns the geometric priors to complete the scene from the context and an auto-regressive feedback to leverage spatio-temporal consistency and use evidence from the past. The network can be solely trained with synthetic data, and due to extensive augmentation, it is robust in the real world, as shown in the validation on a quadrupedal robot, ANYmal, traversing challenging settings. We run the pipeline on the robot’s onboard low-power computer using an efficient sparse tensor implementation and show that the proposed method outperforms classical map representations.

JSL3d: Joint subspace learning with implicit structure supervision for 3D pose estimation

Estimating 3D human poses from a single image is an important task in computer graphics. Most model-based estimation methods represent the labeled/detected 2D poses and the projection of approximated 3D poses using vector representations of body joints. However, such lower-dimensional vector representations fail to maintain the spatial relations of original body joints, because the representations do not consider the inherent structure of body joints. In this paper, we propose JSL3d, a novel joint subspace learning approach with implicit structure supervision based on Sparse Representation (SR) model, capturing the latent spatial relations of 2D body joints by an end-to-end autoencoder network. JSL3djointly combines the learned latent spatial relations and 2D joints as inputs for the standard SR inference frame. The optimization is simultaneously processed via geometric priors in both latent and original feature spaces. We have evaluated JSL3dusing four large-scale and well-recognized benchmarks, including Human3.6M, HumanEva-I, CMU MoCap and MPII. The experiment results demonstrate the effectiveness of JSL3d.

Real time volumetric MRI for 3D motion tracking via geometry-informed deep learning.

To develop a geometry-informed deep learning framework for volumetric MRI with sub-second acquisition time in support of 3D motion tracking, which is highly desirable for improved radiotherapy precision but hindered by the long image acquisition time. A 2D-3D deep learning network with an explicitly defined geometry module that embeds geometric priors of the k-space encoding pattern was investigated, where a 2D generation network first augmented the sparsely sampled image dataset by generating new 2D representations of the underlying 3D subject. A geometry module then unfolded the 2D representations to the volumetric space. Finally, a 3D refinement network took the unfolded 3D data and outputted high-resolution volumetric images. Patient-specific models were trained for seven abdominal patients to reconstruct volumetric MRI from both orthogonal cine slices and sparse radial samples. To evaluate the robustness of the proposed method to longitudinal patient anatomy and position changes, we tested the trained model on separate datasets acquired more than one month later and evaluated 3D target motion tracking accuracy using the model-reconstructed images by deforming a reference MRI with gross tumor volume (GTV) contours to a 5-min time series of both ground truth and model-reconstructed volumetric images with a temporal resolution of 340ms. Across the seven patients evaluated, the median distances between model-predicted and ground truth GTV centroids in the superior-inferior direction were 0.4 ± 0.3mm and 0.5 ± 0.4mm for cine and radial acquisitions, respectively. The 95-percentile Hausdorff distances between model-predicted and ground truth GTV contours were 4.7 ± 1.1mm and 3.2 ± 1.5mm for cine and radial acquisitions, which are of the same scale as cross-plane image resolution. Incorporating geometric priors into deep learning model enables volumetric imaging with high spatial and temporal resolution, which is particularly valuable for 3D motion tracking and has the potential of greatly improving MRI-guided radiotherapy precision.

A geometry-informed deep learning framework for ultra-sparse 3D tomographic image reconstruction

Deep learning affords enormous opportunities to augment the armamentarium of biomedical imaging. However, the pure data-driven nature of deep learning models may limit the model generalizability and application scope. Here we establish a geometry-informed deep learning framework for ultra-sparse 3D tomographic image reconstruction. We introduce a novel mechanism for integrating geometric priors of the imaging system. We demonstrate that the seamless inclusion of known priors is essential to enhance the performance of 3D volumetric computed tomography imaging with ultra-sparse sampling. The study opens new avenues for data-driven biomedical imaging and promises to provide substantially improved imaging tools for various clinical imaging and image-guided interventions.