Surface Normal Estimation Research Articles

Mitigating RGB-D camera errors for robust ultrasonic inspections using a force-torque sensor

ABSTRACT Robot-based phased array ultrasonic testing is widely used for precise defect detection, particularly in complex geometries and various materials. Compact robots with miniature arms can inspect constrained areas, but payload limitations restrict sensor choice. RGB-D cameras, due to their small size and light weight, capture RGB colour and depth data, creating colourised 3D point clouds for scene representation. These point clouds help estimate surface normals to align the ultrasound transducer on complex surfaces. However, sole reliance on RGB-D cameras can lead to inaccuracies, affecting ultrasonic beam direction and test results. This paper investigates the impact of transducer pose and RGB-D camera limitations on ultrasonic inspections and proposes a novel method using force-torque sensors to mitigate errors from inaccurately estimated normals from the camera. The force-torque sensor, integrated into the robot end effector, provides tactile feedback to the controller, enabling joint angle adjustments to correct errors in the estimated normal. Experimental results show the successful application of ultrasound transducers using this method, even with significant misalignment. Adjustments took approximately 4 seconds to correct deviations from 12.55°, with an additional 4 seconds to ensure the probe was parallel to the surface, enhancing ultrasonic inspection accuracy in complex, constrained environments.

A Scale-space Approach for Surface Normal Vector Estimation from Depth Maps

Surface normal vectors provide cues about the local geometric features of the scene which are utilized in many computer vision and computer graphics applications. Thus, the estimation of surface normals by utilizing structured range sensor data is an important research field. Thereupon, we propose a learning-free algorithm to estimate the surface normal vectors from depth maps. Our simple yet effective method relies on computations carried out in scale-space. Our main idea is to estimate the surface normals which cannot be properly computed in the finest scale from the coarser scales. Our method can estimate the surface normals even for images included in datasets that have challenging characteristics such as noisy real-world data or significantly large planar regions that either have a small or no gradient change. We analyze our algorithm’s performance by utilizing five benchmarks, namely, the MIT-Berkeley Intrinsic Images dataset, the New Tsukuba Dataset, the SceneNet RGB-D dataset, the IID-NORD dataset, and the NYU Depth Dataset V2, and by using two different evaluation strategies. According to the experimental results, our method can estimate surface normals efficiently without requiring neither complex computations nor huge amounts of data.

Monocular Depth and Ego-motion Estimation with Scale Based on Superpixel and Normal Constraints

3D perception in intelligent VR/AR and autonomous vehicles (AV) applications is critical and attracting significant attention. The self-supervised monocular depth and ego-motion estimation serves as a more intelligent learning approach that provides the required scene depth and location for 3D perception. However, the existing self-supervised learning methods suffer from scale ambiguity, boundary blur and imbalanced depth distribution, limiting the practical applications of VR/AR and AV. In this paper, we propose a new self-supervised learning framework based on superpixel and normal constraints to address these problems. Specifically, we formulate a novel 3D edge structure consistency loss to alleviate the boundary blur of depth estimation. To address the scale ambiguity of estimated depth and ego-motion, we propose a novel surface normal network for efficient camera height estimation. The surface normal network is composed of a deep fusion module and a full-scale hierarchical feature aggregation module. Meanwhile, to realize the global smoothing and boundary discriminability of the predicted normal map, we introduce a novel fusion loss which is based on the consistency constraints of the normal in edge domains and superpixel regions. Experiments are conducted on several benchmarks, and the results illustrate that the proposed approach outperforms the state-of-the-art methods in depth, ego-motion and surface normal estimation.

High-Fidelity Specular SVBRDF Acquisition From Flash Photographs.

Obtaining accurate SVBRDFs from 2D photographs of shiny, heterogeneous 3D objects is a highly sought-after goal for domains like cultural heritage archiving, where it is critical to document color appearance in high fidelity. In prior work such as the promising framework by Nam et al., the problem is simplified by assuming that specular highlights exhibit symmetry and isotropy about an estimated surface normal. The present work builds on this foundation with several significant modifications. Recognizing the importance of the surface normal as an axis of symmetry, we compare nonlinear optimization for normals with a linear approximation proposed by Nam et al. and find that nonlinear optimization is superior to the linear approximation, while noting that the surface normal estimates generally have a very significant impact on the reconstructed color appearance of the object. We also examine the use of a monotonicity constraint for reflectance and develop a generalization that also enforces continuity and smoothness when optimizing continuous monotonic functions like a microfacet distribution. Finally, we explore the impact of simplifying from an arbitrary 1D basis function to a traditional parametric microfacet distribution (GGX), and we find this to be a reasonable approximation that trades some fidelity for practicality in certain applications. Both representations can be used in existing rendering architectures like game engines or online 3D viewers, while retaining accurate color appearance for fidelity-critical applications like cultural heritage or online sales.

ResNet-based surface normal estimator with multilevel fusion approach with adaptive median filter region growth algorithm for road scene segmentation

ResNet-based surface normal estimator with multilevel fusion approach with adaptive median filter region growth algorithm for road scene segmentation

Wide-Baseline Light Fields using Ellipsoidal Mirrors.

Traditional hand-held light field cameras only observe a small fraction of the cone of light emitted by a scene point. As a consequence, the study of interesting angular effects like iridescence are beyond the scope of such cameras. This paper envisions a new design for sensing light fields with wide baselines, so as to sense a significantly larger fraction of the cone of light emitted by scene points. Our system achieves this by imaging the scene, indirectly, through an ellipsoidal mirror. We show that an ellipsoidal mirror maps a wide cone of light from locations near one of its foci to a narrower cone at its other focus; thus, by placing a conventional light field camera at a focus, we can observe a wide-baseline light field from the scene near the other focus. We show via simulations and a lab prototype that wide-baseline light fields excel in the traditional applications involving changes in focus and perspective. Additionally, the larger cone of light that they observe allows the study of iridescence and thin-film interference. Perhaps surprisingly, the larger cone of light allows us to estimate surface normals of scene points by reasoning about their visibility.

Deep Semi-calibrated Near-field Photometric Stereo

Light calibration plays a crucial role in near-field photometric stereo for accurate surface normal estimation. It comprises light position calibration and light intensity calibration. In industrial applications, the light position calibration is typically performed once, as the facility remains fixed. However, the light intensity calibration needs to be repeated for real industrial scenes with variations in light intensity and fails for autoexposure industrial scenes. This study introduces a deep semi-calibrated near-field photometric stereo technique that achieves precise surface normal estimation without light intensity calibration. The technique consists of two parts: the near-field light intensity calibration network (NFLICN) and the near-field normal estimation network (NFNEN). NFLICN utilizes a point light model to self-calibrate pixel-wise light intensity. NFNEN, on the other hand, takes the calibrated light intensity as input and effectively utilizes local image features to estimate the surface normal. Experimental results on both synthetic testing datasets and the benchmark LUCES dataset confirm the state-of-the-art accuracy of our proposed method in surface normal estimation, even without light intensity calibration.

Three Dimensional Shape Reconstruction via Polarization Imaging and Deep Learning

Deep-learning-based polarization 3D imaging techniques, which train networks in a data-driven manner, are capable of estimating a target’s surface normal distribution under passive lighting conditions. However, existing methods have limitations in restoring target texture details and accurately estimating surface normals. Information loss can occur in the fine-textured areas of the target during the reconstruction process, which can result in inaccurate normal estimation and reduce the overall reconstruction accuracy. The proposed method enables extraction of more comprehensive information, mitigates the loss of texture information during object reconstruction, enhances the accuracy of surface normal estimation, and facilitates more comprehensive and precise reconstruction of objects. The proposed networks optimize the polarization representation input by utilizing the Stokes-vector-based parameter, in addition to separated specular and diffuse reflection components. This approach reduces the impact of background noise, extracts more relevant polarization features of the target, and provides more accurate cues for restoration of surface normals. Experiments are performed using both the DeepSfP dataset and newly collected data. The results show that the proposed model can provide more accurate surface normal estimates. Compared to the UNet architecture-based method, the mean angular error is reduced by 19%, calculation time is reduced by 62%, and the model size is reduced by 11%.

Multi-stage information diffusion for joint depth and surface normal estimation

Depth and surface normal estimations are important for 3D geometric perception, which has numerous applications including autonomous vehicles and robots. In this paper, we propose a lightweight Multi-stage Information Diffusion Network (MIDNet) for the simultaneous prediction of depth and surface normals from a single RGB image. To obtain semantic and detail-preserving features, we adopt a high-resolution network as our backbone to learn multi-scale features, which are then fused into shared features for the two tasks. To mutually boost each task, a Cross-Correlation Attention Module (CCAM) is proposed to adaptively integrate information for the prediction of the two tasks in multiple stages, including feature-level information interaction and task-level information interaction. Ablation studies show that the proposed multi-stage information diffusion strategy can boost the performance gain for the two tasks at different levels. Compared to current state-of-the-art methods on the NYU Depth V2, Stanford 2D-3D-Semantic and KITTI datasets, our method achieves superior performance for both monocular depth and surface normal estimation.

Geometry-guided multilevel RGBD fusion for surface normal estimation

Developments in 3D computer vision have advanced scene understanding and 3D modeling. Surface normal estimation is a basic task in these fields. In this paper, we propose a geometry-guided multilevel fusion scheme for high-quality surface normal estimation by exploiting texture and geometry information from color and depth images. The surface normal is progressively predicted with a coarse-to-fine strategy. First, an initial surface normal (IniNormal) Nini is predicted by a hierarchical confidence reweighting convolution neural network to merge texture and geometry information in a CNN feature level. Although a general accuracy is achieved, the long tail problem makes the IniNormal always fails in special areas where the depth map is high-quality while the intensity interference is challenging, such as repeating textures and abnormal exposures. Further, a traditional geometry-consistent based surface normal(GeoNormal) Ngeo is calculated based on traditional constraints, and a surface normal level fusion module is designed to remap the depth to different representations and reconsider scene information. Then, the final clear surface normal N is estimated by adaptively reintegrating the IniNormal and GeoNormal in a decision level. To overcome disturbances in the dataset and ensure the trainability of the network, a carefully designed hybrid objective function and an annealed term are applied. An explainable analysis is attached. The experimental results on two benchmark datasets demonstrate that the proposed GMLF(geometry-guided multilevel RGBD fusion for surface normal estimation) can achieve better quantitative and qualitative performance. The proposed method may be useful for robots and auto-driving which can be applied in the next-generation Internet-of-Things (NG-IoT).

Tube-Embedded Transformer for Pixel Prediction

Multi-task pixel-level learning, which aims to exploit the inter-task interactions to improve the learning of each task, is an important but challenging issue in visual reception and multimedia applications. Measuring the inter-task correlation and intra-task specificity, we propose a tube-embedded transformer (TET) framework for robust multi-task pixel prediction. To facilitate inter-task interactions, we aggregate and project all tasks into a shared tube pool to generate the latent multi-task representation during the coarse-to-fine decoding stages. The resulting task-tube interactions replace the two-by-two task-task interactions to reduce the model complexity significantly. In addition, we introduce the transformer mechanism to adaptively transfer tube features to the target task. Concretely, on the one hand, multi-task features aggregate in the tube to generate the shared feature representation bases; on the other hand, based on the task-tube association and complementarity, the tube outputs the query entry and the weighting coefficients of the target task. Experimentally, on the joint learning of semantic segmentation, depth estimation, and surface normal estimation, the comparison experiments show the superiority of the TET multi-task learning method over other state-of-the-art approaches, and the ablation experiments verify the effectiveness of the TET mechanism.

Automatic Extrinsic Calibration of Dual LiDARs With Adaptive Surface Normal Estimation

Solutions equipped with multiple light detection and ranging (LiDAR) systems have been widely used in several fields including mobile mapping, navigation, robot, and others. Accurate and robust extrinsic calibration between multiple scanners is necessary for the integration of point cloud data. An automatic method for the calibration of dual LiDARs with adaptive surface normal estimation is presented in this article. Specifically, this approach begins with environment detection from different positions and attitudes. A novel surface normal estimation method is conducted to take account of the uneven distribution of point cloud density and the edge information of planes. Finally, the calibration parameters are calculated by iteratively minimizing the cost function that consists of implicit point-to-plane distances. The experimental results on simulation and real-world data demonstrate that for different types of LiDAR, the proposed algorithm can achieve high-accuracy calibration in different scenes, without manual intervention. The rotation and translation calibration errors between Velodyne LiDARs are less than 1° and 0.02 m, respectively.

Human Pose and Shape Estimation From Single Polarization Images

This paper focuses on a new problem of estimating human pose and shape from single polarization images. Polarization camera is known to be able to capture the polarization of reflected lights that preserves rich geometric cues of an object surface. Inspired by the recent applications in surface normal reconstruction from polarization images, in this paper, we attempt to estimate human pose and shape from single polarization images by leveraging the polarization-induced geometric cues. A dedicated two-stage pipeline is proposed: given a single polarization image, stage one (Polar2Normal) focuses on the fine detailed human body surface normal estimation; stage two (Polar2Shape) then reconstructs clothed human shape from the polarization image and the estimated surface normal. To empirically validate our approach, a dedicated dataset (PHSPD) is constructed, consisting of over 500K frames with accurate pose and parametric shape annotations. Empirical evaluations on this real-world dataset as well as a synthetic dataset, SURREAL, demonstrate the effectiveness of our approach. It suggests polarization camera as a promising alternative to the more conventional RGB camera for human pose and shape estimation.

Self-Calibrating Sparse Far-Field Photometric Stereo With Collocated Light

Photometric stereo is one common surface normal estimation technique, which can be utilized for various industrial tasks such as 3-D surface detail reconstruction and defect inspection. To further satisfy the requirement of the industrial inspection, the uncalibrated photometric stereo is needed. This article proposes a collocated light method for uncalibrated far-field photometric stereo to achieve accurate surface normal estimation under a small number of lights. Collocated light is the light close to the camera, with the angle between the lighting direction of the collocated light and the viewing direction of the camera being less than 10°. The image under the collocated light could be utilized to extract features related to the shadings and the shadows that are the key to the light self-calibration. Thus, the collocated light is introduced to realize the light self-calibration, with the accuracy being less influenced by the number of lights. Then, the normal estimation network with collocated light is integrated to give precise surface normal estimations under a small number of lights. Both synthetic and real experiments validate that the proposed method can achieve the state-of-the-art performance.

ResNet-based surface normal estimator with multilevel fusion approach with adaptive median filter region growth algorithm for road scene segmentation

ResNet-based surface normal estimator with multilevel fusion approach with adaptive median filter region growth algorithm for road scene segmentation

Discrete Search Photometric Stereo for Fast and Accurate Shape Estimation.

We consider the problem of estimating surface normals of a scene with spatially varying, general bidirectional reflectance distribution functions (BRDFs) observed by a static camera under varying distant illuminations. Unlike previous approaches that rely on continuous optimization of surface normals, we cast the problem as a discrete search problem over a set of finely discretized surface normals. In this setting, we show that the expensive processes can be precomputed in a scene-independent manner, resulting in accelerated inference. We discuss two variants of our discrete search photometric stereo (DSPS), one working with continuous linear combinations of BRDF bases and the other working with discrete BRDFs sampled from a BRDF space. Experiments show that DSPS has comparable accuracy to state-of-the-art exemplar-based photometric stereo methods while achieving 10-100x acceleration.

3D Fourier ghost imaging via semi-calibrated photometric stereo.

We achieved three-dimensional (3D) computational ghost imaging with multiple photoresistors serving as single-pixel detectors using the semi-calibrated lighting approach. We performed imaging in the spatial frequency domain by having each photoresistor obtain the Fourier spectrum of the scene at a low spectral coverage ratio of 5%. To retrieve a depth map of a scene, we inverted, apodized, and applied semi-calibrated photometric stereo (SCPS) to the spectra. At least 93.5% accuracy was achieved for the 3D results of the apodized set of images applied with SCPS in comparison with the ground truth. Furthermore, intensity error map statistics obtained at least 97.0% accuracy for the estimated surface normals using our method. Our system does not need special calibration objects or any additional optical components to perform accurate 3D imaging, making it easily adaptable. Our method can be applied in current imaging systems where multiple detectors operating at any wavelength are used for two-dimensional (2D) imaging, such as imaging cosmological objects. Employing the idea of changing light patterns to illuminate a target scene and having stored information about these patterns, the data retrieved by one detector will give the 2D information while the multiple-detector system can be used to get a 3D profile.

Geometry Guided Deep Surface Normal Estimation

We propose a geometry-guided neural network architecture for robust and detail-preserving surface normal estimation for unstructured point clouds. Previous deep normal estimators usually estimate the normal directly from the neighbors of a query point, which lead to poor performance. The proposed network is composed of a weight learning sub-network (WL-Net) and a lightweight normal learning sub-network (NL-Net). WL-Net first predicates point-wise weights for generating an optimized point set (OPS) from the input. Then, NL-Net estimates a more accurate normal from the OPS especially when the local geometry is complex. To boost the weight learning ability of the WL-Net, we introduce two geometric guidance in the network. First, we design a weight guidance using the deviations between the neighbor points and the ground truth tangent plane of the query point. This deviation guidance offers a “ground truth” for weights corresponding to some reliable inliers and outliers determined by the tangent plane. Second, we integrate the normals of multiple scales into the input. Its performance and robustness are further improved without relying on multi-branch networks, which are employed in previous multi-scale normal estimators. Thus our method is more efficient. Qualitative and quantitative evaluations demonstrate the advantages of our approach over the state-of-the-art methods, in terms of estimation accuracy, model size and inference time. Code is available at https://github.com/2429581027/local-geometric-guided.

Surface normal estimation from polarization and shading under the convexity assumption

Surface normal estimation from polarization and shading under the convexity assumption

Three-Filters-to-Normal: An Accurate and Ultrafast Surface Normal Estimator

This paper proposes three-filters-to-normal (3F2N), an accurate and ultrafast surface normal estimator (SNE), which is designed for structured range sensor data, e.g., depth/disparity images. 3F2N SNE computes surface normals by simply performing three filtering operations (two image gradient filters in horizontal and vertical directions, respectively, and a mean/median filter) on an inverse depth image or a disparity image. Despite the simplicity of 3F2N SNE, no similar method already exists in the literature. To evaluate the performance of our proposed SNE, we created three large-scale synthetic datasets (easy, medium and hard) using 24 3D mesh models, each of which is used to generate 1800--2500 pairs of depth images (resolution: 480X640 pixels) and the corresponding ground-truth surface normal maps from different views. 3F2N SNE demonstrates the state-of-the-art performance, outperforming all other existing geometry-based SNEs, where the average angular errors with respect to the easy, medium and hard datasets are 1.66 degrees, 5.69 degrees and 15.31 degrees, respectively. Furthermore, our C++ and CUDA implementations achieve a processing speed of over 260 Hz and 21 kHz, respectively. Our datasets and source code are publicly available at sites.google.com/view/3f2n.