3D Depth Map Research Articles

Automated body measurement of beef cattle based on keypoint detection and local point cloud clustering

Abstract Body size parameters of beef cattle are crucial for assessing growth status and breeding value. In actual farming environments, the various postures of beef cattle and complex backgrounds can affect the accuracy and stability of non-contact body measurement methods. Therefore, this paper proposes a novel method called the cattle body measurement method (CBMM), which combines keypoint detection with local point cloud clustering. First, a keypoint detection model based on YOLOv8-SimBiFPN is constructed. This model enhances the feature extraction and fusion capabilities of YOLOv8-pose by introducing SimAM and BiFPN into the backbone and neck networks, respectively, and realizes 2D keypoint detection for beef cattle in various postures. Second, a 3D keypoint-locating algorithm based on Density-based spatial clustering of applications with noise (DBSCAN) is proposed. This algorithm utilizes 2D keypoints, depth maps and camera parameters to generate local point clouds, which are then clustered using DBSCAN to segment cattle body point clouds, thereby relocating the 3D keypoints based on their positional features. Finally, body size parameters are calculated based on the 3D keypoints and distance formulae. In our experiment, the mean average precision (mAP@0.5) of YOLOv8-SimBiFPN reached 99.1% on an Angus beef cattle keypoint detection dataset. The mean absolute percentage errors for measuring beef cattle withers height, hip height, body depth, body length, and oblique body length using the CBMM were 4.37%, 4.96%, 6.47%, 4.84%, and 4.14%, respectively. In summary, our method can achieve non-contact body measurement for beef cattle in a free-moving state with high accuracy and stability.

Dense Feature Pyramid Deep Completion Network

Most current point cloud super-resolution reconstruction requires huge calculations and has low accuracy when facing large outdoor scenes; a Dense Feature Pyramid Network (DenseFPNet) is proposed for the feature-level fusion of images with low-resolution point clouds to generate higher-resolution point clouds, which can be utilized to solve the problem of the super-resolution reconstruction of 3D point clouds by turning it into a 2D depth map complementation problem, which can reduce the time and complexity of obtaining high-resolution point clouds only by LiDAR. The network first utilizes an image-guided feature extraction network based on RGBD-DenseNet as an encoder to extract multi-scale features, followed by an upsampling block as a decoder to gradually recover the size and details of the feature map. Additionally, the network connects the corresponding layers of the encoder and decoder through pyramid connections. Finally, experiments are conducted on the KITTI deep complementation dataset, and the network performs well in various metrics compared to other networks. It improves the RMSE by 17.71%, 16.60%, 7.11%, and 4.68% compared to the CSPD, Spade-RGBsD, Sparse-to-Dense, and GAENET.

Simple modulation of Lissajous MEMS laser beam scanning with reconfigurable structured light patterns for 3D imaging

Structured light 3D imaging systems commonly employ panel-based projectors or 1-axis MEMS mirrors with beam expander lens to project multi-frame barcodes or dot clouds, addressing challenges posed by objects with multi-scale feature sizes. However, these methods often result in large system volumes due to the required projection multi-lens modules, high hardware costs, or limited light pattern generation capabilities that hindering measurement precision enhancement. This paper introduces an innovative approach to reconfigurable spatial light pattern projection using a single bi-axial MEMS mirror with Lissajous scanning. In contrast to the pixel-by-pixel pre-defined image patterns encoding of conventional 2D laser beam scanning, the proposed method simply aligns the MEMS bi-axial resonance frequencies with laser pulse modulation, enabling the projection of diverse structured light patterns such as stripes, lines, dot matrices, and random dot clouds, which can adapt to different 3D imaging algorithms demands. It eliminates the need for multi-frame encoding and streamlines data caching, simplifies digital logic hardware. A prototype 3D imaging system was developed to demonstrate the mathematical model for laser modulation and the technical feasibility based on the proposed principle. Beyond its lens-free essence, the system supports focal-free optics and a compact projection form factor, which accommodates to a broad range of projection distances and field-of-views based on object’s location. 3D depth map of polynomial surface and blocks objects are extracted through single-frame pattern projection with a relative high accuracy. The presented modulation theory for diverse structured light pattern generation opens avenues for versatile and compact 3D imaging applications of LiDAR and robotic 3D vision.

Self-Supervised 3D Human Mesh Recovery from a Single Image with Uncertainty-Aware Learning

Despite achieving impressive improvement in accuracy, most existing monocular 3D human mesh reconstruction methods require large-scale 2D/3D ground-truths for supervision, which limits their applications on unlabeled in-the-wild data that is ubiquitous. To alleviate the reliance on 2D/3D ground-truths, we present a self-supervised 3D human pose and shape reconstruction framework that relies only on self-consistency between intermediate representations of images and projected 2D predictions. Specifically, we extract 2D joints and depth maps from monocular images as proxy inputs, which provides complementary clues to infer accurate 3D human meshes. Furthermore, to reduce the impacts from noisy and ambiguous inputs while better concentrate on the high-quality information, we design an uncertainty-aware module to automatically learn the reliability of the inputs at body-joint level based on the consistency between 2D joints and depth map. Experiments on benchmark datasets show that our approach outperforms other state-of-the-art methods at similar supervision levels.

Decoupling Dynamic Monocular Videos for Dynamic View Synthesis.

The challenge of dynamic view synthesis from dynamic monocular videos, i.e., synthesizing novel views for free viewpoints given a monocular video of a dynamic scene captured by a moving camera, mainly lies in accurately modeling the dynamic objects of a scene using limited 2D frames, each with a varying timestamp and viewpoint. Existing methods usually require pre-processed 2D optical flow and depth maps by off-the-shelf methods to supervise the network, making them suffer from the inaccuracy of the pre-processed supervision and the ambiguity when lifting the 2D information to 3D. In this paper, we tackle this challenge in an unsupervised fashion. Specifically, we decouple the motion of the dynamic objects into object motion and camera motion, respectively regularized by proposed unsupervised surface consistency and patch-based multi-view constraints. The former enforces the 3D geometric surfaces of moving objects to be consistent over time, while the latter regularizes their appearances to be consistent across different viewpoints. Such a fine-grained motion formulation can alleviate the learning difficulty for the network, thus enabling it to produce not only novel views with higher quality but also more accurate scene flows and depth than existing methods requiring extra supervision. We will make the code publicly available at https://github.com/mengyou2/DecoulpingNeRF.

Understanding the contagiousness of Covid-19 strains: A geometric approach

Protein–protein interaction occurs on surface patches with some degree of complementary geometric and chemical features. Building on this understanding, this study endeavors to characterize the spike protein of the SARS-CoV-2 virus at the morphological and geometrical levels in its Alpha, Delta, and Omicron variants. In particular, the affinity between different SARS-CoV-2 spike proteins and the ACE2 receptor present on the membrane of the human respiratory system cells is investigated. To achieve an adequate degree of geometrical accuracy, the 3D depth maps of the proteins in exam are filtered by developing an ad-hoc convolutional filter with a kernel implemented as a sphere of varying radius, simulating a ball rolling on the surface (similar to the 'rolling ball' filter). This ball ideally models a hypothetical molecule that could interface with the protein and is inspired by the geometric approach to macromolecule–ligand interactions proposed by Kuntz et al. in 1982. The aim is to mitigate the imperfections and to obtain a smoother surface that could be studied from a geometrical perspective for binding purposes. A set of geometric descriptors, borrowed from the 3D face analysis context is then mapped point-by-point onto protein depth maps. Following a feature extraction phase inspired by Histogram of Oriented Gradients and Local Binary Patterns, the final histogram features are used as input for a Support Vector Machine classifier to automatically classify the proteins according to their surface affinity, where a similarity in shape is observed between ACE2 and the spike protein of the SARS-CoV-2 Omicron variant. Finally, Root Mean Square Error analysis is used to quantify the geometrical affinity between the ACE2 receptor and the respective Receptor Binding Domains of the three SARS-CoV-2 variants, culminating in a geometrical explanation for the higher contagiousness of Omicron relative to the other variants under study.

Joint representation learning for text and 3D point cloud

Recent advancements in vision-language pre-training (e.g., CLIP) have enabled 2D vision models to benefit from language supervision. However, the joint representation learning of 3D point cloud with text remains under-explored due to challenges in acquiring 3D-Text data pairs. Prior works propose to project point clouds into 2D depth maps and directly use CLIP, while they sacrifice 3D structural information, limiting its applicability. In this paper, we put forward Text4Point, a novel framework to construct language-guided 3D models for dense prediction tasks. Text4Point utilizes 2D images as a bridge to connect the point cloud and language modalities. It follows a pre-training and fine-tuning paradigm. During pre-training, we leverage dense contrastive learning to align the image and point cloud representations using the readily available RGB-D data. Together with the well-aligned image and text features achieved by CLIP, the point cloud features are implicitly aligned with the text embeddings. Further, we propose a Text Querying Module to integrate language information into 3D representation learning by querying text embeddings with point cloud features. For fine-tuning, the model learns 3D representations under informative language guidance without 2D images. Extensive experiments demonstrate consistent improvement on various dense prediction tasks with Text4Point.

Surface roughness evaluation based on near point lighting photometric stereo

Surface roughness evaluation is crucial for enhancing surface properties. Traditional offline human-assisted inspection techniques are labor-intensive and time-consuming, hindering manufacturing productivity. Current machine vision-based roughness measurement methods primarily rely on grayscale or color images, neglecting the benefits of 3D morphology's multidimensional information. This results in complex and inaccurate algorithms. To address these issues, we propose an efficient surface roughness evaluation method using Near Point Lighting Photometric Stereo (NPL-PS). Our NPL-PS scheme, configured with one camera and 24 LED point light sources, generates a 3D depth map of the machined surface. Under the classic Lambertian model and camera weak perspective transform, we derive a linearized photometric equation for NPL-PS containing light intensity attenuation. By solving the linear equation using our proposed methods for illumination position calibration and non-uniform illumination correction, we estimate the normal vector corresponding to each pixel. We then use the Frankot-Chellapa algorithm to recover the height map from the vector field. Finally, we extract roughness texture features from the height map using polynomial fitting and quantitatively evaluate the roughness. Our experiments using real surface roughness specimens demonstrate the feasibility and effectiveness of this method compared to traditional methods, laying the foundation for subsequent practical production applications.

DGE-CNN: 2D-to-3D holographic display based on a depth gradient extracting module and ZCNN network.

Holography is a crucial technique for the ultimate three-dimensional (3D) display, because it renders all optical cues from the human visual system. However, the shortage of 3D contents strictly restricts the extensive application of holographic 3D displays. In this paper, a 2D-to-3D-display system by deep learning-based monocular depth estimation is proposed. By feeding a single RGB image of a 3D scene into our designed DGE-CNN network, a corresponding display-oriented 3D depth map can be accurately generated for layer-based computer-generated holography. With simple parameter adjustment, our system can adapt the distance range of holographic display according to specific requirements. The high-quality and flexible holographic 3D display can be achieved based on a single RGB image without 3D rendering devices, permitting potential human-display interactive applications such as remote education, navigation, and medical treatment.

Evaluation of a new connected portable camera for the analysis of skin microrelief and the assessment of the effect of skin moisturisers.

Silicone replicas and non-contact methods are effective methods to analyse the micrometric scale of the skin microrelief. Yet, they imply data capture in research facilities. The capabilities of a new connected portable camera were evaluated to analyse microrelief under nomadic conditions, also studying the effect of moisturisers. 3D depth maps were constructed using shape-from-shading algorithms. Roughness heterogeneity (Spa) was computed, and skin profiles were extracted to calculate roughness amplitude (Ra, Rq), as well as furrows/plateaus characteristics. Validation of the connected camera was performed on tanned cowhide leather and on the inner forearm skin of a single subject. The forearms of 18 subjects (23-60 years old) were also evaluated. While living their regular life, they self-performed triplicate acquisitions at various times. The effects of a placebo and of cream containing moisturisers-saccharide isomerate, urea or xylitylglucoside-anhydroxylitol-xylitol-were investigated, using untreated control skin as a reference. Validation of the device on leather and forearm skin shows high repeatability. The 18 subjects show the known correlation between age and changes in microrelief. While testing formulas, 8h after a single application, all decreased Spa (-1.6/-2.1 folds). Only saccharide isomerate and xylitylglucoside-anhydroxylitol-xylitol decreased Ra (-2.4/-2.8 folds). The sectional area of plateaus was reduced from -1.5 (urea) to -2.1 folds (xylitylglucoside-anhydroxylitol-xylitol). The height of plateaus is also decreased by all moisturisers, from -1.5 (urea) to -2.1 folds (xylitylglucoside-anhydroxylitol-xylitol). This novel camera device enables microrelief analysis under nomadic conditions, allowing monitoring its changes along the day and upon moisturisers' application.

CMAN: Leaning Global Structure Correlation for Monocular 3D Object Detection

The key to 3D object detection is proper utilization of depth data. Compared with LiDAR based approaches, 3D object detection from a single image remains a challenging task due to the lack of structure information. Recent methods leverage monocular depth estimation as a way to produce 2D depth maps, and adopt the depth maps as additional source of input to explore structure information. However, these methods either encode local structure correlations, or encode long range structure correlations by iteratively passing local messages. In this work, we propose a cross modal attention network (CMAN) for monocular 3D object detection. It is built upon the self-attention module which learns attention map from single modal data. Our CMAN is able to encode structure correlations from depth data, and embed the structure correlations with appearance information which is learned from RGB data. Thanks to the attention learning mechanism, our CMAN learns global structure correlations without iteration. In order to reduce the computational burden, our CMAN adopts a novel node sampler to eliminate redundant nodes during the attention map calculation. Experiment results on benchmark KITTI3D dataset show that our proposed CMAN outperforms the state-of-the-art methods.

Application of 3D point cloud map and image identification to mobile robot navigation

This paper presents application of the combination of mapping, image identification, and indoor navigation for the omnidirectional wheeled mobile robot (WMR). The mapping part uses the point clouds converted from the depth map of the LiDAR camera to construct the indoor 3D point cloud map. Combining this map with the You Only Look Once ver.4 (YOLOv4) network, the door number and the doorknob can be displayed on the map by image identification. The navigation part applies 3D point cloud map, image, and depth map to an omnidirectional WMR so that it can know the current position and plan the path to the designated location. The proposed control scheme can make the robot perform indoor navigation and reach designated location.

GCNDepth: Self-supervised monocular depth estimation based on graph convolutional network

Depth estimation is a challenging task of 3D reconstruction to enhance the accuracy sensing of environment awareness. This work brings a new solution with improvements, which increases the quantitative and qualitative understanding of depth maps compared to existing methods. Recently, convolutional neural networks (CNN) have demonstrated their extraordinary ability to estimate depth maps from monocular videos. However, traditional CNN does not support a topological structure, and they can work only on regular image regions with determined sizes and weights. On the other hand, graph convolutional networks (GCN) can handle the convolution of non-Euclidean data, and they can be applied to irregular image regions within a topological structure. Therefore, to preserve object geometric appearances and objects locations in the scene, in this work, we aim to exploit GCN for a self-supervised monocular depth estimation model. Our model consists of two parallel auto-encoder networks: the first is an auto-encoder that will depend on ResNet-50 and extract the feature from the input image and on multi-scale GCN to estimate the depth map. In turn, the second network will be used to estimate the ego-motion vector (i.e., 3D pose) between two consecutive frames based on ResNet-18. The estimated 3D pose and depth map will be used to construct the target image. A combination of loss functions related to photometric, reprojection, and smoothness is used to cope with bad depth prediction and preserve the discontinuities of the objects. Our method and performance are improved quantitatively and qualitatively. In particular, our method provided comparable and promising results with a high prediction accuracy of 89% on the publicly available KITTI dataset. Our method also offers 40% reduction in the number of trainable parameters compared to the state of the art solutions.In addition, we tested our trained model with Make3D dataset to evaluate the trained model on a new dataset with low resolution images. The source code is publicly available at (https://github.com/ArminMasoumian/GCNDepth.git)

Highly precise AMCW time-of-flight scanning sensor based on parallel-phase demodulation

A novel amplitude-modulated continuous wave (AMCW) time-of-flight (ToF) scanning sensor based on parallel-phase demodulation is proposed and demonstrated. Contrast to the conventional AMCW ToF sensor which utilizes serial-phase demodulation, the proposed sensor processes each different phase shift demodulation independently in parallel manner, so that the proposed sensor can maintain high demodulation contrast and short total integration time. Based on these characteristics of parallel-phase demodulation, the proposed sensor platform can be implemented with 2D laser scanning system with a single photodetector, maintaining a moderate frame rate, high optical SNR, and very high pixel resolution. In the validating demonstration, the proposed AMCW ToF scanning sensor shows extremely low measured distance noise per reference distance of 0.056 % at 1.50 m with only 30 mW illumination power. Additionally, FHD scale (1920 × 1080) highly precise 3D depth map with 800 nsec integration time can be achieved using the proposed AMCW ToF scanning sensor.

Faster than Real-Time Surface Pose Estimation with Application to Autonomous Robotic Grasping

Motivated by grasp planning applications within cluttered environments, this paper presents a novel approach to performing real-time surface segmentations of never-before-seen objects scattered across a given scene. This approach utilizes an input 2D depth map, where a first principles-based algorithm is utilized to exploit the fact that continuous surfaces are bounded by contours of high gradient. From these regions, the associated object surfaces can be isolated and further adapted for grasp planning. This paper also provides details for extracting the six-DOF pose for an isolated surface and presents the case of leveraging such a pose to execute planar grasping to achieve both force and torque closure. As a consequence of the highly parallel software implementation, the algorithm is shown to outperform prior approaches across all notable metrics and is also shown to be invariant to object rotation, scale, orientation relative to other objects, clutter, and varying degree of noise. This allows for a robust set of operations that could be applied to many areas of robotics research. The algorithm is faster than real time in the sense that it is nearly two times faster than the sensor rate of 30 fps.

152 Evaluation of the SkinCam, a Connected Portable Camera Device, for the Analysis of Skin Microrelief

Silicone replicas and non-contact methods are effective in analysing the micrometric scale of the plateaus and furrows that forms the skin microrelief. Yet, a major drawback is the obligation to capture data in research facilities. Therefore, the capacities of a connected portable camera device, the SkinCam, was evaluated to analyse the skin microrelief in real-time and while subjects were living their regular daily life routine. The effect of moisturisers on microrelief was also analysed. From RGB data, 3D depth maps were constructed using shape from shading algorithms.

MIMONet: Structured-light 3D shape reconstruction by a multi-input multi-output network.

Reconstructing 3D geometric representation of objects with deep learning frameworks has recently gained a great deal of interest in numerous fields. The existing deep-learning-based 3D shape reconstruction techniques generally use a single red-green-blue (RGB) image, and the depth reconstruction accuracy is often highly limited due to a variety of reasons. We present a 3D shape reconstruction technique with an accuracy enhancement strategy by integrating the structured-light scheme with deep convolutional neural networks (CNNs). The key idea is to transform multiple (typically two) grayscale images consisting of fringe and/or speckle patterns into a 3D depth map using an end-to-end artificial neural network. Distinct from the existing autoencoder-based networks, the proposed technique reconstructs the 3D shape of target using a refinement approach that fuses multiple feature maps to obtain multiple outputs with an accuracy-enhanced final output. A few experiments have been conducted to verify the robustness and capabilities of the proposed technique. The findings suggest that the proposed network approach can be a promising 3D reconstruction technique for future academic research and industrial applications.

가중 메디안 필터를 이용한 16채널 라이다의 깊이정보 업샘플링

Three-dimensional (3D) LiDAR is widely used because it can accurately measure 360° 3D depth. One of the most widely used sensors to this end is VLP-16, a 16-channel LiDAR manufactured by Velodyne. This 16-channel LiDAR provides dense 3D depth data in the horizontal direction. However, in the vertical direction, it provides only sparse depth data owing to the LiDAR architecture. We propose an upsampling method to obtain a dense 3D depth map from a 16-channel LiDAR by applying an edge-aware filter to the LiDAR data. A camera- LiDAR sensor system composed of a 16-channel LiDAR and six vision cameras is used for this purpose. First, the LiDAR depth data are projected onto the camera image coordinates to generate a 2.5D depth map. Next, bilinear interpolation and a weighted median filter are applied to upsample the original depth map. Compared with the original LiDAR data, we can generate an extremely dense depth map by using the proposed method. The results of quantitative analysis show that the proposed method can dramatically increase the number of 3D depth points and reduce the depth reconstruction error.

3D-3D Self-Calibration of Sensors Using Point Cloud Data

<div class="section abstract"><div class="htmlview paragraph">Self-calibration of sensors has become highly essential in the era of self-driving cars. Reducing the sensors’ errors increases the reliability of the decisions made by the autonomous systems. Various methods are currently under investigation but the traditional methods still prevail which maintain a strong dependency on human experts and expensive equipment that consume significant amounts of labor and time. Recently, various calibration techniques proposed for extrinsic calibration for Autonomous Vehicles (AVs) mostly rely on the camera 2D images and depth map to calibrate the 3D LiDAR points. While most methods work with the whole frame, some methods use the objects in the frame to perform the calibration. To the best of our knowledge, majority of these self-calibration methods rely on using actual or falsified ground truth values.</div><div class="htmlview paragraph">We propose a 3D-3D point cloud based continuous self-calibration approach that uses one or many objects identified in the sensor frames to cross-calibrate sensors without any reliance on initial calibration parameters or ground truth values. Considering the fact that multiple sensors have multiple views of the same scene, the common features or objects within the scene can be used to calibrate one sensor node with respect to the other. Our approach relies on point cloud data (PCD) generated from at least two sensors to cross-calibrate the mis-calibrated sensor with respect to the calibrated sensor. In this paper, we demonstrate that we can apply the method either by extracting the essential feature points of the object PCD (i.e., centroids of the objects) or the whole object PCD. Following which we optimize the cost function to obtain the extrinsic calibration parameters. Our method handles the problem of pose correction similar to calibration without any ground truth value from sensors just by using multiple consecutive frames. In comparison to other methods, our method performs calibration with low margins of errors in rotation and translation. This method has been tested on the publicly available KITTI dataset, which is widely used to assess various problems (object detection, segmentation, tracking, depth prediction etc.) related to self-driving cars. The simplicity of the method lets us do on-the-fly calibration of sensors with a high level of accuracy.</div></div>

High Contrast Ultrathin Light‐Field Camera Using Inverted Microlens Arrays with Metal–Insulator–Metal Optical Absorber

AbstractLight‐field imaging has attracted much attention in constructing 3D objects with a simple configuration and capturing all the spatial and directional data in a single photographic exposure. Here, an ultrathin light‐field camera (ULFC) for high contrast and high‐resolution light‐field imaging using a metal–insulator–metal optical absorber based inverted microlens arrays (MIM‐iMLA) is reported. A metal–insulator–metal based optical absorber (MIM‐OA) between microlenses exhibits high light absorption in the full visible region, thereby highly blocking microlens crosstalk. The MIM‐iMLA double the image contrast and improve MTF50 by up to 32%, compared to conventional light‐field image. In addition, the MIM‐iMLA substantially reduces an image plane distance and brings the objective lens position closer to the MLA. The ULFC exhibits a short total track length of 5.1 mm, demonstrating high contrast light‐field image acquisition and high accuracy 3D depth map estimation after light‐field rendering. This ultrathin and high contrast light‐field camera can provide a new platform for miniaturized 3D cameras in biomedical applications, biometrics, automated inspection, or mobile camera applications.