Depth Information Research Articles

3D Industrial anomaly detection via dual reconstruction network

Currently, 2D anomaly detection has demonstrated outstanding performance. However, 2D images limit the improvement of anomaly detection accuracy without utilizing depth information. Therefore, this paper proposes a Dual Reconstruction viAInpainting Network for 3D industrial anomaly detection (DRAIN). Firstly, we design a 3D reconstruction network using an encoder-decoder-based U-shaped network for processing RGB images and depth images. Subsequently, accurate anomaly segmentation is implemented through a 3D segmentation network. We introduce a lightweight MLP module to enhance segmentation performance to capture long-range dependencies in the reconstructed images. Furthermore, we propose a dual attention-based information entropy fusion module to expedite feature fusion in the inference process, aiming for enhanced deployment in the industry. Extensive experiments demonstrate that DRAIN achieves a 94.3% AUROC on the 3D anomaly detection dataset MVTec 3D-AD, surpassing other research methods.Graphical abstractOverall architecture for 3D industrial anomaly detection via dual reconstruction network

Incomplete multi-view partial multi-label classification via deep semantic structure preservation

Recent advances in multi-view multi-label learning are often hampered by the prevalent challenges of incomplete views and missing labels, common in real-world data due to uncertainties in data collection and manual annotation. These challenges restrict the capacity of the model to fully utilize the diverse semantic information of each sample, posing significant barriers to effective learning. Despite substantial scholarly efforts, many existing methods inadequately capture the depth of semantic information, focusing primarily on shallow feature extractions that fail to maintain semantic consistency. To address these shortcomings, we propose a novel Deep semantic structure-preserving (SSP) model that effectively tackles both incomplete views and missing labels. SSP innovatively incorporates a graph constraint learning (GCL) scheme to ensure the preservation of semantic structure throughout the feature extraction process across different views. Additionally, the SSP integrates a pseudo-labeling self-paced learning (PSL) strategy to address the often-overlooked issue of missing labels, enhancing the classification accuracy while preserving the distribution structure of data. The SSP model creates a unified framework that synergistically employs GCL and PSL to maintain the integrity of semantic structural information during both feature extraction and classification phases. Extensive evaluations across five real datasets demonstrate that the SSP method outperforms existing approaches, including lrMMC, MVL-IV, MvEL, iMSF, iMvWL, NAIML, and DD-IMvMLC-net. It effectively mitigates the impacts of data incompleteness and enhances semantic representation fidelity.

3DGEN: a framework for generating custom-made synthetic 3D datasets for civil structure health monitoring

The availability of high-quality datasets is increasingly critical in the field of computer vision-based civil structural health monitoring, where deep learning approaches have gained prominence. However, the lack of specialized datasets for such tasks poses a significant challenge for training a reliable model. To address this challenge, a framework, 3DGEN, is proposed to swiftly generate realistic synthetic 3D datasets which can be targeted for specific tasks. The framework is based on diverse 3D civil structural models, rendering them from various angles and providing depth information and camera parameters for training neural networks. By employing mathematical methods, such as analytical solutions and/or numerical simulations, deformation of civil engineering structures can be generated, ensuring a reliable representation of their real-world shapes and characteristics in the 3D datasets. For texture generation, a generative 3D texturing method enables users to specify desired textures using plain English sentences. Two successful experiments are conducted to (1) assess the efficiency of generating the 3D datasets using two distinct structures, (2) train a monocular depth estimation network to perform 3D surface reconstruction with the generated dataset. Notably, 3DGEN is not limited to 3D surface reconstruction; it can also be used for training neural networks for various other tasks. The code and dataset are available at: https://github.com/YANDA-SHAO/Beam-Dataset-SE

Real-time testing of AI enabled automatic emergency braking system for ADAS vehicle using 3D point cloud and precise depth information

At the forefront of automobile safety technology, Automatic Emergency Braking (AEB) represents a major advancement in collision avoidance systems. The cutting-edge technology provides an additional layer of security at pivotal times, making it a vital part of the changing landscape of vehicle safety. This research introduces an effective system for automatic emergency braking in ADAS-equipped or Autonomous vehicles using a combination of 3d lidar and stereo vision camera for a swift and robust system in the vehicle that is faster than human drivers in cases of unexpected emergencies. Utilizing the power of clustering algorithms on 3d point clouds and state-of-the-art computer vision algorithms on an RGB image mapped to a depth frame from the stereo vision camera, the system as a whole provides a comprehensive system adding to the safety of the vehicle and the passengers. Further, the efficiency of the system is studied based on various parameters. The data from an external inertial measurement unit is also utilized to derive results that support the claims of the study. The system has been developed and implemented on a passenger car that has been modified into an electric vehicle and further tested in real-world traffic conditions in autonomous driving mode. The findings of the study were having exceptionally good precision in split-second decision-making in emergency maneuvers.

A deep learning-based method for estimating the main stem length of sweet potato seedlings

The length of sweet potato seedling main stems is critical for harvest decisions, but accurately measuring it is challenging due to leaf occlusion and self-curvature. This study proposes a method for stem reconstruction and length estimation using a two-stage classifier, feature extraction, and semantic segmentation to obtain stem masks. Image processing separates clustered skeletons into singular lines for detailed analysis. Object detection identifies roots, stem-leaf intersections, and establishes a search relationship among segments. Stems are reconstructed based on directional, distance, and curvature constraints, then divided into four parts for length estimation using stereo depth information. Experimental results demonstrate effectiveness under various conditions: un-occluded, partially occluded, curved stems, and low-density clusters. Median length estimation error is 1.7 cm, with maximum and minimum errors of 3.2 cm and 0.3 cm respectively. This automated method provides a reliable solution for sweet potato seedling harvest and grading.

Taking a Consumer-Led Approach to Identify Key Characteristics of an Effective Ecolabelling Scheme

A gap exists between consumer adoption of sustainable purchasing habits and the behavioural changes required to significantly decrease planetary environmental impact. Current ecolabels do not enable purchasing based on environmental impact, and an effective labelling scheme is required to change purchasing priorities. Using focus groups, ecolabels with components varying in the depth of environmental information conveyed, visual features employed, as well as product comparability and scalability of design, were evaluated by consumers with differing levels of environmental awareness. Independent, credible information accessible to consumers of varying environmental awareness was a participant priority. In-depth data communicated through a bar or pie chart with an interpretable holistic message across all environmental factors was preferred. Colour-coded components offering product comparisons and scalable to the size of the packaging, product value, and lifetime were most likely to empower environmentally motivated purchasing. Insights can inform the development of an effective ecolabelling scheme that equips consumers to adopt more sustainable purchasing habits.

Improved Earthquake Source Parameters with 3D Wavespeed Models in California and Nevada

Abstract Seismic tomography harnesses earthquake data to explore the inaccessible structure of the Earth. Adjoint waveform tomography (AWT), a method of seismic tomography, updates the tomographic model by optimizing the fit between observed earthquake data and synthetic waveforms. The synthetic data are calculated by solving the wave equation through a given 3D model. An important requirement to calculating synthetics is the source information (location, centroid time, depth, and moment tensor). Errors in source information affect the quality of the synthetics produced, which in turn can limit how structure can be inferred in the AWT workflow. To test the effect of updating source information, we used MTTime (Chiang, 2020), a time-domain full-waveform moment tensor inversion code, to calculate the moment tensors and depths of 118 earthquakes that occurred in California and Nevada over a 20-yr period. We calculated 3D Green’s functions using a 3D seismic wavespeed model of California and Nevada (Doody et al., 2023b). We show that the inverted solutions provide better waveform fits than the Global Centroid Moment Tensor catalog and increase usable, well-correlated data by up to 7%. Therefore, we argue that recalculating source parameters should be considered in AWT workflows, particularly for smaller magnitude events (Mw<5.0).

Improving Neural Radiance Fields Using Near-Surface Sampling with Point Cloud Generation

Neural radiance field (NeRF) is an emerging view synthesis method that samples points in a three-dimensional (3D) space and estimates their existence and color probabilities. The disadvantage of NeRF is that it requires a long training time since it samples many 3D points. In addition, if one samples points from occluded regions or in the space where an object is unlikely to exist, the rendering quality of NeRF can be degraded. These issues can be solved by estimating the geometry of 3D scene. This paper proposes a near-surface sampling framework to improve the rendering quality of NeRF. To this end, the proposed method estimates the surface of a 3D object using depth images of the training set and performs sampling only near the estimated surface. To obtain depth information on a novel view, the paper proposes a 3D point cloud generation method and a simple refining method for projected depth from a point cloud. Experimental results show that the proposed near-surface sampling NeRF framework can significantly improve the rendering quality, compared to the original NeRF and three different state-of-the-art NeRF methods. In addition, one can significantly accelerate the training time of a NeRF model with the proposed near-surface sampling framework.

Ten lessons for rare wildlife species conservation when coming from a common species background

AbstractEfforts to slow biodiversity loss have ushered in an era of proactive conservation for rare species, or species that have any combination of low relative abundance, high habitat specificity, and limited geographic range. Approaches to wildlife management and conservation were often developed for more abundant, generalist species, and frequently do not translate well to rare species. Conservation for rare species therefore requires adapting knowledge, action and inquiry tools for common species to rare species. We therefore present 10 lessons for adapting conservation tools to rare species, each with advice to implement the lessons. Several themes emerge from these lessons. Information scarcity necessitates high standards of information for rare species. Missing rare species information can be complemented with multiple forms of knowledge, particularly when the need for conservation action precludes additional data collection. Investment in depth (over breadth) of information for individuals of rare species can provide alternatives to population trend, which has lower reliability for rare species. Natural history documentation can improve rare species conservation strategies. Finally, curiosity‐driven inquiry can inspire rare species conservation innovation. We hope providing guidance that addresses challenges for rare species will help others bridge the gap between common wildlife approaches and proactive rare species conservation.

Three-dimensional vascular graph construction from depth information of blood vessel centerlines in optical coherence tomography angiography

Optical coherence tomography angiography (OCTA) is a noninvasive imaging technique for visualizing high-resolution volumetric vascular networks. Although OCTA has been widely employed in vascular network analysis, most studies have been limited to the analysis of two-dimensional (2D) en-face projection images because leveraging the full potential of OCTA’s three-dimensional (3D) information has been challenging due to projection artifacts beneath blood vessels. In this study, we propose a semi-automatic method for constructing a 3D vascular graph from 3D OCTA images without relying on data-driven learning strategies such as deep learning. The proposed method estimates the depth information of blood vessel centerlines in 2D en-face images and constructs a 3D vascular graph by integrating the depth estimation results for all vessel centerline segments. We demonstrate the effectiveness of the proposed method through experiments conducted on both simulated datasets and real datasets acquired from the dorsal dermis of mice.

High-speed all-in-focus 3D imaging method based on liquid lens focus scanning

A large open aperture in an optical system can capture high-resolution images but yields a shallow depth of field. To overcome this issue, we propose a method for retrofitting microscopy imaging systems by using a variable-focus liquid lens to achieve 3D focus scanning. Specifically, the focal length of the imaging system was changed by controlling the liquid lens, and a sequence of images was captured at different focal planes in milliseconds. The image scale and phase were corrected. Then the in-focus pixels were abstracted by employing the Laplacian operator. These pixels were marked in the index table. According to the index table, the focused parts of multiple images were merged, and an all-in-focus image was generated. At the same time, a depth map was constructed based on the image number in the index table and the extracted depth information. Additionally, we have optimized the image processing flow; the processing speed was improved to around 6.5 fps.

Deep learning aided computer vision system for automated linear type trait evaluation in dairy cows

The assessment of traits is important in determining production potential, reproductive performance, and overall health of dairy cows. The assessment of these traits typically involves visual inspection and manual measurement, which can be time-consuming, subject to bias, and potentially distressing for the animals. To address these challenges, convolutional neural networks (CNNs)-aided non-invasive computer vision system was developed in the present study. This system consists of a depth camera to acquire the RGB images and depth information of cows. The DeepLabV3+ model, having the ResNet50 model as a backbone, was utilized to segment the body parts of cows from RGB images. Image processing-based algorithms were developed to extract key pixel locations for each trait from these segmented images. The system estimated trait dimensions utilizing 3D data of respective key points. The mean-IoU (intersection-over-union) values for the developed segmentation models were 93.46%, 91.25%, and 99.27% for side-view, back-view traits, and stature, respectively. Additionally, the vision system was able to estimate the trait dimensions with mean absolute percentage error (MAPE) below 6.0%. For a few traits, MAPE, however, exceeded 10.0%, indicating higher error. Inaccurate segmentation, imprecise key point extraction, visual overlaps of specific body parts, and variations in cow postures contribute to such errors. The developed system attained a Ratio of Performance to Deviation (RPD) above 1.2 for all traits, indicating its ability to estimate the dimensions of traits efficaciously. Thus, the present study demonstrated the potential of a CNN-based computer vision-based system for automating the trait measurement process in cows.

Edge highlighting with depth-dependent opacity gradation of laser-scanned point clouds improves the accuracy of perceived depth in transparent multi-view 3D visualizations

There is an activity called "digital archiving" in which cultural property is digitized for preservation and utilization, and transparent visualization using 3D point clouds is a valuable tool for understanding the complex internal structure of cultural property. However, when 3D data is transparently visualized, depth information may become unclear. In this paper, we investigated whether the depth perception of transparently visualized 3D objects can be improved by highlighting the 3D edges of the structures and adjusting the opacity of the edges according to the depth. In order to verify the effect of the depth-dependent 3D edges, we conducted a psychophysical experiment in which subjects were asked to report the magnitude of perceived depth for 3D structures of two cultural properties using a multi-view 3D display. The perceived depth was smaller than the simulated depth under all conditions. However, the opacity adjustment in edge highlighting mitigated the depth underestimation from an average of 69.4 to 35.5%. These results indicate that edge highlighting with opacity adjusted according to depth improves the accuracy of the perceived depth of 3D structures of cultural property visualized transparently.Graphical abstract

BiDiCOS: Camouflaged object segmentation via bilateral diffusion model

The task of camouflaged object segmentation (COS) is to extract objects with camouflage characteristics that are difficult to detect in an image. The current mainstream COS method mainly relies on the general object segmentation framework using the encoder-decoder structure to predict the objects. However, the current COS framework suffers from over-reliance on the labeled ground truth and the output results. To this end, we propose a bilateral diffusion model for camouflaged object segmentation (BiDiCOS). The main innovation of BiDiCOS is that we design a new bilateral diffusion paradigm, transforming the segmentation and depth estimation tasks into denoising tasks. At the same time, the depth estimation branch can provide semantic information to the segmentation branch. Specifically, we design the adaptive feature fusion module for two branches to provide the required features adaptively. We design the bilateral feature fusion module to fuse the depth information with the mask information to reduce the semantic and logical errors during segmentation. Finally, we develop the specialized training strategy and inference strategy for BiDiCOS, which provide a solid foundation for realizing rational training and accurate prediction. Comprehensive experiments on four challenging COS datasets attest to the superior performance of BiDiCOS compared to the existing 20 COS methods. Our BiDiCOS achieves state-of-the-art with different sizes of backbones, where the MAE is reduced by 4.3 % on average. We have also performed comparative experiments on depth estimation. The experiments demonstrate that BiDiCOS can achieve a win–win situation for both tasks. The code will be available at https://github.com/jiangxinhao2020/BiDiCOS.

Enhancing Visual Odometry with Estimated Scene Depth: Leveraging RGB-D Data with Deep Learning

Advances in visual odometry (VO) systems have benefited from the widespread use of affordable RGB-D cameras, improving indoor localization and mapping accuracy. However, older sensors like the Kinect v1 face challenges due to depth inaccuracies and incomplete data. This study compares indoor VO systems that use RGB-D images, exploring methods to enhance depth information. We examine conventional image inpainting techniques and a deep learning approach, utilizing newer depth data from devices like the Kinect v2. Our research highlights the importance of refining data from lower-quality sensors, which is crucial for cost-effective VO applications. By integrating deep learning models with richer context from RGB images and more comprehensive depth references, we demonstrate improved trajectory estimation compared to standard methods. This work advances budget-friendly RGB-D VO systems for indoor mobile robots, emphasizing deep learning’s role in leveraging connections between image appearance and depth data.

Monocular Absolute Depth Estimation from Motion for Small Unmanned Aerial Vehicles by Geometry-Based Scale Recovery.

In recent years, there has been extensive research and application of unsupervised monocular depth estimation methods for intelligent vehicles. However, a major limitation of most existing approaches is their inability to predict absolute depth values in physical units, as they generally suffer from the scale problem. Furthermore, most research efforts have focused on ground vehicles, neglecting the potential application of these methods to unmanned aerial vehicles (UAVs). To address these gaps, this paper proposes a novel absolute depth estimation method specifically designed for flight scenes using a monocular vision sensor, in which a geometry-based scale recovery algorithm serves as a post-processing stage of relative depth estimation results with scale consistency. By exploiting the feature correspondence between successive images and using the pose data provided by equipped navigation sensors, the scale factor between relative and absolute scales is calculated according to a multi-view geometry model, and then absolute depth maps are generated by pixel-wise multiplication of relative depth maps with the scale factor. As a result, the unsupervised monocular depth estimation technology is extended from relative depth estimation in semi-structured scenes to absolute depth estimation in unstructured scenes. Experiments on the publicly available Mid-Air dataset and customized data demonstrate the effectiveness of our method in different cases and settings, as well as its robustness to navigation sensor noise. The proposed method only requires UAVs to be equipped with monocular camera and common navigation sensors, and the obtained absolute depth information can be directly used for downstream tasks, which is significant for this kind of vehicle that has rarely been explored in previous depth estimation studies.

Highly sensitive volumetric single-molecule imaging.

Volumetric subcellular imaging has long been essential for studying structures and dynamics in cells and tissues. However, due to limited imaging speed and depth of field, it has been challenging to perform live-cell imaging and single-particle tracking. Here we report a 2.5D fluorescence microscopy combined with highly inclined illumination beams, which significantly reduce not only the image acquisition time but also the out-of-focus background by ∼2-fold compared to epi-illumination. Instead of sequential z-scanning, our method projects a certain depth of volumetric information onto a 2D plane in a single shot using multi-layered glass for incoherent wavefront splitting, enabling high photon detection efficiency. We apply our method to multi-color immunofluorescence imaging and volumetric super-resolution imaging, covering ∼3-4 µm thickness of samples without z-scanning. Additionally, we demonstrate that our approach can substantially extend the observation time of single-particle tracking in living cells.

Synthetic color-and-depth encoded (sCade) illumination-based high-resolution light field particle imaging velocimetry.

Light-field particle imaging velocimetry (LF-PIV) is widely used in large-scale flow field measurement scenarios due to its instant 3D imaging capability. However, conventional LF-PIV systems suffer low axial resolution and thereby have limited application in high-resolution and volumetric velocity measurements. Here, we report the use of synthetic color-and-depth-encoded (sCade) illumination to improve the axial resolution of LF-PIV. The sCade LF-PIV illuminated the imaging region with a color-and-depth encoded beam synthesized by structured beams of three lasers with distinct wavelengths and attained high-fidelity particle localization by decoding the color and depth information encoded in the acquired image. We systematically characterized the system performance by imaging particles and obtained 29 times improvement in axial resolution when compared to traditional LF-PIV. The high axial resolution of sCade LF-PIV allowed it to reconstruct vortices generated by square lid-driven cavity flow and a stirring disk with higher accuracy and smaller errors than the conventional method, highlighting the possibility and advantage of sCade LF-PIV for high-resolution and volumetric flow measurement applications. This approach can favorably advance the development of fluid measurement technology.

Real-Time Multimodal 3D Object Detection with Transformers

The accuracy and real-time performance of 3D object detection are key factors limiting its widespread application. While cameras capture detailed color and texture features, they lack depth information compared to LiDAR. Multimodal detection combining both can improve results but incurs significant computational overhead, affecting real-time performance. To address these challenges, this paper presents a real-time multimodal fusion model called Fast Transfusion that combines the benefits of LiDAR and camera sensors and reduces the computational burden of their fusion. Specifically, our Fast Transfusion method uses QConv (Quick Convolution) to replace the convolutional backbones compared to other models. QConv concentrates the convolution operations at the feature map center, where the most information resides, to expedite inference. It also utilizes deformable convolution to better match the actual shapes of detected objects, enhancing accuracy. And the model incorporates EH Decoder (Efficient and Hybrid Decoder) which decouples multiscale fusion into intra-scale interaction and cross-scale fusion, efficiently decoding and integrating features extracted from multimodal data. Furthermore, our proposed semi-dynamic query selection refines the initialization of object queries. On the KITTI 3D object detection dataset, our proposed approach reduced the inference time by 36 ms and improved 3D AP by 1.81% compared to state-of-the-art methods.

Estimation of human body 3D pose for parent-infant interaction settings using azure Kinect and OpenPose

Automatic pose estimation has become a valuable tool for the study of human behavior, including dyadic interactions. It allows researchers to analyze the nuanced dynamics of interactions more effectively, and facilitates the integration of behavioral data with other modalities (EEG, etc.). However, many technical difficulties remain. Particularly, for parent-infant interactions, automatic pose estimation for infants is unpredictable; the immature proportions and smaller bodies of children may cause misdetections. OpenPose is one tool that has shown high performance in pose tracking from video, even in infants. However, OpenPose is limited to 2D (i.e., coordinates relative to the image space). This may be undesirable in a multitude of paradigms (e.g., naturalistic settings). We developed a method for expanding the functionality of OpenPose to 3D, tailored to parent-infant interaction paradigms. This method merges the estimations from OpenPose with the depth information from a depth camera to obtain a 3D pose that works even for young infants.•Video recordings of interactions of parents and infants are taken using a dual color-depth camera.•2D-positions of parents and their infants are estimated from the color video.•Using the depth camera, we transform the 2D estimations into real-world 3D positions, allowing movement analysis in full-3D space.