Depth Information Research Articles

Automated Weld Path Generation in Cluttered Environments Using Segmentation and Iterative Closest Point Workpiece Localization

Abstract A significant amount of manufacturing is performed by small to medium enterprises (SMEs), but these manufacturers often have lower adoption rates of automation. The cost and complexity associated with traditional robot systems slow the adoption of robotic welding operations for SMEs. The recent increase in collaborative robotic welding systems is bridging the gap, however, by reducing the complexity of installing, maintaining, and training operators to perform weld operations with collaborative robotics. These systems add flexibility in the range of operator use and ease of deployment. These systems however still rely on kinematic registration between the robot-mounted torch and workpiece on any open-loop weld. This requires precise placement of the workpiece prior to performing a weld. This work will discuss a method for part identification and registration in a welding task as a step toward automated weld path generation. The method is based on lower-resolution 3D cameras (RGBD cameras) using a combination of color and depth information. This information is used to both identify the workpiece within a workspace that may have other, non-workpiece items, and then provide registration or localization information of the workpiece within a resolution that could allow follow-on near-position strategies to achieve final weld path identification.

Camera-view supervision for bird's-eye-view semantic segmentation

Bird's-eye-view Semantic Segmentation (BEVSS) is a powerful and crucial component of planning and control systems in many autonomous vehicles. Current methods rely on end-to-end learning to train models, leading to indirectly supervised and inaccurate camera-to-BEV projections. We propose a novel method of supervising feature extraction with camera-view depth and segmentation information, which improves the quality of feature extraction and projection in the BEVSS pipeline. Our model, evaluated on the nuScenes dataset, shows a 3.8% improvement in Intersection-over-Union (IoU) for vehicle segmentation and a 30-fold reduction in depth error compared to baselines, while maintaining competitive inference times of 32 FPS. This method offers more accurate and reliable BEVSS for real-time autonomous driving systems. The codes and implementation details and code can be found at https://github.com/bluffish/sucam.

Depth-resolved imaging through scattering media using time-gated light field tomography.

We present a novel, to the best of our knowledge, approach to overcome the limitations imposed by scattering media using time-gated light field tomography. By integrating the time-gating technique with light field imaging, we demonstrate the ability to capture and reconstruct images with different depths through highly scattering environments. Our method exploits the temporal characteristics of light propagation to selectively isolate ballistic photons, enabling enhanced depth resolution and improved imaging quality. Through comprehensive experimental validation and analysis, we showcase the effectiveness of our technique in resolving depth information with high fidelity, even in the presence of significant scattering. The resultant system can simultaneously acquire multi-angled projections of the object without requiring prior knowledge of the media or the target. This advancement holds promise for a wide range of applications, including non-invasive medical imaging, environmental monitoring, and industrial inspection, where imaging through scattering media is critical for an accurate and reliable analysis.

The “Key” to Bringing DNA Collections to the Next Level: A DiSSCo Flanders Working Group Product

The DiSSCo (Distributed System of Scientific Collections) Flanders consortium, with one of the set goals being "maturing" (i.e., optimizing the management of) and unlocking (i.e., publishing) their DNA collections, identified 1) the need for actively sharing best practices on the management of DNA collections; and 2) a need for guidance on how to bring theory into practice. During the DiSSCo Flanders project, a DNA collection working group was created. The working group is open to all biodiversity-related DNA collections associates in Belgium, including those in diverse roles such as researchers, lab technicians, collection managers and data managers. Around 50 people from 13 organizations are currently participating. Members can be passively (reading only) or actively (joining events) engaged. The strength, as well as one of the challenges, of the DiSSCo Flanders community is that the natural science collections are created and managed in different organizational contexts: universities, museum institutes and both governmental and non-governmental research organizations. This translates to a variety of collection management decisions and structures such as: decentralized or centralized; cold or room temperature storage; managed by an appointed curator or by a lab technician. The working group organizes meetings and workshops, tours of each other's collections, and shares a mailing list and an online document space. As its principal output, the group has co-created: “The key to bringing DNA collections to the next level” (Veltjen et al. 2024) with two main results: the "Challenges" and the “Key”. The "Challenges" is a list of 23 challenges applicable to DNA collection management. For example, challenge 8: "Select or customise collection management systems to meet the needs of DNA collections". They are intended to spark debate and give focus to the second output: the "Key." The "Key" lists seven yes/no questions: Do you have an up-to-date overview of all direct, internal stakeholders of the institute’s DNA collection and are you involving them in the (current) intent to “bring the DNA collection to the next level”? Is preserving a DNA collection within the scope of the institute? And is the DNA collection officially recognized within the institute? Do you have, on paper, a clear description of the scope of the DNA collection? Have you outlined the current overarching workflow of the DNA collection? Have you been able to establish your starting level on the "DNA collection maturation chart" and is the assessment properly logged? Level up, one level at a time, and log the process. Have you reached all of the goals in level 3 on the "DNA collection maturation chart"? Do you have a re-evaluation strategy for your DNA collection? Do you have an up-to-date overview of all direct, internal stakeholders of the institute’s DNA collection and are you involving them in the (current) intent to “bring the DNA collection to the next level”? Is preserving a DNA collection within the scope of the institute? And is the DNA collection officially recognized within the institute? Do you have, on paper, a clear description of the scope of the DNA collection? Have you outlined the current overarching workflow of the DNA collection? Have you been able to establish your starting level on the "DNA collection maturation chart" and is the assessment properly logged? Level up, one level at a time, and log the process. Have you reached all of the goals in level 3 on the "DNA collection maturation chart"? Do you have a re-evaluation strategy for your DNA collection? The "DNA collection maturation chart" has 11 categories (rows), three levels (columns) and 33 goals (see Table 1 in Veltjen et al. 2024). The Key provides 18 guidance chapters, which give in depth information, literature and user experiences (Suppl. material 2 in Veltjen et al. 2024). The Key is a specialized tool for DNA collections. It facilitates a standardized and holistic approach, allowing both a helicopter view of the maturation process and close-up view of specific goals. The working group aims to test the Key, whereby the process of "leveling up" is embedded in a community setting: sharing ambitions, setbacks, changes of plans and success stories. The output is ready in its first version. It is published as a reviewable publication, allowing post-publication peer review (Veltjen et al. 2024). The works are expected to evolve through time, depending on user feedback and user experiences. The working group and co-created output are positive examples of how a local community—sometimes managing smaller, or less conspicuous types of natural science collections—can work together and use their unique perspectives, experiences and needs to contribute to the international natural science collection and biobanking communities.

Robust pollination for tomato farming using deep learning and visual servoing

Abstract In this work, we propose a novel approach for tomato pollination that utilizes visual servo control. The objective is to meet the growing demand for automated robotic pollinators to overcome the decline in bee populations. Our approach focuses on addressing this challenge by leveraging visual servo control to guide the pollination process. The proposed method leverages deep learning to estimate the orientations and depth of detected flower, incorporating CAD-based synthetic images to ensure dataset diversity. By utilizing a 3D camera, the system accurately estimates flower depth information for visual servoing. The robustness of the approach is validated through experiments conducted in a laboratory environment with a 3D printed tomato flower plant. The results demonstrate a high detection rate, with a mean average precision of 91.2 %. Furthermore, the average depth error for accurately localizing the pollination target is impressively minimal, measuring only 1.1 cm. This research presents a promising solution for tomato pollination, showcasing the effectiveness of visual-guided servo control and its potential to address the challenges posed by diminishing bee populations in greenhouses.

Polarimetric 3D integral imaging profilometry under degraded environmental conditions

We propose polarimetric three-dimensional (3D) integral imaging profilometry and investigate its performance under degraded environmental conditions in terms of the accuracy of object depth acquisition. Integral imaging based profilometry provides depth information by capturing and utilizing multiple perspectives of the observed object. However, the performance of depth map generation may degrade due to light condition, partial occlusions, and object surface material. To improve the accuracy of depth estimation in these conditions, we propose to use polarimetric profilometry. Our experiments indicate that the proposed approach may result in more accurate depth estimation under degraded environmental conditions. We measure a number of metrics to evaluate the performance of the proposed polarimetric profilometry methods for generating the depth map under degraded conditions. Experimental results are presented to evaluate the robustness of the proposed method under degraded environment conditions and compare its performance with conventional integral imaging. To the best of our knowledge, this is the first report on polarimetric 3D integral imaging profilometry, and its performance under degraded environments.

MonoFG: Monocular 3D Object Detection with Knowledge Distillation for Human-Centric Autonomous Driving Systems

Monocular 3D object detection is essential for identifying objects in road images, thus offering valuable environmental perception data that are crucial for human-centric autonomous driving systems. However, due to the inherent limitations of camera imaging, obtaining precise depth information from images alone is challenging, which hampers the accuracy of within-scene object localization. In this paper, we introduce a monocular 3D object detection method called MonoFG that uses knowledge distillation with separated foreground and background components to improve the accuracy of object localization. First, detached foreground and background distillation processes can strategically leverage the distinct positional information acquired from each location to optimize the produced global distillation effects. This step serves as the foundation for the subsequent feature and response distillation process, which focuses on the distilled foreground and background rather than isolated object distillation. Second, triple attention mechanism-based feature distillation intensifies the feature imitation and feature representation capabilities of the student network. Spatial and channel attention mechanisms encourage the student network to capture crucial pixels and channels from the teacher network, whereas a self-attention mechanism globally transfers the learned relationships between pixels. Third, localization error-based response distillation facilitates a clearer transfer of positional information from the teacher network to the student network. Only when the positioning ability of the teacher network exceeds that of the student network can knowledge be comprehensively distilled across both the foreground and background. Therefore, the distillation process is constrained to specific content, which is delineated by positioning errors that serve as the boundaries. Finally, experiments conducted on the KITTI benchmark dataset demonstrate that our method outperforms many well-known baseline methods in several representative evaluation tasks (e.g., 3D object detection and bird's-eye view (BEV) detection) involving human-centric autonomous driving systems.

Global Semantic Localization from Abstract Ellipse-Ellipsoid Model and Object-Level Instance Topology

Robust and highly accurate localization using a camera is a challenging task when appearance varies significantly. In indoor environments, changes in illumination and object occlusion can have a significant impact on visual localization. In this paper, we propose a visual localization method based on an ellipse-ellipsoid model, combined with object-level instance topology and alignment. First, we develop a CNN-based (Convolutional Neural Network) ellipse prediction network, DEllipse-Net, which integrates depth information with RGB data to estimate the projection of ellipsoids onto images. Second, we model environments using 3D (Three-dimensional) ellipsoids, instance topology, and ellipsoid descriptors. Finally, the detected ellipses are aligned with the ellipsoids in the environment through semantic object association, and 6-DoF (Degree of Freedom) pose estimation is performed using the ellipse-ellipsoid model. In the bounding box noise experiment, DEllipse-Net demonstrates higher robustness compared to other methods, achieving the highest prediction accuracy for 11 out of 23 objects in ellipse prediction. In the localization test with 15 pixels of noise, we achieve ATE (Absolute Translation Error) and ARE (Absolute Rotation Error) of 0.077 m and 2.70∘ in the fr2_desk sequence. Additionally, DEllipse-Net is lightweight and highly portable, with a model size of only 18.6 MB, and a single model can handle all objects. In the object-level instance topology and alignment experiment, our topology and alignment methods significantly enhance the global localization accuracy of the ellipse-ellipsoid model. In experiments involving lighting changes and occlusions, our method achieves more robust global localization compared to the classical bag-of-words based localization method and other ellipse-ellipsoid localization methods.

Speech recognition using an english multimodal corpus with integrated image and depth information

Traditional English corpora mainly collect information from a single modality, but lack information from multimodal information, resulting in low quality of corpus information and certain problems with recognition accuracy. To solve the above problems, this paper proposes to introduce depth information into multimodal corpora, and studies the construction method of English multimodal corpora that integrates electronic images and depth information, as well as the speech recognition method of the corpus. The multimodal fusion strategy adopted integrates speech signals and image information, including key visual information such as the speaker’s lip movements and facial expressions, and uses deep learning technology to mine acoustic and visual features. The acoustic model in the Kaldi toolkit is used for experimental research.Through experimental research, the following conclusions were drawn: Under 15-dimensional lip features, the accuracy of corpus A under monophone model was 2.4% higher than that of corpus B under monophone model when the SNR (signal-to-noise ratio) was 10dB, and the accuracy of corpus A under the triphone model at the signal-to-noise ratio of 10dB was 1.7% higher than that of corpus B under the triphone model at the signal-to-noise ratio of 10dB. Under the 32-dimensional lip features, the speech recognition effect of corpus A under the monophone model at the SNR of 10dB was 1.4% higher than that of corpus B under the monophone model at the SNR of 10dB, and the accuracy of corpus A under the triphone model at the SNR of 10dB was 2.6% higher than that of corpus B under the triphone model at the SNR of 10dB. The English multimodal corpus with image and depth information has a high accuracy, and the depth information helps to improve the accuracy of the corpus.

DETrack: Depth information is predictable for tracking

DETrack: Depth information is predictable for tracking

Insights into the efficiencies of different biological treatment systems for pharmaceuticals removal: A review.

This review presents a comprehensive analysis of current research on biological treatment processes for removing pharmaceutical compounds (PhCs) from wastewater. Unlike previous studies on this topic, our study specifically delves into the effectiveness and drawbacks of various treatment approaches such as traditional wastewater treatment facilities (WWTP), membrane bioreactors (MBRs), constructed wetlands (CW), and moving bed biofilm reactors (MBBR). Through the examination and synthesis of information gathered from more than 200 research studies, we have created a comprehensive database that delves into the effectiveness of eliminating 19 particular PhCs, including commonly studied compounds such as acetaminophen, ibuprofen, diclofenac, naproxen, ketoprofen, indomethacin, salicylic acid, codeine, and fenoprofen, amoxicillin, azithromycin, ciprofloxacin, ofloxacin, tetracycline, atenolol, propranolol, and metoprolol. This resource provides a depth and scope of information that was previously lacking in this area of study. Notably, among these pharmaceuticals, azithromycin demonstrated the highest removal rates across all examined treatment systems, with the exception of WWTPs, while carbamazepine consistently exhibited the lowest removal efficiencies across various systems. The analysis showcases the diverse results in removal efficiency impacted by factors such as system configuration, operation specifics, and environmental circumstances. The findings emphasize the critical need for continued innovation and research, specifically recommending the integration of advanced oxidation processes (AOPs) with existing biological treatment methods to improve the breakdown of recalcitrant compounds like carbamazepine. PRACTITIONER POINTS: Persistent pharmaceuticals harm aquatic ecosystems and human health. Biological systems show varying pharmaceutical removal efficiencies. Enhancing HRT and SRT improves removal but adds complexity and costs. Tailored treatment approaches needed based on contaminants and conditions.

Estimating TYLCV resistance level using RGBD sensors in production greenhouse conditions

Automated phenotyping is the task of automatically measuring plant attributes to help farmers and breeders in developing and growing strong robust plants. An automated tool for early illness detection can accelerate the process of identifying plant resistance and quickly pinpoint problematic breeding. Many such phenotyping tasks can be achieved by analyzing images from simple, low cost, RGB-D sensors. In this paper we focused on a particular case study — identifying the resistance level of tomato hybrids to the tomato yellow leaf curl virus (TYLCV) in production greenhouses. This is a difficult task, as separating between resistance levels based on images is difficult even for expert breeders. We collected a large dataset of images from an experiment containing many tomato hybrids with varying resistance levels. We used the depth information to identify the topmost part of the tomato plant. We then used deep learning models to classify the various resistance levels. For identifying plants with visual symptoms, our methods achieved an accuracy of 0.928, a precision of 0.934, and a recall of 0.95. In the multi-class case we achieved an accuracy of 0.76 in identifying the correct level, and an error of 0.278. Our methods are not particularly tailored for the specific task, and can be extended to other tasks that identify various plant diseases with visual symptoms such as ToBRFV,mildew, ToMV and others.

Inhibition of return in a 3D scene depends on the direction of depth switch between cue and target.

Inhibition of return (IOR) is a phenomenon that reflects slower target detection when the target appears at a previously cued rather than uncued location. In the present study, we investigated the extent to which IOR occurs in three-dimensional (3D) scenes comprising pictorial depth information. Peripheral cues and targets appeared on top of 3D rectangular boxes placed on the surface of a textured ground plane in virtual space. When the target appeared at a farther location than the cue, the magnitude of the IOR effect in the 3D condition remained similar to the values found in the two-dimensional (2D) control condition (IOR was depth-blind). When the target appeared at a nearer location than the cue, the magnitude of the IOR effect was significantly attenuated (IOR was depth-specific). The present findings address inconsistencies in the literature on the effect of depth on IOR and support the notion that visuospatial attention exhibits a near-space advantage even in 3D scenes consisting entirely of pictorial depth information.

Feature fusion and context interaction for RGB-D indoor semantic segmentation

Depth information in indoor scenes is crucial to understanding the spatial and occluding relationships of objects. Depth information contains rich geometric details and is unaffected by lighting, texture, and color, significantly enhancing traditional color images. RGB images and depth images have been widely applied in various image analysis. However, effectively leveraging the complementary information from both modalities remains a challenge. To address this issue, we propose a unified and effective feature fusion and context interaction network (FCINet). It includes three modules. Firstly, we construct a multi-modal feature fusion module (MFFM), which can achieve the aggregation of two modalities along spatial and channel dimensions. Secondly, we construct a global and local information interaction context module (GLCM) to encode rich semantic information. Finally, we construct a feature alignment and fusion module (FAFM), which integrates upsampled low-level features with high-level features to mitigate spatial information loss. Experimental results indicate that the model achieved favorable outcomes on both the NYU Depth v2 dataset and the more complex SUN RGB-D dataset.

Map construction algorithm based on dense point cloud

AbstractIn order to address the issue of the difficulty in constructing maps from sensor‐acquired point clouds, a map construction algorithm is proposed for the effective conversion from sparse point clouds to dense point clouds. First, it constructs a dense point cloud map in real‐time using colour and depth information of keyframes. Then, it generates an accurate point cloud map by voxel filtering and coordinate conversion. Finally, the point cloud map is converted into an octree map using OctoMap. This article uses the TUM dataset for simulation analysis to verify the effectiveness of the algorithm. The results demonstrate that the algorithm can construct dense point cloud maps with high positioning accuracy in sparse point cloud scenarios. The algorithm improves the positioning accuracy by more than 15% under the fr1_xyz, fr1_room, fr1_desk2, fr1_desk1, fr2_desk and fr3_str_tex_near sequences, compared to ORB‐SLAM2, ORB‐SLAM3 and GX ORB‐SLAM2 algorithm

Motion Recognition Based on Deep Learning Algorithm

Advanced scientific and technological measurement methods are the basis of sports research and the progress of sports research methods. It promotes the continuous improvement of sports technology to meet athletes’ pursuit of excellent sports achievements. Therefore, the method of sports scientific research must be changed from the traditional empirical training mode to the programmed training method, from the qualitative analysis of the training effect to the nuanced analysis of the training process. Based on this, this paper uses in-depth learning to study sports motion capture. First, two-dimensional human joints are extracted based on Mask R-CNN. Then, the 3D human motion skeleton is constructed by using the binocular vision system, and the Mask R-CNN human pose estimation algorithm is optimized. On this basis, a sports motion capture system is designed, and the system’s accuracy is verified. The error of the depth information obtained by the sports capture system constructed in this paper is less than 3% in the experiment of about 2[Formula: see text]m. It has strong practicability.

Effect of attention on ensemble perception: Comparison between exogenous attention, endogenous attention, and depth.

Ensemble perception is an important ability of human beings that allows one to extract summary information for scenes and environments that contain information that far exceeds the processing limit of the visual system. Although attention has been shown to bias ensemble perception, two important questions remain unclear: (1) whether direct manipulations on different types of spatial attention could produce similar effects on ensembles and (2) whether factors potentially influencing the attention distribution, such as depth perception, could evoke an indirect effect of attention on ensemble representation. This study aims to address these questions. In Experiments 1 and 2, two types of precues were used to evoke exogenous and endogenous attention, respectively, and the ensemble color perceptions were examined. We found that both exogenous and endogenous attention biased ensemble representation towards the attended items, and the latter produced a greater effect. In Experiments 3 and 4, we examined whether depth perception could affect color ensembles by indirectly influencing attention allocation in 3D space. The items were separated in two depth planes, and no explicit cues were applied. The results showed that color ensemble was biased to closer items when depth information was task relevant. This suggests that ensemble perception is naturally biased in 3D space, probably through the mechanism of attention. Computational modeling consistently showed that attention exerted a direct shift on the ensemble statistics rather than averaging the feature values over the cued and noncued items, providing evidence against an averaging process of individual perception.

The effect of the product categories diversity recommended on cross‐buying in electronic commerce platforms: The moderating role of user navigation heterogeneity

AbstractE‐commerce platforms encourage consumers' cross‐buying behavior to boost user traffic and trading volume. However, balancing the precision and richness of recommended information with the mechanisms influencing consumer cross‐buying remains unclear, which also poses challenges to the specific operation of these platforms. Therefore, this study leverages extensive online behavioral data from an e‐commerce platform, encompassing detailed records of 26,034 consumers. By employing unsupervised machine learning algorithms to distinguish the heterogeneity of consumer browsing, we apply clickstream data to explore the mechanisms of the impact of information diversity, specifically the breadth and depth of recommended product categories, on cross‐buying. This study reveals that the effect of diverse recommendations on online decision‐making undergoes marginal variations due to alterations in both the breadth and depth of these recommendations. Specifically, the depth and breadth of recommended product categories exhibit an inverted U‐shaped relationship with cross‐buying, initially promoting, and subsequently suppressing it. Additionally, consumers with different browsing behavior characteristics respond differently to recommendation diversity. Confused visitors and hedonic visitors positively moderate the depth of information and negatively moderate the breadth of recommendations with the inverted U‐shaped effect on cross‐buying. Search‐oriented visitors positively adjust the inverted U‐shaped relationship between the depth of information and the breadth of recommendations on consumer cross‐buying. This study enriches research on cross‐buying in the domain of recommendations on e‐commerce platforms from the perspective of information diversity, providing valuable insights for optimizing recommendation strategies, promoting cross‐buying behavior, and enhancing consumer loyalty.

Real-time volume rendering for three-dimensional fetal ultrasound using volumetric photon mapping

Three-dimensional (3D) fetal ultrasound has been widely used in prenatal examinations. Realistic and real-time volumetric ultrasound volume rendering can enhance the effectiveness of diagnoses and assist obstetricians and pregnant mothers in communicating. However, this remains a challenging task because (1) there is a large amount of speckle noise in ultrasound images and (2) ultrasound images usually have low contrasts, making it difficult to distinguish different tissues and organs. However, traditional local-illumination-based methods do not achieve satisfactory results. This real-time requirement makes the task increasingly challenging. This study presents a novel real-time volume-rendering method equipped with a global illumination model for 3D fetal ultrasound visualization. This method can render direct illumination and indirect illumination separately by calculating single scattering and multiple scattering radiances, respectively. The indirect illumination effect was simulated using volumetric photon mapping. Calculating each photon’s brightness is proposed using a novel screen-space destiny estimation to avoid complicated storage structures and accelerate computation. This study proposes a high dynamic range approach to address the issue of fetal skin with a dynamic range exceeding that of the display device. Experiments show that our technology, compared to conventional methodologies, can generate realistic rendering results with far more depth information.

Adversarial Unsupervised Domain Adaptation for 3D Semantic Segmentation with 2D Image Fusion of Dense Depth

AbstractUnsupervised domain adaptation (UDA) is increasingly used for 3D point cloud semantic segmentation tasks due to its ability to address the issue of missing labels for new domains. However, most existing unsupervised domain adaptation methods focus only on uni‐modal data and are rarely applied to multi‐modal data. Therefore, we propose a cross‐modal UDA on multi‐modal datasets that contain 3D point clouds and 2D images for 3D Semantic Segmentation. Specifically, we first propose a Dual discriminator‐based Domain Adaptation (Dd‐bDA) module to enhance the adaptability of different domains. Second, given that the robustness of depth information to domain shifts can provide more details for semantic segmentation, we further employ a Dense depth Feature Fusion (DdFF) module to extract image features with rich depth cues. We evaluate our model in four unsupervised domain adaptation scenarios, i.e., dataset‐to‐dataset (A2D2 → SemanticKITTI), Day‐to‐Night, country‐to‐country (USA → Singapore), and synthetic‐to‐real (VirtualKITTI → SemanticKITTI). In all settings, the experimental results achieve significant improvements and surpass state‐of‐the‐art models.