Depth Fusion Research Articles

Adversarial Unsupervised Domain Adaptation for 3D Semantic Segmentation with 2D Image Fusion of Dense Depth

AbstractUnsupervised domain adaptation (UDA) is increasingly used for 3D point cloud semantic segmentation tasks due to its ability to address the issue of missing labels for new domains. However, most existing unsupervised domain adaptation methods focus only on uni‐modal data and are rarely applied to multi‐modal data. Therefore, we propose a cross‐modal UDA on multi‐modal datasets that contain 3D point clouds and 2D images for 3D Semantic Segmentation. Specifically, we first propose a Dual discriminator‐based Domain Adaptation (Dd‐bDA) module to enhance the adaptability of different domains. Second, given that the robustness of depth information to domain shifts can provide more details for semantic segmentation, we further employ a Dense depth Feature Fusion (DdFF) module to extract image features with rich depth cues. We evaluate our model in four unsupervised domain adaptation scenarios, i.e., dataset‐to‐dataset (A2D2 → SemanticKITTI), Day‐to‐Night, country‐to‐country (USA → Singapore), and synthetic‐to‐real (VirtualKITTI → SemanticKITTI). In all settings, the experimental results achieve significant improvements and surpass state‐of‐the‐art models.

Radiation Anomaly Detection of Sub-Band Optical Remote Sensing Images Based on Multiscale Deep Dynamic Fusion and Adaptive Optimization

Radiation anomalies in optical remote sensing images frequently occur due to electronic issues within the image sensor or data transmission errors. These radiation anomalies can be categorized into several types, including CCD, StripeNoise, RandomCode1, RandomCode2, ImageMissing, and Tap. To ensure the retention of image data with minimal radiation issues as much as possible, this paper adopts a self-made radiation dataset and proposes a FlexVisionNet-YOLO network to detect radiation anomalies more accurately. Firstly, RepViT is used as the backbone network with a vision transformer architecture to better capture global and local features. Its multiscale feature fusion mechanism efficiently handles targets of different sizes and shapes, enhancing the detection ability for radiation anomalies. Secondly, a feature depth fusion network is proposed in the Feature Fusion part, which significantly improves the flexibility and accuracy of feature fusion and thus enhances the detection and classification performance of complex remote sensing images. Finally, Inner-CIoU is used in the Head part for edge regression, which significantly improves the localization accuracy by finely adjusting the target edges; Slide-Loss is used for classification loss, which enhances the classification robustness by dynamically adjusting the category probabilities and markedly improves the classification accuracy, especially in the sample imbalance dataset. Experimental results show that, compared to YOLOv8, the proposed FlexVisionNet-YOLO method improves precision, recall, mAP0.5, and mAP0.5:0.9 by 3.5%, 7.1%, 4.4%, and 13.6%, respectively. Its effectiveness in detecting radiation anomalies surpasses that of other models.

Improving real-time apple fruit detection: Multi-modal data and depth fusion with non-targeted background removal

In automated fruit detection, RGB-Depth (RGB-D) images aid the detection model with additional depth information to enhance detection accuracy. However, outdoor depth images are usually of low quality, which limits the quality of depth data. In this study, an approach/technique for real-time apple fruit detection in a high-density orchard environment by using multi-modal data is presented. Non-targeted background removal using the depth fusion (NBR-DF) method was developed to reduce the high noise condition of depth images. The noise occurred due to the uncontrolled lighting condition and holes with incomplete depth information in the depth images. NBR-DF technique follows three primary steps: pre-processing of depth images (point cloud generation), target object extraction, and background removal. The NBR-DF method serves as a pipeline to pre-process multi-modal data to enhance features of depth images by filling holes to eliminate noise generated by depth holes. Further, the NBR-DF implemented with the YOLOv5 enhances the detection accuracy in dense orchard conditions by using multi-modal information as input. An attention-based depth fusion module that adaptively fuses the multi-modal features was developed. The integration of the depth-attention matrix involved pooling operations and sigmoid normalization, both of which are efficient methods for summarizing and normalizing depth information. The fusion module improves the identification of multiscale objects and strengthens the network's resistance to noise. The network then detects the fruit position using multiscale information from the RGB-D images in highly complex orchard environments. The detection results were compared and validated with other methods using different input modals and fusion strategies. The results showed that the detection accuracy using the NBR-DF approach achieved an average precision rate of 0.964 in real time. The performance comparison with other state-of-the-art methods and the model generalization study also establish that the present advanced depth-fusion attention mechanism and effective preprocessing steps in NBR-DF-YOLOv5 significantly surpass those in performance. In conclusion, the developed NBR-DF technique showed the potential to improve real-time apple fruit detection using multi-modal information.

Deep learning model based on molecular spectra to determine chlorpyrifos residues in corn oil

This study presents a novel molecular spectroscopy depth fusion strategy for rapid detection of residual chlorpyrifos in corn oil. Raman and Fourier transform near-infrared (FT-NIR) spectrometers were employed to collect contaminated corn oil sample spectra. The multi-spectral fusion enhances spectral features, improving data reliability and outcomes. The research employed low and mid-level fusion strategies alongside a one-dimensional convolutional neural network (1D-CNN) for modeling. The mid-level data fusion model outperformed the low-level model, achieving a relative percent deviation (RPD) of 11.6517 and a prediction coefficient of determination (RP2) of 0.9874. The results indicate that the mid-level fusion strategy is suitable for multivariate model calibration of Raman and FT-NIR, providing a rapid detection method for pesticide residues in edible oils. This study offers a method reference for fusion strategies in molecular spectroscopy and expands the application scope of deep learning in spectroscopy.

Robot Unknown Objects Instance Segmentation Based on Collaborative Weight Assignment RGB–Depth Fusion Strategy

Robot Unknown Objects Instance Segmentation Based on Collaborative Weight Assignment RGB–Depth Fusion Strategy

MRFTrans: Multimodal Representation Fusion Transformer for monocular 3D semantic scene completion

The complete understanding of 3D scenes is crucial in robotic visual perception, impacting tasks such as motion planning and map localization. However, due to the limited field of view and scene occlusion constraints of sensors, inferring complete scene geometry and semantic information from restricted observations is challenging. In this work, we propose a novel Multimodal Representation Fusion Transformer framework (MRFTrans) that robustly fuses semantic, geometric occupancy, and depth representations for monocular-image-based scene completion. MRFTrans centers on an affinity representation fusion transformer, integrating geometric occupancy and semantic relationships within a transformer architecture. This integration enables the modeling of long-range dependencies within scenes for inferring missing information. Additionally, we present a depth representation fusion method, efficiently extracting reliable depth knowledge from biased monocular estimates. Extensive experiments demonstrate MRFTrans’s superiority, setting a new benchmark on SemanticKITTI and NYUv2 datasets. It significantly enhances completeness and accuracy, particularly in large structures, movable objects, and scene components with major occlusions. The results underscore the benefits of the affinity-aware transformer and robust depth fusion in monocular-image-based completion.

RCDformer: Transformer-based dense depth estimation by sparse radar and camera

Accurate depth cues are crucial for 3D perception tasks, and monocular depth estimation networks are no longer sufficient for realistic scenarios. Currently, the most effective approaches are to introduce depth information from other modalities into the image. Radar has become a popular sensor for fusion with cameras due to its low price and all-weather working characteristics. This paper aims to explore how to more effectively integrate the heterogeneous data of radar point clouds and RGB images to improve the performance of depth estimation. Most of the previous works have not fully exploited the potential of integrating these two modalities, so we propose RCDformer, a novel network based on the transformer architecture that fuses radar-camera for dense depth estimation. Without reducing the receptive field, our approach can fully model the contextual relationships between sensors to reduce the impact of radar noise on overall performance. With the proposed Radar-guided Multi-scale Depth Fusion (RGDF) module, the prior spatial information mapped by the Radar Feature Extractor (RFE) is embedded into a set of multi-scale hierarchical features output by Image Feature Extractor (IFE) via the modified deformable cross-attention, which aims to guide the depth prediction of images. Furthermore, we discover that incorporating the Radar Cross Section (RCS) attribute as an extended channel for the radar map is beneficial for dense depth estimation, which improves the overall performance of our model. We evaluate the proposed method on the nuScenes dataset, and the experiment results show that our method still achieves significant advantages in most metrics compared to the state-of-the-art models.

Enhancing automated lower limb rehabilitation exercise task recognition through multi-sensor data fusion in tele-rehabilitation

BackgroundTele-rehabilitation is the provision of physiotherapy services to individuals in their own homes. Activity recognition plays a crucial role in the realm of automatic tele-rehabilitation. By assessing patient movements, identifying exercises, and providing feedback, these platforms can offer insightful information to clinicians, thereby facilitating an improved plan of care. This study introduces a novel deep learning approach aimed at identifying lower limb rehabilitation exercises. This is achieved through the integration of depth data and pressure heatmaps. We hypothesized that combining pressure heatmaps and depth data could improve the model’s overall performance.MethodsIn this study, depth videos and body pressure data from an accessible online dataset were used. This dataset comprises data from 30 healthy individuals performing 7 lower limb rehabilitation exercises. To accomplish the classification task, three deep learning models were developed, all based on an established 3D-CNN architecture. The models were designed to classify the depth videos, sequences of pressure data frames, and combination of depth videos and pressure frames. The models’ performance was assessed through leave-one-subject-out and leave-multiple-subjects-out cross-validation methods. Performance metrics, including accuracy, precision, recall, and F1 score, were reported for each model.ResultsOur findings indicated that the model trained on the fusion of depth and pressure data showed the highest and most stable performance when compared with models using individual modality inputs. This model could effectively identify the exercises with an accuracy of 95.71%, precision of 95.83%, recall of 95.71%, and an F1 score of 95.74%.ConclusionOur results highlight the impact of data fusion for accurately classifying lower limb rehabilitation exercises. We showed that our model could capture different aspects of exercise movements using the visual and weight distribution data from the depth camera and pressure mat, respectively. This integration of data provides a better representation of exercise patterns, leading to higher classification performance. Notably, our results indicate the potential application of this model in automatic tele-rehabilitation platforms.

Liquid lens based holographic camera for real 3D scene hologram acquisition using end-to-end physical model-driven network

With the development of artificial intelligence, neural network provides unique opportunities for holography, such as high fidelity and dynamic calculation. How to obtain real 3D scene and generate high fidelity hologram in real time is an urgent problem. Here, we propose a liquid lens based holographic camera for real 3D scene hologram acquisition using an end-to-end physical model-driven network (EEPMD-Net). As the core component of the liquid camera, the first 10 mm large aperture electrowetting-based liquid lens is proposed by using specially fabricated solution. The design of the liquid camera ensures that the multi-layers of the real 3D scene can be obtained quickly and with great imaging performance. The EEPMD-Net takes the information of real 3D scene as the input, and uses two new structures of encoder and decoder networks to realize low-noise phase generation. By comparing the intensity information between the reconstructed image after depth fusion and the target scene, the composite loss function is constructed for phase optimization, and the high-fidelity training of hologram with true depth of the 3D scene is realized for the first time. The holographic camera achieves the high-fidelity and fast generation of the hologram of the real 3D scene, and the reconstructed experiment proves that the holographic image has the advantage of low noise. The proposed holographic camera is unique and can be used in 3D display, measurement, encryption and other fields.

Three-dimentional reconstruction of underwater side-scan sonar images based on shape-from-shading and monocular depth fusion

Modern marine research requires high-precision three-dimensional (3D) underwater data. Underwater environments experience severe visible light attenuation, which leads to inferior imaging compared with air. In contrast, sound waves are less affected underwater; hence side-scan sonar is used for underwater 3D reconstruction. Typically, the shape-from-shading algorithm (SfS) is widely used to reconstruct surface normal or heights from side-scan sonar images. However, this approach has challenges because of global information loss and noise. To address these issues, this study introduces a surface-normal fusion method. Specifically, we propose a frequency separation SfS algorithm using a discrete cosine transform, which provides a surface-normal map with less noise. We then fuse the surface-normal map with a novel depth estimation network to achieve high-precision 3D reconstruction of underwater side-scan sonar images. We conducted experiments on synthetic, NYU-depth-v2, and real side-scan sonar datasets to demonstrate the effectiveness of the proposed method.

Multiscale Cross-Connected Dehazing Network With Scene Depth Fusion.

In this article, we propose a multiscale cross-connected dehazing network with scene depth fusion. We focus on the correlation between a hazy image and the corresponding depth image. The model encodes and decodes the hazy image and the depth image separately and includes cross connections at the decoding end to directly generate a clean image in an end-to-end manner. Specifically, we first construct an input pyramid to obtain the receptive fields of the depth image and the hazy image at multiple levels. Then, we add the features of the corresponding dimensions in the input pyramid to the encoder. Finally, the two paths of the decoder are cross-connected. In addition, the proposed model uses wavelet pooling and residual channel attention modules (RCAMs) as components. A series of ablation experiments shows that the wavelet pooling and RCAMs effectively improve the performance of the model. We conducted extensive experiments on multiple dehazing datasets, and the results show that the model is superior to other advanced methods in terms of peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and subjective visual effects. The source code and supplementary are available at https://github.com/CCECfgd/MSCDN-master.

A Collaborative Visual Sensing System for Precise Quality Inspection at Manufacturing Lines

Visual sensing has been widely adopted for quality inspection in production processes. This paper presents the design and implementation of a smart collaborative camera system, called BubCam , for automated quality inspection of manufactured ink bags in Hewlett-Packard (HP) Inc.’s factories. Specifically, BubCam estimates the volume of air bubbles in an ink bag, which may affect the printing quality. The design of BubCam faces challenges due to the dynamic ambient light reflection, motion blur effect, and data labeling difficulty. As a starting point, we design a single-camera system which leverages various deep learning (DL)-based image segmentation and depth fusion techniques. New data labeling and training approaches are proposed to utilize prior knowledge of the production system for training the segmentation model with a small dataset. Then, we design a multi-camera system which additionally deploys multiple wireless cameras to achieve better accuracy due to multi-view sensing. To save power of the wireless cameras, we formulate a configuration adaptation problem and develop the single-agent and multi-agent deep reinforcement learning (DRL)-based solutions to adjust each wireless camera’s operation mode and frame rate in response to the changes of presence of air bubbles and light reflection. The multi-agent DRL approach aims to reduce the retraining costs during the production line reconfiguration process by only retraining the DRL agents for the newly added cameras and the existing cameras with changed positions. Extensive evaluation on a lab testbed and real factory trial shows that BubCam outperforms six baseline solutions including the current manual inspection and existing bubble detection and camera configuration adaptation approaches. In particular, BubCam achieves 1.3x accuracy improvement and 300x latency reduction, compared with the manual inspection approach.

IterDepth: Iterative Residual Refinement for Outdoor Self-Supervised Multi-Frame Monocular Depth Estimation

Self-supervised monocular depth estimation has been a challenging task in computer vision for a long time, and it relies on only monocular or stereo video for its supervision. To address the challenge, we propose a novel multi-frame monocular depth estimation method called IterDepth, which is based on an iterative residual refinement network. IterDepth extracts depth features from consecutive frames and computes a 3D cost volume measuring the difference between current and previous features transformed by PoseCNN (pose estimation convolutional neural network). We reformulate depth prediction as a residual learning problem, revamping the dominating depth regression to enable high-accuracy multi-frame monocular depth estimation. Specifically, we design a gated recurrent depth fusion unit that seamlessly blends depth features from the cost volume, image features, and the depth prediction. The unit updates the hidden states and refines the depth map through iterative refinement, achieving more accurate predictions than existing methods. Our experiments on the KITTI dataset demonstrate that IterDepth is 7× faster in terms of FPS (frames per second) than the recent state-of-the-art DepthFormer model with competitive performance. We also test IterDepth on the Cityscapes dataset to showcase its generalization capability in other real-world environments. Moreover, IterDepth can balance accuracy and computational efficiency by adjusting the number of refinement iterations and performs competitively with other CNN-based monocular depth estimation approaches. Code will be made publicly available.

Exploration of ideological and political education of “Customs Management Function” course based on PBL teaching method

This article in the “introduction to customs management” in the “customs management functions” of the teaching content as the carrier, the PBL teaching method and ideological and political education depth fusion, explore the course ideological and political education design and implementation, cultivate students good professional ethics, law-abiding and law enforcement consciousness, stimulate students enthusiasm for the profession and Chinese feelings, enhance their loyalty to the Customs, social sense of responsibility, so as to fulfill the corresponding international obligations.

​​ACTIVITIES DESIGN STRATEGIES OF BLENDED ENGLISH GRAMMAR TEACHING IN HIGHER VOCATIONAL COLLEGES​

The outbreak of COVID-19 has made online teaching an important part of daily teaching. In the post-epidemic period, blended teaching will inevitably become a new teaching form in teaching activities, bringing a new classroom revolution and learning revolution to English grammar teaching in higher vocational colleges. Therefore, based on the characteristics analysis of EFL learners’ learning behavior, colleges and universities should transform the teaching methods with the combination of information technology and depth fusion of education teaching. It is important to make full use of the Massive Open Online Course (MOOC), Small Private Online Course (SPOC) and flip classroom teaching methods by the cohesion of course activities to improve the teaching paradigm online. Besides, with the help of intelligent platform to make out the learning behavior analysis data is also available which is benefits to content visualization, network class differentiation, entity classroom interaction and intelligent teaching management, realizing the "internet + intelligent" education, to build more high-quality courses. This paper is to adopt two classes students by different teaching methods, which are blended teaching method and traditional teaching method to explore the students’ learning behavior influential factors and activity design for blended teaching mode framework. Based on SPOC blended teaching course into the actual grammar teaching, this study discusses the English grammar blended teaching strategy for EFL learners. The aim is to improve the teaching efficiency of teachers and students' learning ability, and provide theoretical guidance for the design of the blended teaching mode of English grammar in higher vocational colleges.

Three-dimension object detection and forward-looking control strategy for non-destructive grasp of thin-skinned fruits

The dynamic non-destructive grasp of thin-skinned fruits using flexible robotic hands, which requires obtaining three-dimension(3D) spatial structure information along with adaptive planning and motion control toward the target object, is a challenging topic in agricultural intelligence. To tackle the issue of 3D detection, we utilize RGB images and LiDAR point clouds for feature extraction and construct a multi-modal depth fusion convolution neural network (MDF-CNN) to obtain the classification information and perform image segmentation. Incorporating the advantages of a variable palm structure, we establish an evaluation mechanism of the optimal grasping stability (EM-OGS) using a hybrid method of the best configuration and force closure to build a new comprehensive performance optimal configuration planning (CPO-CP) method that is based on the multiple grasping performance indexes. We also create three cross-related nonlinear prediction models P-MGF, P-ODAP, and P-OBA along with a forward-looking non-destructive grasp control algorithm (FL-NGCA) using minimum grasping force to grasp thin-skinned fruits. The control algorithm carries out online, self-directed learning in the actual grasping process of the flexible hand and constantly optimizes the accuracy of the prediction model. Experimental results show that our proposed approach greatly improves the flexible hand comprehensive performance of grasping, outperforming state-of-the-art methods for non-destructive grasping of delicate fruits in most areas.

Improving Lettuce Fresh Weight Estimation Accuracy through RGB-D Fusion

Computer vision provides a real-time, non-destructive, and indirect way of horticultural crop yield estimation. Deep learning helps improve horticultural crop yield estimation accuracy. However, the accuracy of current estimation models based on RGB (red, green, blue) images does not meet the standard of a soft sensor. Through enriching more data and improving the RGB estimation model structure of convolutional neural networks (CNNs), this paper increased the coefficient of determination (R2) by 0.0284 and decreased the normalized root mean squared error (NRMSE) by 0.0575. After introducing a novel loss function mean squared percentage error (MSPE) that emphasizes the mean absolute percentage error (MAPE), the MAPE decreased by 7.58%. This paper develops a lettuce fresh weight estimation method through the multi-modal fusion of RGB and depth (RGB-D) images. With the multimodal fusion based on calibrated RGB and depth images, R2 increased by 0.0221, NRMSE decreased by 0.0427, and MAPE decreased by 3.99%. With the novel loss function, MAPE further decreased by 1.27%. A MAPE of 8.47% helps to develop a soft sensor for lettuce fresh weight estimation.

Fast 3D Reconstruction of UAV Images Based on Neural Radiance Field

Traditional methods for 3D reconstruction of unmanned aerial vehicle (UAV) images often rely on classical multi-view 3D reconstruction techniques. This classical approach involves a sequential process encompassing feature extraction, matching, depth fusion, point cloud integration, and mesh creation. However, these steps, particularly those that feature extraction and matching, are intricate and time-consuming. Furthermore, as the number of steps increases, a corresponding amplification of cumulative error occurs, leading to its continual augmentation. Additionally, these methods typically utilize explicit representation, which can result in issues such as model discontinuity and missing data during the reconstruction process. To effectively address the challenges associated with heightened temporal expenditures, the absence of key elements, and the fragmented models inherent in three-dimensional reconstruction using Unmanned Aerial Vehicle (UAV) imagery, an alternative approach is introduced—the neural radiance field. This novel method leverages neural networks to intricately fit spatial information within the scene, thereby streamlining the reconstruction steps and rectifying model deficiencies. The neural radiance field method employs a fully connected neural network to meticulously model object surfaces and directly generate the 3D object model. This methodology simplifies the intricacies found in conventional 3D reconstruction processes. Implicitly encapsulating scene characteristics, the neural radiance field allows for iterative refinement of neural network parameters via the utilization of volume rendering techniques. Experimental results substantiate the efficacy of this approach, demonstrating its ability to complete scene reconstruction within a mere 5 min timeframe, thereby reducing reconstruction time by 90% while markedly enhancing reconstruction quality.

A depth map fusion algorithm with improved efficiency considering pixel region prediction

A depth map fusion algorithm fuses depth maps from different perspectives into a unified coordinate framework and performs surface calculations to generate dense point clouds of the entire scene. The existing algorithms ensure the quality of these dense point clouds by eliminating inconsistencies between depth maps, but the problem of many redundant calculations often arises. In this paper, a depth map fusion algorithm based on pixel region prediction is proposed. First, the image combination is calculated to select a set of candidate neighbor images for each reference image in the scene. Second, voxels and measure estimates are constructed on a coarse scale, and an inference strategy and a corrector are proposed to merge pixel regions at a fine scale guided by the coarse scale. Finally, the deduced pixel regions at the fine scale are used as the image-space constraints for depth fusion. Public and actual oblique images datasets are used for experimental verification. Compared with the famous COLMAP, OPENMVS, Gipuma and ACMP methods, the number of redundant calculations is significantly reduced; according to Data1 ∼ Data9 in the experiment, as the number of images increases, the fusion efficiency is increased by 47.5% to 156.6%; at the same time, the point cloud accuracy is comparable to other methods.

A Coupled 3d Morphological Reconstruction Approach for Surface Microcrack in Si3 n4 Ceramic Bearing Roller Based on Adaptive Nano Feature Extraction & Multiscale Depth Fusion.

To extract the fuzzy contour features, tiny depth features of surface microcracks in the Si3 N4 ceramic bearings roller. An adaptive nano feature extraction and multiscale deep fusion coupling method is proposed, to sufficiently reconstruct the three-dimensional morphology characteristics of surface microcracks. Construct an adaptive nano feature extraction method, form the surface microcrack image scale space and the Gaussian difference pyramid function equation, realize the detection and matching of global feature points. The sparse point cloud is obtained. Through polar-line correction, depth estimation, and fusion of feature points on the surface microcracks image, a multiscale depth fusion matching cost pixel function is established to realize a dense point cloud reconstruction of surface microcracks. The reconstruction results show that the highest value of the local convex surface reconstructed by the dense point cloud reaches 1183nm, and the lowest local concave surface is accurate to 296nm. Compared with the measurement results of the confocal platform, the relative error of the reconstruction result is 24.6%. The overall feature-matching rate of the reconstruction reaches 93.3%. It provides a theoretical basis for the study of surface microcrack propagation mechanism and the prediction of bearing life.