Depth Feature Extraction Research Articles

Learning Discriminative Cross-Modality Features for RGB-D Saliency Detection.

How to explore useful information from depth is the key success of the RGB-D saliency detection methods. While the RGB and depth images are from different domains, a modality gap will lead to unsatisfactory results for simple feature concatenation. Towards better performance, most methods focus on bridging this gap and designing different cross-modal fusion modules for features, while ignoring explicitly extracting some useful consistent information from them. To overcome this problem, we develop a simple yet effective RGB-D saliency detection method by learning discriminative cross-modality features based on the deep neural network. The proposed method first learns modality-specific features for RGB and depth inputs. And then we separately calculate the correlations of every pixel-pair in a cross-modality consistent way, i.e., the distribution ranges are consistent for the correlations calculated based on features extracted from RGB (RGB correlation) or depth inputs (depth correlation). From different perspectives, color or spatial, the RGB and depth correlations end up at the same point to depict how tightly each pixel-pair is related. Secondly, to complemently gather RGB and depth information, we propose a novel correlation-fusion to fuse RGB and depth correlations, resulting in a cross-modality correlation. Finally, the features are refined with both long-range cross-modality correlations and local depth correlations to predict salient maps. In which, the long-range cross-modality correlation provides context information for accurate localization, and the local depth correlation keeps good subtle structures for fine segmentation. In addition, a lightweight DepthNet is designed for efficient depth feature extraction. We solve the proposed network in an end-to-end manner. Both quantitative and qualitative experimental results demonstrate the proposed algorithm achieves favorable performance against state-of-the-art methods.

Research on intrusion detection method based on SMOTE and DBN-LSSVM

Aiming at the problems of low accuracy and high false alarm rate when traditional machine learning algorithm processes massive and complex intrusion detection data, this paper proposes a network intrusion detection method (SMOTE-DBN-LSSVM) which combines deep belief network (DBN), synthetic minority oversampling technique (SMOTE) and least squares support vector machine (LSSVM). In this algorithm, intrusion detection data is input to the DBN for depth feature extraction, and then a small number of samples are added through SMOTE algorithm. Finally, LSSVM is used for classification. Through the effective evaluation of SMOTE-DBN-LSSVM model by NSL-KDD dataset, the experimental results show that SMOTE-DBN-LSSVM algorithm has the advantages of high accuracy and low false alarm rate compared with other algorithms, and improves the detection rate of small sample attacks.

A novel ResNet101 model based on dense dilated convolution for image classification

Image classification plays an important role in computer vision. The existing convolutional neural network methods have some problems during image classification process, such as low accuracy of tumor classification and poor ability of feature expression and feature extraction. Therefore, we propose a novel ResNet101 model based on dense dilated convolution for medical liver tumors classification. The multi-scale feature extraction module is used to extract multi-scale features of images, and the receptive field of the network is increased. The depth feature extraction module is used to reduce background noise information and focus on effective features of the focal region. To obtain broader and deeper semantic information, a dense dilated convolution module is deployed in the network. This module combines the advantages of Inception, residual structure, and multi-scale dilated convolution to obtain a deeper level of feature information without causing gradient explosion and gradient disappearance. To solve the common feature loss problems in the classification network, the up- down-sampling module in the network is improved, and multiple convolution kernels with different scales are cascaded to widen the network, which can effectively avoid feature loss. Finally, experiments are carried out on the proposed method. Compared with the existing mainstream classification networks, the proposed method can improve the classification performance, and finally achieve accurate classification of liver tumors. The effectiveness of the proposed method is further verified by ablation experiments.HighlightsThe multi-scale feature extraction module is introduced to extract multi-scale features of images, it can extract deep context information of the lesion region and surrounding tissues to enhance the feature extraction ability of the network.The depth feature extraction module is used to focus on the local features of the lesion region from both channel and space, weaken the influence of irrelevant information, and strengthen the recognition ability of the lesion region.The feature extraction module is enhanced by the parallel structure of dense dilated convolution, and the deeper feature information is obtained without losing the image feature information to improve the classification accuracy.

Detection of heavy metal lead in lettuce leaves based on fluorescence hyperspectral technology combined with deep learning algorithm

The feasibility analysis of fluorescence hyperspectral imaging technology was studied for the detection of lead content in lettuce leaves. Further, Monte Carlo optimized wavelet transform stacked auto-encoders (WT-MC-SAE) was proposed for dimensionality reduction and depth feature extraction of fluorescence spectral data. The fluorescence hyperspectral images of 2800 lettuce leaf samples were selected and the whole lettuce leaf was used as the region of interest (ROI) to extract the fluorescence spectrum. Five different pre-processing algorithms were used to pre-process the original ROI spectral data including standard normalized variable (SNV), first derivative (1st Der), second derivative (2ndDer), third derivative (3rd Der) and fourth derivative (4th Der). Moreover, wavelet transform stacked auto-encoders (WT-SAE) and WT-MC-SAE were used for data dimensionality reduction, and support vector machine regression (SVR) was used for modeling analysis. Among them, 4th Der tends to be the most useful fluorescence spectral data for Pb content detection at 0.067 ∼ 1.400 mg/kg in lettuce leaves, with Rc2 of 0.9802, RMSEC of 0.02321 mg/kg, Rp2 of 0.9467, RMSEP of 0.04017 mg/kg and RPD of 3.273, and model scale (the number of nodes in the input layer, hidden layer and output layer) was 407-314-286-121-76 under the fifth level of wavelet decomposition. Further studies showed that WT-MC-SAE realizes the depth feature extraction of the fluorescence spectrum, and it is of great significance to use fluorescence hyperspectral imaging to realize the quantitative detection of lead in lettuce leaves.

Depth feature extraction of surface-breaking defects using laser pulsed thermography

In this paper, the feasibility of the depth feature extraction of surface-breaking defects based on laser pulsed thermography in transmission mode is proposed. This method is adopted to detect the depth of surface-breaking defects. First, a finite element model based on COMSOL is established to simulate structural steel specimens with different defect depths and the contrast method is used to detect these depths. In order to verify its feasibility, the model is simulated at different time nodes. A simulation analysis shows the practicability of this method. Then, through an artificial slot depth feature extraction experiment on structural steel specimens, an algorithm is used to denoise an infrared image and then the contrast method is used to extract the defect depth feature, which verifies the feasibility of the method. The experimental results show that as the depth of artificial slots increases, the temperature at the observation point also reduces. The best fitting equation of the defect depth and the temperature at the observation point at a certain heating time have an exponential relationship. This method can accurately detect defect depths.

Sea Ice Image Classification Based on Heterogeneous Data Fusion and Deep Learning

Sea ice is one of the typical causes of marine disasters. Sea ice image classification is an important component of sea ice detection. Optical data contain rich spectral information, but they do not allow one to easily distinguish between ground objects with a similar spectrum and foreign objects with the same spectrum. Synthetic aperture radar (SAR) data contain rich texture information, but the data usually have a single source. The limitation of single-source data is that they do not allow for further improvements of the accuracy of remote sensing sea ice classification. In this paper, we propose a method for sea ice image classification based on deep learning and heterogeneous data fusion. Utilizing the advantages of convolutional neural networks (CNNs) in terms of depth feature extraction, we designed a deep learning network structure for SAR and optical images and achieve sea ice image classification through feature extraction and a feature-level fusion of heterogeneous data. For the SAR images, the improved spatial pyramid pooling (SPP) network was used and texture information on sea ice at different scales was extracted by depth. For the optical data, multi-level feature information on sea ice such as spatial and spectral information on different types of sea ice was extracted through a path aggregation network (PANet), which enabled low-level features to be fully utilized due to the gradual feature extraction of the convolution neural network. In order to verify the effectiveness of the method, two sets of heterogeneous sentinel satellite data were used for sea ice classification in the Hudson Bay area. The experimental results show that compared with the typical image classification methods and other heterogeneous data fusion methods, the method proposed in this paper fully integrates multi-scale and multi-level texture and spectral information from heterogeneous data and achieves a better classification effect (96.61%, 95.69%).

ON-BOARD GCPS MATCHING WITH IMPROVED TRIPLET LOSS FUNCTION

Abstract. Intelligent remote sensing satellite system is an important direction to solve the problem of intelligent processing on-board. It can realize the real-time on-board intelligent processing of important targets. The accuracy of geometric positioning information is the basis for subsequent intelligent processing. Therefore, this paper corrects the positioning information by GCPs (Ground Control Points) matching on-board. Considering the limited storage and computing performance of satellites, this paper designs a lightweight GCPs deep feature extraction convolutional neural network based on MobileNetV2 as feature extraction model, and trains this network with an improved triplet loss function. The Songshan calibration field images constructed by Wuhan University was used as the GCPs image, and 30,399 image patches were extracted and embedded as GCPs feature library. The size of the GCPs library is a size of 15.3M, and size of the lightweight depth feature extraction model is 9.83M, which can be pre-stored on the satellite for positioning with GCPs matching on-board. In addition, this paper tested feature extraction performance on an embedded device Nvidia Jeston Xavier which simulates the performance of the device on the satellite. At Xavier 30W max power consumption model, a single frame takes 0.005 seconds, and under Xavier 15W power consumption model, a single frame takes 0.009 seconds. At 10W power consumption model, a single frame takes 0.018 seconds, which can meet the performance requirements on the satellite. In addition, the experiments in this paper show that the positioning accuracy is within 30 meters. The work done in this paper will be experimented on the Luojia-3-01 intelligent remote sensing satellite.

Cross-modal feature extraction and integration based RGBD saliency detection

In RGBD saliency detection research field, RGB and depth cues are generally given the same status by RGBD saliency models. However, they ignore that both modalities are significantly different in inherent attribution so that effective features cannot be drawn from depth maps. In order to address this issue, this paper proposes a novel RGBD saliency model including two key components: the contrast-guided depth feature extraction (CDFE) module and the cross-modal feature integration (CFI) module. Specifically, considering the specific properties of depth information, we first design a targeted CDFE module, which learns multi-level deep depth features by strengthening the depth contrast between foreground and background, to provide multi-level deep depth features. Then, to sufficiently and reasonably integrate multi-level cross-modal features, namely the multi-level deep RGB and depth features, we equip the saliency inference branch with the CFI module, which contains two successive steps, i.e. information enrichment and feature enhancement. Extensive experiments are conducted on five challenging RGBD datasets, and the experimental results clearly demonstrate the effectiveness and superiority of the proposed model against the state-of-the-art RGBD saliency models.

A Deep Multiscale Fusion Method via Low-Rank Sparse Decomposition for Object Saliency Detection Based on Urban Data in Optical Remote Sensing Images

The urban data provides a wealth of information that can support the life and work for people. In this work, we research the object saliency detection in optical remote sensing images, which is conducive to the interpretation of urban scenes. Saliency detection selects the regions with important information in the remote sensing images, which severely imitates the human visual system. It plays a powerful role in other image processing. It has successfully made great achievements in change detection, object tracking, temperature reversal, and other tasks. The traditional method has some disadvantages such as poor robustness and high computational complexity. Therefore, this paper proposes a deep multiscale fusion method via low-rank sparse decomposition for object saliency detection in optical remote sensing images. First, we execute multiscale segmentation for remote sensing images. Then, we calculate the saliency value, and the proposal region is generated. The superpixel blocks of the remaining proposal regions of the segmentation map are input into the convolutional neural network. By extracting the depth feature, the saliency value is calculated and the proposal regions are updated. The feature transformation matrix is obtained based on the gradient descent method, and the high-level semantic prior knowledge is obtained by using the convolutional neural network. The process is iterated continuously to obtain the saliency map at each scale. The low-rank sparse decomposition of the transformed matrix is carried out by robust principal component analysis. Finally, the weight cellular automata method is utilized to fuse the multiscale saliency graphs and the saliency map calculated according to the sparse noise obtained by decomposition. Meanwhile, the object priors knowledge can filter most of the background information, reduce unnecessary depth feature extraction, and meaningfully improve the saliency detection rate. The experiment results show that the proposed method can effectively improve the detection effect compared to other deep learning methods.

A Light Field Image Quality Assessment Model Based on Symmetry and Depth Features

This paper presents a new full-reference image quality assessment (IQA) method for conducting the perceptual quality evaluation of the light field (LF) images, called the symmetry and depth feature-based model (SDFM). Specifically, the radial symmetry transform is first employed on the luminance components of the reference and distorted LF images to extract their symmetry features for capturing the spatial quality of each view of an LF image. Second, the depth feature extraction scheme is designed to explore the geometry information inherited in an LF image for modeling its LF structural consistency across views. The similarity measurements are subsequently conducted on the comparison of their symmetry and depth features separately, which are further combined to achieve the quality score for the distorted LF image. Note that the proposed SDFM that explores the symmetry and depth features is conformable to the human vision system, which identifies the objects by sensing their structures and geometries. Extensive simulation results on the dense light fields dataset have clearly shown that the proposed SDFM outperforms multiple classical and recently developed IQA algorithms on quality evaluation of the LF images.

Hilbert stereo reconstruction algorithm based on depth feature and stereo matching

This paper proposes a Hilbert stereo reconstruction algorithm based on depth feature and stereo matching to solve the problem of occlusive region matching errors, namely, the Hilbert stereo network. The traditional stereo network pays more attention to disparity itself, leading to the inaccuracy of disparity estimation. Our design network studies the effective disparity matching and refinement through reconstruction representation of Hilbert’s disparity coefficient. Since the Hilbert coefficient is not affected by the occlusion and texture in the image, stereo disparity matching can conducted effectively. Our network includes three sub-modules, namely, depth feature representation, Hilbert cost volume fusion, and Hilbert refinement reconstruction. Separately, texture features of different depth levels of the image were extracted through Hilbert filtering operation. Next, stereoscopic disparity fusion was performed, and then Hilbert designed to refine the difference regression stereo matching solution was used. Based on the end-to-end design, the structure is refined by combining the depth feature extraction module and Hilbert coefficient disparity. Finally, the Hilbert stereo matching algorithm achieves excellent performance on standard big data set and is compared with other advanced stereo networks. Experiments show that our network has high accuracy and high performance.

Integrating Complementary Appearance, Posture and Motion Cues for RGB-D Action Recognition

This work presents a novel approach to multimodal human action recognition by jointly using visual RGB and depth (including skeleton joint positions) data captured from depth camera. For the depth feature extraction, Local Surface Geometric Feature (LSGF) is adopted to capture the geometric appearance and postures cues. Simultaneously, the improved dense trajectory feature (IDT) is extracted from RGB modality to jointly characterize the motion, visual appearance and trajectory shape information. These features from different modalities are complementary to each other. Then a two-stage integration scheme is proposed, which incorporates the probability weights of each classifier for action recognition. The proposed approach is evaluated on four publicly available human action databases: NJUST RGB-D Action, MSR-ActionPairs, MSR-DailyAct3D, and UTD-MHAD. Experimental results demonstrate that the proposed approach outperforms or is comparable to the state-of-the-art methods.

Sparse auto-encoder based feature learning for human body detection in depth image

Human body detection in depth image is an active research topic in computer vision. But depth feature extraction is still an open problem. In this paper, a novel feature learning method based on sparse auto-encoder (SAE) is proposed for human body detection in depth image. The proposed learning based feature enables capturing the intrinsic human body structure. To further reduce the computation cost of SAE, both convolution neural network and pooling are introduced to reduce the training complexity. In addition, upon learning SAE based depth feature, we further pursuit the detector efficiency. A beyond sliding window localization strategy is proposed based on the fact that the depth values of object surface are almost the same. The proposed localization strategy first uses the histogram of depth to generate candidate detection window center, and then exploits the relationship between human body height and depth to determine the detection window size. Thus, it can avoid the time-consuming sliding window search, and further enables fast human body localization. Experiments on SZU Depth Pedestrian dataset verify the effectiveness of our proposed method.