Feature Representation Network Research Articles

Multimodal imbalanced‐data fault diagnosis method based on a dual‐branch interactive fusion network

AbstractBearing‐fault diagnosis in rotating machinery is essential for ensuring the safety and reliability of mechanical systems. However, under complicated working conditions, the number of normal mechanical equipment samples can far exceed the number of faulty ones. When the data are so imbalanced, data fault diagnosis cannot be easily conducted using conventional deep learning methods. This study proposes a fault diagnosis method based on a dual‐branch interactive fusion network, which improves the accuracy and stability of bearing‐fault diagnosis. First, a dual‐branch feature representation network comprising an iterative attention‐feature fusion residual neural network and a long short‐term memory network is designed for extracting different modal features. Meanwhile, intermodal fusion of the extracted features is performed through multilayer perception. Based on the cost‐sensitive regularization loss, a new joint loss function is then designed for network training. Finally, the effectiveness of the proposed method is verified through comparative experiments, visualization analyses, ablation experiments, and generalization performance experiments.

Analyzing E-Commerce Market Data Using Deep Learning Techniques to Predict Industry Trends

Faced with challenges in sales predicting research, this article combines the capabilities of deep learning algorithms in handling complex tasks and unstructured data. Through analyzing consumer behavior, it selects factors influencing sales, including images, prices and discounts, and historical sales, as input variables for the model. Three different types of neural network models-fully connected neural networks, convolutional neural networks, and recurrent neural networks-are employed to process structured data, image data, and sales sequence data, respectively. This forms a deep neural network for feature representation. Subsequently, based on the outputs of these three types of deep neural networks, a fully connected neural network is employed to train the sales prediction model. Ultimately, experimental results demonstrate that the proposed sales prediction method outperforms exponential regression and shallow neural networks in terms of accuracy.

IE-CFRN: Information exchange-based collaborative feature representation network for multi-modal medical image fusion

Multi-modal medical image fusion (MMIF) is a valuable approach to integrating functional metabolic information and tissue structural details from different modalities, which facilitates clinical diagnosis and surgical navigation efficiently. In this paper, we propose a collaborative feature representation network based on information exchange called IE-CFRN, which distributes the contribution of each encoder feature at the channel level to achieve more accurate fusion. Furthermore, due to the different emphasis on shallow-level and high-level features, we construct a hierarchical feature enhancement network (HFEN) to integrate significant information on multi-level features. Additionally, to adaptively estimate the contribution coefficient in the fidelity loss for guiding the training process, we introduce a learnable weight estimation network (LWEN). Extensive experimentation on eleven state-of-the-art algorithms shows that our IE-CFRN method yields superior consequences with respect to visual effects, and qualitative and quantitative evaluation.

BoAu: Malicious traffic detection with noise labels based on boundary augmentation

The effectiveness of deep-learning-based malicious traffic detection systems relies on high-quality labeled traffic datasets. However, malicious traffic labeling approaches can easily lead to incorrect labeling, which can have a harmful impact on models. To this end, various methods for learning with noise labels have been proposed. They exclude suspected wrong samples from model updates to ensure accuracy. However, this also removes hard samples, resulting in poor model decision boundaries and the loss of ability to classify hard samples. In this paper, we propose a boundary-augmentation-based approach for malicious traffic identification named BoAu. Unlike other approaches, BoAu treats all samples, including hard samples, equally during training to construct more accurate decision boundaries and thus improve accuracy. Meanwhile, a decision boundary augmentation module is designed to mitigate the impact of mislabeled hard samples on decision boundary generation. The decision boundary augmentation module adaptively adjusts the losses of hard samples based on their distance from the cluster to which their labels belong and other clusters, thus driving the shared feature representation network to fit the true label distribution. We validated BoAu in identifying malicious traffic with noise labels on a dataset covering 22 classes of realistic encrypted malicious traffic. Experimental results showed that even under scenarios with up to 90% noise labels, the classification accuracy was still over 80%, which was better than the state-of-the-art approaches. In addition, we validated the applicability of BoAu on several public datasets, including CIC-IDS-2017 and IoT-23.

Automated lung tumor delineation on positron emission tomography/computed tomography via a hybrid regional network.

Multimodality positron emission tomography/computed tomography (PET/CT) imaging combines the anatomical information of CT with the functional information of PET. In the diagnosis and treatment of many cancers, such as non-small cell lung cancer (NSCLC), PET/CT imaging allows more accurate delineation of tumor or involved lymph nodes for radiation planning. In this paper, we propose a hybrid regional network method of automatically segmenting lung tumors from PET/CT images. The hybrid regional network architecture synthesizes the functional and anatomical information from the two image modalities, whereas the mask regional convolutional neural network (R-CNN) and scoring fine-tune the regional location and quality of the output segmentation. This model consists of five major subnetworks, that is, a dual feature representation network (DFRN), a regional proposal network (RPN), a specific tumor-wise R-CNN, a mask-Net, and a score head. Given a PET/CT image as inputs, the DFRN extracts feature maps from the PET and CT images. Then, the RPN and R-CNN work together to localize lung tumors and reduce the image size and feature map size by removing irrelevant regions. The mask-Net is used to segment tumor within a volume-of-interest (VOI) with a score head evaluating the segmentation performed by the mask-Net. Finally, the segmented tumor within the VOI was mapped back to the volumetric coordinate system based on the location information derived via the RPN and R-CNN. We trained, validated, and tested the proposed neural network using 100 PET/CT images of patients with NSCLC. A fivefold cross-validation study was performed. The segmentation was evaluated with two indicators: (1) multiple metrics, including the Dice similarity coefficient, Jacard, 95th percentile Hausdorff distance, mean surface distance (MSD), residual mean square distance, and center-of-mass distance; (2) Bland-Altman analysis and volumetric Pearson correlation analysis. In fivefold cross-validation, this method achieved Dice and MSD of 0.84±0.15 and 1.38±2.2mm, respectively. A new PET/CT can be segmented in 1s by this model. External validation on The Cancer Imaging Archive dataset (63 PET/CT images) indicates that the proposed model has superior performance compared to other methods. The proposed method shows great promise to automatically delineate NSCLC tumors on PET/CT images, thereby allowing for a more streamlined clinical workflow that is faster and reduces physician effort.

SS R-CNN: Self-Supervised Learning Improving Mask R-CNN for Ship Detection in Remote Sensing Images

Due to the cost of acquiring and labeling remote sensing images, only a limited number of images with the target objects are obtained and labeled in some practical applications, which severely limits the generalization capability of typical deep learning networks. Self-supervised learning can learn the inherent feature representations of unlabeled instances and is a promising technique for marine ship detection. In this work, we design a more-way CutPaste self-supervised task to train a feature representation network using clean marine surface images with no ships, based on which a two-stage object detection model using Mask R-CNN is improved to detect marine ships. Experimental results show that with a limited number of labeled remote sensing images, the designed model achieves better detection performance than supervised baseline methods in terms of mAP. Particularly, the detection accuracy for small-sized marine ships is evidently improved.

Classification and Reconstruction of Biomedical Signals Based on Convolutional Neural Network.

The efficient biological signal processing method can effectively improve the efficiency of researchers to explore the work of life mechanism, so as to better reveal the relationship between physiological structure and function, thus promoting the generation of major biological discoveries; high-precision medical signal analysis strategy can, to a certain extent, share the pressure of doctors' clinical diagnosis and assist them to formulate more favorable plans for disease prevention and treatment, so as to alleviate patients' physical and mental pain and improve the overall health level of the society. This article in biomedical signal is very representative of the two types of signals: mammary gland molybdenum target X-ray image (mammography) and the EEG signal as the research object, combined with the deep learning field of CNN; the most representative model is two kinds of biomedical signal classification, and reconstruction methods conducted a series of research: (1) a new classification method of breast masses based on multi-layer CNN is proposed. The method includes a CNN feature representation network for breast masses and a feature decision mechanism that simulates the physician's diagnosis process. By comparing with the objective classification accuracy of other methods for the identification of benign and malignant breast masses, the method achieved the highest classification accuracy of 97.0% under different values of c and gamma, which further verified the effectiveness of the proposed method in the identification of breast masses based on molybdenum target X-ray images. (2) An EEG signal classification method based on spatiotemporal fusion CNN is proposed. This method includes a multi-channel input classification network focusing on spatial information of EEG signals, a single-channel input classification network focusing on temporal information of EEG signals, and a spatial-temporal fusion strategy. Through comparative experiments on EEG signal classification tasks, the effectiveness of the proposed method was verified from the aspects of objective classification accuracy, number of model parameters, and subjective evaluation of CNN feature representation validity. It can be seen that the method proposed in this paper not only has high accuracy, but also can be well applied to the classification and reconstruction of biomedical signals.

Sampling Agnostic Feature Representation for Long-Term Person Re-Identification.

Person re-identification is a problem of identifying individuals across non-overlapping cameras. Although remarkable progress has been made in the re-identification problem, it is still a challenging problem due to appearance variations of the same person as well as other people of similar appearance. Some prior works solved the issues by separating features of positive samples from features of negative ones. However, the performances of existing models considerably depend on the characteristics and statistics of the samples used for training. Thus, we propose a novel framework named sampling independent robust feature representation network (SirNet) that learns disentangled feature embedding from randomly chosen samples. A carefully designed sampling independent maximum discrepancy loss is introduced to model samples of the same person as a cluster. As a result, the proposed framework can generate additional hard negatives/positives using the learned features, which results in better discriminability from other identities. Extensive experimental results on large-scale benchmark datasets verify that the proposed model is more effective than prior state-of-the-art models.

Self-Supervised Feature Representation and Few-Shot Land Cover Classification of Multimodal Remote Sensing Images

Although deep learning-based approaches have made significant progress in remote sensing image classification, the supervised learning paradigm has shortcomings under a limited number of labeled samples, which restricts the classification performance to a great extent. In this article, we investigate an effective self-supervised feature representation architecture (SSFR) for multimodal remote sensing images few-shot land cover classification. Specifically, we exploit multiview learning strategy to construct multiple views from multimodal remote sensing images. This method builds several complementary views of the same observed scenes from hyperspectral images or different modalities of remote sensing data. Then we build the deep feature extractor to learn high-level feature representations from each view via contrastive learning. The contrastive learning aggregates the samples of the same scene while separating samples of different scenes in the latent space, and this process does not require any labeled information. What’s more, to learn more robust features from different views, we utilize multitask learning strategy to train the feature extraction network. Finally, a lightweight machine learning method is employed to classify the learned features using a few annotated samples. To further demonstrate the self-supervised feature learning capability of the proposed model, we train the feature representation network in multiple source datasets. Comprehensive feature learning and classification experiments have certified the effectiveness and superiority of the proposed method.

Deep Feature Representation Based Imitation Learning for Autonomous Helicopter Aerobatics

The drastic changes in flight parameters during aerobatics and the high instability of the system make the control of autonomous aerobatics unusually difficult. In this paper, we propose a deep feature representation based imitation learning method for autonomous aerobatics, which leverages expert demonstrations to efficiently learn the mapping of high-dimensional flight observations onto continuous actions (pitch, tail, and thrust). Different from the existing methods, our proposed method requires neither trajectory specification and alignment nor any assumptions and processing of system uncertainties, so it can greatly simplify the controller solution steps and reduce the computational burden. Particularly, our method uses the proposed deep feature representation network (DFR-network) to directly map the experts’ demonstration trajectory to a deep representation space spanned by a set of learned subspaces which represent the motion patterns with the same statistical property among demonstration trajectories. Various aerobatic maneuvers can be encoded in the deep representation space through a simple combination of embedding features. Accordingly, the proposed method can perform arbitrary aerobatic maneuvers by observing a limit set of expert demonstrations. The effectiveness of the deep feature representation based imitation learning method is verified on the real-world flight data. Experiments show that compared with the existing methods, our proposed method has higher control accuracy, stronger robustness and anti-interference ability.

Consistent-Separable Feature Representation for Semantic Segmentation

Cross-entropy loss combined with softmax is one of the most commonly used supervision components in most existing segmentation methods. The softmax loss is typically good at optimizing the inter-class difference, but not good at reducing the intra-class variation, which can be suboptimal for semantic segmentation task. In this paper, we propose a Consistent-Separable Feature Representation Network to model the Consistent-Separable (C-S) features, which are intra-class consistent and inter-class separable, improving the discriminative power of the deep features. Specifically, we develop a Consistent-Separable Feature Learning Module to obtain C-S features through a new loss, called Class-Aware Consistency loss. This loss function is proposed to force the deep features to be consistent among the same class and apart between different classes. Moreover, we design an Adaptive feature Aggregation Module to fuse the C-S features and original features from backbone for the better semantic prediction. We show that compared with various baselines, the proposed method brings consistent performance improvement. Our proposed approach achieves state-of-the-art performance on Cityscapes (82.6% mIoU in test set), ADE20K (46.65% mIoU in validation set), COCO Stuff (41.3% mIoU in validation set) and PASCAL Context (55.9% mIoU in test set).

FT-MDnet: A Deep-Frozen Transfer Learning Framework for Person Search

Matching manually cropped pedestrian images between queries and candidates, termed as person re-identification, has achieved significant progress with deep convolutional neural networks. Recently, a topic called ‘person search’ is proposed for the end-to-end application of re-identification technologies. It integrates object detection and person re-identification and aims to both locate and match pedestrians on a gallery of raw images. However, the design and implementation of such kind of hybrid network are difficult and computationally consuming in real practical situations. In order to fasten the design and ease the implementation, this paper proposes a deep-frozen transfer learning framework, named FT-MDnet, to extract re-identification features from a pre-trained detection network in two steps. First, using a channel-wise attention mechanism, a network called adaptive transfer learning network (ATLnet) is used to convert the sharing data of the underlying detection network to a re-identification feature map. Then, a multi-branch feature representation network called multiple descriptor network (MDnet) is proposed to extract re-identification features from the re-identification feature map. Our proposed solution has been verified on different types of mainstream detection networks, including YOLOv3, YOLOv4, Mask RCNN, and CenterNet. The experimental results show that our solution outperforms all other person search solutions by a large margin. It proves that the feature representations of detection networks are highly compatible with re-identification, and the proposed framework effectively extracts these features out. To encourage further research, we have made our framework open source.

Joint Pyramid Feature Representation Network for Vehicle Re-identification

Vehicle re-identification (Re-ID) technology plays an important role in the intelligent transportation system for smart city. Due to various uncertain factors in the real-world scenarios, (e.g., resolution variation, viewpoint variation, illumination changes, occlusion, etc., vehicle Re-ID is a very challenging task. To resist the adverse effect of resolution variation, a joint pyramid feature representation network (JPFRN) for vehicle Re-ID is proposed in this paper. Based on the consideration that various convolution blocks with different depths hold different resolutions and semantic information of the vehicle image, the proposed JPFRN method employs a base network to obtain multi-resolution vehicle features in the first stage. Then, a pyramid feature representation scheme is developed to reconstruct and integrate the obtained multi-resolution vehicle features together. Finally, these pyramid features are jointly represented for learning a more discriminative feature under the supervision of joint Triplet loss and softmax loss. Extensive experimental results on two commonly-used vehicle databases (i.e., VehicleID and VeRi) show that the proposed JPFRN is superior to multiple recently-developed vehicle Re-ID methods.

End-to-end DeepNCC framework for robust visual tracking

In this paper, we propose an NCC-based object tracking deep framework, which can be well initialized with the limited target samples in the first frame. The proposed framework contains a pretrained model, online feature fine-tuning layers and tracking processes. The pretrained model provides rich feature representations while online feature fine-tuning layers select discriminative and generic features for the tracked object. We choose normalized cross-correlation as a template tracking layer to perform the tracking process. To enable the learned features representation closely coordinated to the tracked target, we jointly train the feature representation network and tracking processes. In online tracking, an adaptive template and a fixed template are fused to find the optimal tracking results. Scale estimation and a high-confidence model update scheme are perfectly integrated into the framework to adapt to the target appearance changes. The extensive experiments demonstrate that the proposed tracker achieves superior performance compared with other state-of-the-art trackers.

A New Benchmark and an Attribute-Guided Multilevel Feature Representation Network for Fine-Grained Ship Classification in Optical Remote Sensing Images

Maritime activities are essential aspects of human society. Accurate classification of ships is vital for maritime surveillance and meaningful to numerous civil and military applications. However, most studies conducted are limited to the coarse-grained ship classification. Few studies on fine-grained ship classification have been undertaken despite its accuracy and practicability. In this study, we construct a new benchmark for fine-grained ship classification which consists of 23 fine-grained categories of ships. Besides the category label, the benchmark contains several other attribute information. To solve the problem of interclass similarity, an attribute-guided multilevel enhanced feature representation network (AMEFRN) is proposed. Concretely, a multilevel enhanced visual feature representation is designed to fuse the reweighted regional features in order to focus more on the silent region and suppress the other regions. Further to this, considering the complementary role of attribute information in ship identification, an attribute-guided feature extraction branch is proposed, which extracts the auxiliary attribute features by utilizing the attribute information as supervision. Finally, the attribute features and the enhanced visual features jointly function as a feature representation for classification. Compared to other existing classification models, AMEFRN has better performance with an overall accuracy rate of 93.58% on the established fine-grained ship classification dataset. Moreover, it can be easily embedded into most CNN models as well as trained end-to-end.

Deeply-Learned Spatial Alignment for Person Re-Identification

A large class of Person Re-identification (ReID) approaches identify pedestrians with the TriHard loss. Though the TriHard loss is a robust ReID method, pose variance and viewpoint in pedestrians constrain the performance. To address this problem, we introduce a spatial transformer network (STN) to align pedestrians. Then, we illustrate the generality of the STN module in pose variance problem through the evaluations on feature representation network (FRN) like VGG, ResNet and DenseNet architectures respectively. Furthermore, based on the evaluation results, we propose a robust and high-performance ReID model which consists of the STN module, DenseNet backbone and TriHard loss. And finally, we prove that our ReID model is whole differentiable by formula derivation, therefore achieving an end-to-end high-performance ReID system. The experiments show that our ReID system outperforms the state-of-art methods on Market-1501, DukeMTMC-reID and CUHK03 datasets.