Discriminative Feature Representation Research Articles

Successively focused details: attention-based recurrent multiscale network for sketch-based image retrieval

Sketch-based image retrieval (SBIR) is a cross-domain retrieval problem whose main challenge is that sketches and images differ in many aspects, such as visual content and abstract degree. Existing approaches solve cross-domain gaps using deep neural networks (DNNs) to learn a joint global feature representation for sketches and images. To improve the retrieval effect using local features, some researchers have introduced attention mechanisms and multi-scale inputs. However, these cannot effectively locate regions of detail difference between sketch–image pairs, and the discriminative ability of these models is insufficient. Just as human observational habits range from global to local, this paper constructs a multiscale network with the help of attention modules. The network is stacked in a recurrent way and focuses stepwise on the most discriminative regions from coarse to fine through region-specific attention models. The part-aware attention module is specifically designed for each scale network to make it robust against local spatial misalignment. To address the challenges of SBIR, we extend the network into a three-channel structure that integrates the sketch, edge map, and RGB photo branches to acquire a more discriminative cross-domain feature representation. The proposed network is alternately optimized by weighted regularization triplet loss, classification loss, and pairwise ranking loss. We conduct extensive experiments on several commonly used SBIR datasets, quantitative, and qualitative results demonstrate that our network learns more fine-grained features and outperforms all baselines.

Spectral–Spatial Feature Extraction With Dual Graph Autoencoder for Hyperspectral Image Clustering

Autoencoder (AE) is an unsupervised neural network framework for efficient and effective feature extraction. Most AE-based methods do not consider spatial information and band correlations for hyperspectral image (HSI) analysis. In addition, graph-based AE methods often learn discriminative representations with the assumption that connected samples share the same label and they cannot directly embed the geometric structure into feature extraction. To address these issues, in this paper, we propose a dual graph autoencoder (DGAE) to learn discriminative representations for HSIs. Utilizing the relationships of pair-wise pixels within homogenous regions and pair-wise spectral bands, DGAE first constructs the superpixel-based similarity graph with spatial information and band-based similarity graph to characterize the geometric structures of HSIs. With the developed dual graph convolution, more discriminative feature representations are learnt from the hidden layer via the encoder-decoder structure of DGAE. The main advantage of DGAE is that it fully exploits both the geometric structures of pixels with spatial information and spectral bands to promote nonlinear feature extraction of HSIs. Experiments on HSI datasets show the superiority of the proposed DGAE over the state-of-the-art methods. The source code of DGAE is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/ZhangYongshan/DGAE</uri> .

Cross-Domain Recognition via Projective Cross-Reconstruction

This article proposes a novel data reconstruction method, called projective cross-reconstruction (PCR) for cross-domain recognition. The intrinsic philosophy behind PCR is that the data from different domains but with the same label have a strong correlation and thus they can be reconstructed with each other. To this end, we first rearrange the data of source and target domains to form two new cross-data matrices, and the data with the same label but from different domains can be arranged together. Then, we use two different projection matrices to project the new cross-data into two approximate subspaces and perform the cross-reconstruction without introducing any extra matrix as the reconstruction coefficient matrix. This guarantees that the data from different domains can be interlaced well and the data from different domains but sharing the same label can be aligned together. In doing so, the problem of cross-domain distribution mismatch is solved and a discriminative and transferrable feature representation can be obtained. Moreover, PCR integrates the classifier learning and feature representation learning into a unified framework so that these two tasks can be iteratively improved until the termination condition is met. Extensive experiments on six datasets validate the effectiveness of our proposed PCR, compared with the state-of-the-art methods.

An adaptive frame selection network with enhanced dilated convolution for video smoke recognition

Recognizing smoke from surveillance videos is crucial to achieve robust fire detection due to the rich temporal cues in video data. However, the slow-moving smoke makes it difficult to learn complete motion information of smoke. In addition, the widely-used max/average pooling operations in smoke recognition models may lead to the loss of spatial detail information. To solve the above issues, we propose an adaptive frame selection network (AFSNet) with enhanced dilated convolution for video smoke recognition. Specifically, first, to reduce information redundancy for learning discriminative feature representations, we propose an adaptive frame selection convolution to automatically select the most useful frames in the image sequence. Second, to utilize all the pixels for reducing the loss of detail information and carrying over global information, we propose an enhanced dilated convolution by considering the pixels with high responses as well as the local region to learn the contribution of each pixel to the output automatically. Finally, a feature extraction module without any average and max pooling operations is designed to learn multi-scale, context, and spatiotemporal information simultaneously. Experiments show that our model achieves the highest detection rate of 0.9673 and the lowest false alarm rate of 0.0316 on the SRSet dataset, and achieves the highest F1-score accuracy of 0.85, 0.86, and 0.91 on the three subsets of the RISE dataset, respectively.

Learning Discriminative Feature Representation for Estimating Smoke Density of Smoky Vehicle Rear

Estimating the smoke density from a single image of the smoky vehicle rear is significant for smoke level estimation and smoky vehicle recognition. However, this is a highly ill-posed problem. To solve it, this paper presents a novel smoke density estimation network (SDENet). First, to reduce the susceptibility of deeper convolutions towards the smoke scale variant and enhance feature diversity, we propose the attention multi-scale encoding blocks based on multi-scale blocks and the convolutional block attention. The multi-scale block exploits the multi-scale features at a granular level within a single basic block. The attention makes our model pay more attention to specific regions which are beneficial to smoke density estimation. Second, the extracted features for smoke recognition may contain background interference information due to the smoke translucency, so we propose semantic-guided feature selection blocks to progressively select smoke-relevant features and suppress background interference features from the encoded feature via global high-level semantic information for learning more discriminative features. Finally, to make the learned features more adaptive to smoke feature resolution and visual appearance, we design attention gate decoding blocks to fuse different features via gate blocks, which enhance the features at spatial locations and channel locations where the features are essential for smoke density estimation. Extensive experiments on smoke density estimation show that our model achieves the best performance among existing methods.

MULTforAD: Multimodal MRI Neuroimaging for Alzheimer’s Disease Detection Based on a 3D Convolution Model

Alzheimer’s disease (AD) is a neurological disease that affects numerous people. The condition causes brain atrophy, which leads to memory loss, cognitive impairment, and death. In its early stages, Alzheimer’s disease is tricky to predict. Therefore, treatment provided at an early stage of AD is more effective and causes less damage than treatment at a later stage. Although AD is a common brain condition, it is difficult to recognize, and its classification requires a discriminative feature representation to separate similar brain patterns. Multimodal neuroimage information that combines multiple medical images can classify and diagnose AD more accurately and comprehensively. Magnetic resonance imaging (MRI) has been used for decades to assist physicians in diagnosing Alzheimer’s disease. Deep models have detected AD with high accuracy in computing-assisted imaging and diagnosis by minimizing the need for hand-crafted feature extraction from MRI images. This study proposes a multimodal image fusion method to fuse MRI neuroimages with a modular set of image preprocessing procedures to automatically fuse and convert Alzheimer’s disease neuroimaging initiative (ADNI) into the BIDS standard for classifying different MRI data of Alzheimer’s subjects from normal controls. Furthermore, a 3D convolutional neural network is used to learn generic features by capturing AlD biomarkers in the fused images, resulting in richer multimodal feature information. Finally, a conventional CNN with three classifiers, including Softmax, SVM, and RF, forecasts and classifies the extracted Alzheimer’s brain multimodal traits from a normal healthy brain. The findings reveal that the proposed method can efficiently predict AD progression by combining high-dimensional MRI characteristics from different public sources with an accuracy range from 88.7% to 99% and outperforming baseline models when applied to MRI-derived voxel features.

Four‐branch Siamese network based on sketch‐specific data augmentation for sketch recognition

The shortened abstract is as follows: Sketch recognition has become an important hotspot issue because of sketch's intuitiveness and visualization. The existing sketch-recognition methods based on handcrafted features and deep features are insufficient in the recognition of the local information of sketches, and the recognition accuracy is not ideal. Accordingly, this paper proposes a four-branch Siamese network based on sketch-specific data augmentation to generate discriminative feature representations and improve the sketch-recognition accuracy. A sketch is an ordered list of strokes, we adopt the semantic information of strokes as the decomposition criteria to divide a sketch into three disjoint local blocks, and then combine the local blocks in pairs to form three new sketches. In order to give full play to the positive effect of local blocks on category prediction and enhance the fine-grained capability of the network, three newly generated sketches and the original sketch are combined to construct a four-branch Siamese network. Each branch network adopts the Sketch-A-Net architecture with the fully connected layer removed as the basic network, and we improve it by adding shortcut connection layer and multi-scale weighted bilinear coding (MWBC) modules. Compared with the state-of-the-art methods, the experimental results on the TU-Berlin dataset demonstrate the excellent performance of our model.

Image local structure information learning for fine-grained visual classification

Learning discriminative visual patterns from image local salient regions is widely used for fine-grained visual classification (FGVC) tasks such as plant or animal species classification. A large number of complex networks have been designed for learning discriminative feature representations. In this paper, we propose a novel local structure information (LSI) learning method for FGVC. Firstly, we indicate that the existing FGVC methods have not properly considered how to extract LSI from an input image for FGVC. Then an LSI extraction technique is introduced which has the ability to properly depict the properties of different local structure features in images. Secondly, a novel LSI learning module is proposed to be added into a given backbone network for enhancing the ability of the network to find salient regions. Thirdly, extensive experiments show that our proposed method achieves better performance on six image datasets. Particularly, the proposed method performs far better on datasets with a limited number of images.

Self-supervised transformer-based pre-training method using latent semantic masking auto-encoder for pest and disease classification

Pest and disease classification is a challenging issue in agriculture. Currently, the classification algorithms of pests and diseases based on CNN models have become popular. However, these methods have a limited performance improvement due to a lack of global information interaction and discriminative feature representation. Therefore, we propose a self-supervised transformer-based pre-training method using latent semantic masking auto-encoder (LSMAE). In this method, a feature relationship conditional filtering (FRCF) based on k-NN graph is proposed for filtering irrelevant data from the source domain and generating a subset of source domain. The data in this subset are similar to that of target domain which can supplement feature learning of the target domain. To further improve the performance, a novel auto-encoder based on latent semantic masking is proposed for transformer model pre-training. This auto-encoder can select key patches of each image in the subset of the source domain and let the transformer model learn a more discriminative feature representation. Finally, the target domain data are utilized to fine-tune the pre-trained transformer model. Experiments conducted on public datasets, such as IP102, CPB, and Plant Village, show that our method outperforms the state-of-the-art methods. For example, our method achieves 74.69%/76.99%/99.93% accuracy on IP102/CPB/Plant Village, demonstrating that the proposed self-supervised transformer-based pre-training method is more effective in the pest and disease classification field than CNN-based methods.

A Spectral-Spatial-Dependent Global Learning Framework for Insufficient and Imbalanced Hyperspectral Image Classification.

Deep learning techniques have been widely applied to hyperspectral image (HSI) classification and have achieved great success. However, the deep neural network model has a large parameter space and requires a large number of labeled data. Deep learning methods for HSI classification usually follow a patchwise learning framework. Recently, a fast patch-free global learning (FPGA) architecture was proposed for HSI classification according to global spatial context information. However, FPGA has difficulty in extracting the most discriminative features when the sample data are imbalanced. In this article, a spectral-spatial-dependent global learning (SSDGL) framework based on the global convolutional long short-term memory (GCL) and global joint attention mechanism (GJAM) is proposed for insufficient and imbalanced HSI classification. In SSDGL, the hierarchically balanced (H-B) sampling strategy and the weighted softmax loss are proposed to address the imbalanced sample problem. To effectively distinguish similar spectral characteristics of land cover types, the GCL module is introduced to extract the long short-term dependency of spectral features. To learn the most discriminative feature representations, the GJAM module is proposed to extract attention areas. The experimental results obtained with three public HSI datasets show that the SSDGL has powerful performance in insufficient and imbalanced sample problems and is superior to other state-of-the-art methods.

Efficient and Refined Deep Convolutional Features Network for the Crack Segmentation of Solar Cell Electroluminescence Images

High-quality and fast crack segmentation in solar cell electroluminescence images with heterogeneously textured background disturbance is challenging. We propose an end-to-end Efficient and Refined Deep Convolutional Features Network (ERDCF-Net) for precise and efficient crack segmentation. Firstly, we design a lightweight, efficient deep convolutional features (EDCF) encoder by adopting multilevel deep convolutional features to generate a discriminative feature representation of cracks. Moreover, some complicated background disturbance is restrained by reusing rich crack and background features. Secondly, we present a refined side output (RS) decoder by automatically refining side outputs from the EDCF encoder. The experiments demonstrate that the proposed network excellently performs on the solar cell cracks (MIoU of 92.82%, F - measure of 93.58%, and 89 FPS) and open crack datasets (MIoU of 85.6%, F - measure of 84.1%, and 90 FPS).

Learning With Noisy Labels via Self-Reweighting From Class Centroids.

Although deep neural networks have been proved effective in many applications, they are data hungry, and training deep models often requires laboriously labeled data. However, when labeled data contain erroneous labels, they often lead to model performance degradation. A common solution is to assign each sample with a dynamic weight during optimization, and the weight is adjusted in accordance with the loss. However, those weights are usually unreliable since they are measured by the losses of corrupted labels. Thus, this scheme might impede the discriminative ability of neural networks trained on noisy data. To address this issue, we propose a novel reweighting method, dubbed self-reweighting from class centroids (SRCC), by assigning sample weights based on the similarities between the samples and our online learned class centroids. Since we exploit statistical class centers in the image feature space to reweight data samples in learning, our method is robust to noise caused by corrupted labels. In addition, even after reweighting the noisy data, the decision boundaries might still suffer distortions. Thus, we leverage mixed inputs that are generated by linearly interpolating two random images and their labels to further regularize the boundaries. We employ the learned class centroids to evaluate the confidence of our generated mixed data via measuring feature similarities. During the network optimization, the class centroids are updated as more discriminative feature representations of original images are learned. In doing so, SRCC will generate more robust weighting coefficients for noisy and mixed data and facilitates our feature representation learning in return. Extensive experiments on both the synthetic and real image recognition tasks demonstrate that our method SRCC outperforms the state of the art on learning with noisy data.

Attention-Aggregated Attribute-Aware Network With Redundancy Reduction Convolution for Video-Based Industrial Smoke Emission Recognition

Existing video-based industrial smoke emission recognition methods face the issues of low detection rates and high false alarm rates. An important reason is that they only consider binary category information as supervision information and ignore video attribute information, which provides important supplementary in improving model performances. To solve it, we propose an attention-aggregated attribute-aware network (AANet). First, to effectively guide the model for discriminative feature learning, a video attribute information decoding module is proposed to increase supervision information by designing attribute vector construction and attribute information decoding methods. Second, to learn discriminative feature representations, some attentions are designed to aggregate spatiotemporal and context information based on ConvLSTM, global feature extraction, and cascaded pyramid attention. Final, the redundancy reduction convolution is proposed to reduce redundant channels by channelwise weights considering matrix elements summation and information spatial distribution characterized by information entropy. Extensive experiments show that AANet significantly outperforms existing methods

Rapid Non-Destructive Analysis of Food Nutrient Content Using Swin-Nutrition.

Food non-destructive detection technology (NDDT) is a powerful impetus to the development of food safety and quality. One of the essential tasks of food quality regulation is the non-destructive detection of the food's nutrient content. However, existing food nutrient NDDT performs poorly in terms of efficiency and accuracy, which hinders their widespread application in daily meals. Therefore, this paper proposed an end-to-end food nutrition non-destructive detection method, named Swin-Nutrition, which combined deep learning and NDDT to evaluate the nutrient content of food. The method aimed to fully capture the feature information from the food images and thus accurately estimate the nutrient content. Swin-Nutrition resorted to Swin Transformer, the feature fusion module (FFM), and the nutrient prediction module to evaluate nutrient content. In particular, Swin Transformer acted as the backbone network for feature extraction of food images, and FFM was used to obtain the discriminative feature representation to improve the accuracy of prediction. The experimental results on the Nutrition5k dataset demonstrated the effectiveness and efficiency of our proposed method. Specifically, the mean value of the percentage mean absolute error (PMAE) for calories, mass, fat, carbohydrate, and protein were only 15.3%, 12.5%, 22.1%, 20.8%, and 15.4%, respectively. We hope that our simple and effective method will provide a solid foundation for the research of food NDDT.

Automatic Laboratory Martian Rock and Mineral Classification Using Highly-Discriminative Representation Derived from Spectral Signatures

The optical properties of rocks and minerals provide a reliable way to measure their chemical and mineralogical composition due to the specific reflection behaviors, which is also the key insight behind most automatic identification and classification approaches. However, the inter-category spectral similarity poses a great challenge to the automatic identification and classification tasks because of the diversity of rocks and minerals. Therefore, this paper develops a recognition and classification approach of rocks and minerals using the highly discriminative representation derived from their raw spectral signatures. More specifically, a transformer-based classification approach integrated with category-aware contrastive learning is constructed and trained in an end-to-end manner, which would force instances of the same category to remain close-by while pushing instances of a dissimilar category far apart in the high-dimensional feature space, in order to produce the highly discriminative feature representation of the rocks and minerals. From both qualitative and quantitative views, experiments are conducted on the laboratory sample dataset with 30 types of rocks and minerals shared from the National Mineral Rock and Fossil Specimens Resource Center, and the spectral information of the laboratory rocks and minerals is captured using a multi-spectral sensor, with a duplicated payload of the counterpart onboard the Zhurong rover. Quantitative results demonstrate that the developed approach can effectively distinguish 30 types of rocks and minerals, with a high overall accuracy of 96.92%. Furthermore, the developed approach is remarkably superior to other existing methods, with average differences of 4.75% in the overall accuracy. Furthermore, we also visualized the derived highly discriminative features of different types of rocks and minerals by projecting them onto a two-dimensional map, where the same categories tend to be modeled by nearby locations and the dissimilar categories by distant locations with high probability. It can be observed that, compared with those in the raw spectral feature space, the clusters are formed better in the derived highly discriminative feature space, which further confirms the promising representation capability.

Disentangling Identity and Pose for Facial Expression Recognition

Facial expression recognition (FER) is a challenging problem because the expression component is always entangled with other irrelevant factors, such as identity and head pose. In this work, we propose an identity and pose disentangled facial expression recognition (IPD-FER) model to learn more discriminative feature representation. We regard the holistic facial representation as the combination of identity, pose and expression. These three components are encoded with different encoders. For identity encoder, a well pre-trained face recognition model is utilized and fixed during training, which alleviates the restriction on specific expression training data in previous works and makes the disentanglement practicable on in-the-wild datasets. At the same time, the pose and expression encoder are optimized with corresponding labels. Combining identity and pose feature, a neutral face of input individual should be generated by the decoder. When expression feature is added, the input image should be reconstructed. By comparing the difference between synthesized neutral and expressional images of the same individual, the expression component is further disentangled from identity and pose. Experimental results verify the effectiveness of our method on both lab-controlled and in-the-wild databases and we achieve state-of-the-art recognition performance.

CENet: A Channel-Enhanced Spatiotemporal Network With Sufficient Supervision Information for Recognizing Industrial Smoke Emissions

Vision-based industrial smoke emission recognition technology can identify smoke emissions and provide a visual evidence for humans to pursue environmental justice. However, the existing methods still face the issues of low detection rates (DRs) and high false alarm rates (FARs) due to the insufficient supervision information and limited feature representation capability. To solve these issues, this article presents a channel-enhanced spatiotemporal network (CENet) with sufficient supervision information for recognizing industrial smoke emissions. First, to provide sufficient supervision information for learning discriminative feature representation, we propose a new loss function by collaboratively taking the binary category, pixel-level smoke density, and background information as supervision information and use them in the network final layer as well as the network middle layers to guide model training. Second, to solve the deficiencies of max/average pooling and convolution operations in feature extraction, we propose channel-enhanced modules, including channel-enhanced pooling (CEPool), channel-enhanced convolution (CEConv), and channel-enhanced upsampling (CEUpsample) to learn high-response values of bright features as well as the low-response values with discriminative features. The channel-enhanced modules selectively enhance the learned features with large amount of information and suppress those useless features. Third, we propose a two-stage method based on spatiotemporal information extraction module (SIEM) and smoke recognition module (SRM), which are designed to learn spatiotemporal information between input frames and that between smoke density frames, respectively. Extensive experiments show that CENet achieves the best performance among the existing smoke recognition methods.

ARHPE: Asymmetric Relation-Aware Representation Learning for Head Pose Estimation in Industrial Human–Computer Interaction

Head pose estimation (HPE) has wide industrial applications, such as online education, human–robot interaction, and automatic manufacturing. In this article, we address two key problems in HPE based on label learning and asymmetric relation cues: 1) how to bridge the gap between the better prediction performance of networks and incorrectly label pose images in the HPE datasets and 2) how to take full advantage of the adjacent poses information around the centered pose image. We reconstruct all the incorrect labels as a two-dimensional Lorentz distribution to tackle the first problem. Instead of directly adopting the angle values as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">hard</i> labels, we assign part of the probability values ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">soft</i> labels) to adjacent labels for learning discriminative feature representations. To address the second problem, we reveal the asymmetric relation nature of HPE datasets. The yaw direction and pitch direction are assigned different weights by introducing the half at half-maximum of the Lorentz distribution. Compared with the traditional end-to-end frameworks, the proposed one can leverage the asymmetric relation cues for predicting the head pose angle in the incorrect label scenarios. Extensive experiments on two public datasets and our infrared dataset demonstrate that the proposed ARHPE network significantly outperforms other state-of-the-art approaches.

Learning the cross-modal discriminative feature representation for RGB-T crowd counting

To reduce the interference of the arbitrary crowd distribution and complex background on the counting accuracy in the unconstrained scene, this paper presents a novel RGB-T crowd counting approach based on the cross-modal discriminative feature representation learning, including two main stages. Specially, the first stage explicitly establishes a non-linear mapping from the RGB domain to the thermal domain so as to learn the cross-modal discriminative feature representation about the crowd distribution, given that the thermal image of the crowded scene presents more intuitive information related to the crowd and would be more insensitive to the background element than the conventional optical image. The second stage incorporates the complementary features of the RGB and thermal image pair based on combining the learned feature representation from the first stage to yield the final counting result. Experiments on the RGB-T crowd counting benchmark verify the superiority of the proposed approach with the state-of-the-art methods. The ablation studies on RGB-T and the evaluation on RGB crowd counting benchmarks validate the effectiveness of the designed cross-modal discriminative feature representation learning. Experimental results demonstrate that the proposed approach could obviate the need of relying on the specific structure of modality-specific feature extraction parts, rather than the conventional works of treating the paired RGB-thermal image as the indiscriminative information source and fusing them directly. The proposed approach in this study could realize the cross-modal discriminative feature representation learning in an efficient way of establishing cross-modal domain explicitly for the algorithm research of the intelligent surveillance system development.

ReID-DeePNet: A Hybrid Deep Learning System for Person Re-Identification

Person re-identification has become an essential application within computer vision due to its ability to match the same person over non-overlapping cameras. However, it is a challenging task because of the broad view of cameras with a large number of pedestrians appearing with various poses. As a result, various approaches of supervised model learning have been utilized to locate and identify a person based on the given input. Nevertheless, several of these approaches perform worse than expected in retrieving the right person in real-time over multiple CCTVs/camera views. This is due to inaccurate segmentation of the person, leading to incorrect classification. This paper proposes an efficient and real-time person re-identification system, named ReID-DeePNet system. It is based on fusing the matching scores generated by two different deep learning models, convolutional neural network and deep belief network, to extract discriminative feature representations from the pedestrian image. Initially, a segmentation procedure was developed based on merging the advantages of the Mask R-CNN and GrabCut algorithm to tackle the adverse effects caused by background clutter. Afterward, the two different deep learning models extracted discriminative feature representations from the pedestrian segmented image, and their matching scores were fused to make the final decision. Several extensive experiments were conducted, using three large-scale and challenging person re-identification datasets: Market-1501, CUHK03, and P-DESTRE. The ReID-DeePNet system achieved new state-of-the-art Rank-1 and mAP values on these three challenging ReID datasets.