Global Feature Representation Research Articles

Part-Aware Mask-Guided Attention for Thorax Disease Classification.

Thorax disease classification is a challenging task due to complex pathologies and subtle texture changes, etc. It has been extensively studied for years largely because of its wide application in computer-aided diagnosis. Most existing methods directly learn global feature representations from whole Chest X-ray (CXR) images, without considering in depth the richer visual cues lying around informative local regions. Thus, these methods often produce sub-optimal thorax disease classification performance because they ignore the very informative pathological changes around organs. In this paper, we propose a novel Part-Aware Mask-Guided Attention Network (PMGAN) that learns complementary global and local feature representations from all-organ region and multiple single-organ regions simultaneously for thorax disease classification. Specifically, multiple innovative soft attention modules are designed to progressively guide feature learning toward the global informative regions of whole CXR image. A mask-guided attention module is designed to further search for informative regions and visual cues within the all-organ or single-organ images, where attention is elegantly regularized by automatically generated organ masks and without introducing computation during the inference stage. In addition, a multi-task learning strategy is designed, which effectively maximizes the learning of complementary local and global representations. The proposed PMGAN has been evaluated on the ChestX-ray14 dataset and the experimental results demonstrate its superior thorax disease classification performance against the state-of-the-art methods.

Hashing person re-ID with self-distilling smooth relaxation

Person re-identification (re-ID) has made substantial progress in recent years; however, it is still challenging to search for the target person in a short time. Re-ID with deep hashing is a shortcut for that but, limited by the expression of binary code, the performance of the hashing method is not satisfactory. Besides, to further speed up retrieval, researchers tend to reduce the number of feature bits, which will cause more performance degradation. In this paper, we design the attribute-based fast retrieval (AFR), which leverages the attribute prediction of the model trained in a binary classification manner tailor-made for hashing. The attribute information is also used to refine the global feature representation by an attribute-guided attention block (AAB). Then, to fully exploit deep feature to generate the hash codes, we propose a binary code learning method, named self-distilling smooth relaxation (SSR). In this method, a simple yet effective regularization is presented to distill the quantized knowledge in the model itself, thus mitigating the lack of semantic guidance in the traditional non-linear relaxations. We manually label attributes for each person in dataset CUHK03 and evaluate our method on four authoritative public benchmarks (Market-1501, Market-1501+500K, CUHK03, and DukeMTMC-reID). The experimental results indicate that with the SSR and AAB, we surpass all the state-of-the-art hashing methods. And compared with reducing the feature bits, the AFR strategy is more effective to save search time.

Dual adaptive alignment and partitioning network for visible and infrared cross-modality person re-identification

Visible and infrared person re-identification (VI-ReID) describes the task of matching the images of a person, captured by visible-light and infrared cameras; this is a particular challenge in night time surveillance applications. Existing cross-modality recognition studies have been conducted mainly with a focus on learning the global and shareable feature representation of pedestrians to handle cross-modality discrepancies. However, the global features of pedestrian images cannot solve the unaligned image pairs efficiently, particularly when encountering the human appearance or posture misalignment caused by inaccurate pedestrian detection boxes. To mitigate the impact of these problems, we propose an end-to-end dual alignment and partitioning network to simultaneously learn global and local modal invariant features of pedestrians. First, we use two adaptive spatial transform modules to align the visible and infrared input images. Subsequently, the aligned image is divided horizontally, and the features of each local block are extracted. Then, we fuse these local features with global features. To alleviate the differences between heterogeneous modals and learn the common feature representation of heterogeneous modals, we map the features of heterogeneous modes into the same feature embedding space. Finally, we use the combination of identity loss and weighted regularized TriHard loss to improve the recognition accuracy. Extensive experimental results on two cross-modality datasets, RegDB and SYSU-MM01, demonstrate the superiority of the proposed method over other existing state-of-the-art methods.

Multiblock temporal convolution network-based temporal-correlated feature learning for fault diagnosis of multivariate processes

A new temporal-correlated feature learning method, multiblock temporal convolutional network (MBTCN), is proposed for supervised fault diagnosis of multivariate processes in this paper. The MBTCN used a new idea, "local extraction and global integration," to consider the cross-correlation and the temporal-correlation in the multivariate processes' data. First, MBTCN divides the overall variables into several sub-blocks based on process mechanisms and uses one-dimensional convolutional neural network (1D-CNN) architectures to extract temporal-correlated features in each sub-block, with the 1D-CNN network sliding over time steps. Thus, the adjacent samples and the close-related variables can be considered together in the network. Then, MBTCN constructs a global feature representation built by concatenating local features of sub-blocks. Besides, in the supervised training phase, training labels are modified by label smoothing technology to alleviate the overfitting. Finally, a Tennessee Eastman (TE) process is used to test the proposed model's effectiveness. The code of MBTCN can be found in https://github.com/heyumi0901/MBTCN.

Joint feature representation learning and progressive distribution matching for cross-project defect prediction

Cross-Project Defect Prediction (CPDP) aims to leverage the knowledge from label-rich source software projects to promote tasks in a label-poor target software project. Existing CPDP methods have two major flaws. One is that previous CPDP methods only consider global feature representation and ignores local relationship between instances in the same category from different projects, resulting in ambiguous predictions near the decision boundary. The other one is that CPDP methods based on pseudo-labels assume that the conditional distribution can be well matched at one stroke, when instances of target project are correctly annotated pseudo labels. However, due to the great gap between projects, the pseudo-labels seriously deviate from the real labels. To address above issues, this paper proposed a novel CPDP method named Joint Feature Representation with Double Marginalized Denoising Autoencoders (DMDA_JFR). Our method mainly includes two parts: joint feature representation learning and progressive distribution matching. We utilize two novel autoencoders to jointly learn the global and local feature representations simultaneously. To achieve progressive distribution matching, we introduce a repetitious pseudo-labels strategy, which makes it possible that distributions are matched after each stack layer learning rather than in one stroke. The effectiveness of the proposed method was evaluated through experiments conducted on 10 open-source projects, including 29 software releases from PROMISE repository. Overall, experimental results show that our proposed method outperformed several state-of-the-art baseline CPDP methods. It can be concluded that (1) joint deep representations are promising for CPDP compared with only considering global feature representation methods, (2) progressive distribution matching is more effective for adapting probability distributions in CPDP compared with existing CPDP methods based on pseudo-labels. • This paper proposed a novel CPDP method named Joint Feature Representation with Double Marginalized Denoising Autoencoders (DMDA-JFR). • Our method mainly includes two parts: joint feature representation learning and progressive distribution matching. • We utilize two novel autoencoders to jointly learn the global and local feature representations simultaneously. • We introduce a repetitious pseudo-labels strategy to progressively match distribution.

Tra2Tra: Trajectory-to-Trajectory Prediction With a Global Social Spatial-Temporal Attentive Neural Network

Accurate trajectory prediction plays a key role in robot navigation. It is beneficial for planning a collision-free and appropriate path for the autonomous robots, especially in crowded scenes. However, it is a particularly challenging task because there are complex and subtle interactions among pedestrians. There have been many studies focusing on how to model this spatial interactions but most of them neglected the temporal characteristic. Towards this end, we propose a novel Global Social Spatial-Temporal Attentive Neural Network for trajectory-to-trajectory prediction (Tra2Tra). In this model, we first extract features of spatial interactions with decentralization operation and attention mechanism, and then iteratively extract its temporal dependency through the Long Short-Term Memory network for obtaining the global spatial-temporal feature representation. We further aggregate this global spatial-temporal feature representation and velocity features into our encoder-decoder module for prediction. In order to make multi-modality predictions, we introduce a random noise perturbation while decoding, which enhances the robustness and the generalization ability of our model. Experimental results demonstrate that our Tra2Tra model can achieve better performance than the state-of-the-art methods not only on two pedestrian-walking datasets, i.e. ETH and UCY, but also on three other complex trajectory datasets, i.e. Collisions, NGsim and Charges.

Learning deep part-aware embedding for person retrieval

Person retrieval is an important vision task, aiming at matching the images of the same person under various camera views. The key challenge of person retrieval lies in the large intra-class variations among the person images. Therefore, how to learn discriminative feature representations becomes the core problem. In this paper, we propose a deep part-aware representation learning method for person retrieval. First, an improved triplet loss is introduced such that the global feature representations from the same identity are closely clustered. Meanwhile, a localization branch is proposed to automatically localize those discriminative person-wise parts or regions, only using identity labels in a weakly supervised manner. Via the learning simultaneously guided by the global branch and the localization branch, the proposed method can further improve the performance for person retrieval. Through an extensive set of ablation studies, we verify that the localization branch and the improved triplet loss each contributes to the performance boosts of the proposed method. Our model obtains superior (or comparable) performance compared to state-of-the-art methods for person retrieval on the four public person retrieval datasets. On the CUHK03-labeled dataset, for instance, the performance increases from 73.0% mAP and 77.9% rank-1 accuracy to 80.8% (+7.8%) mAP and 83.9% (+6.0%) rank-1 accuracy.

Attention based convolutional network for automatic sleep stage classification.

Sleep staging is an important basis for diagnosing sleep-related problems. In this paper, an attention based convolutional network for automatic sleep staging is proposed. The network takes time-frequency image as input and predict sleep stage for each 30-s epoch as output. For each CNN feature maps, our model generate attention maps along two separate dimensions, time and filter, and then multiplied to form the final attention map. Residual-like fusion structure is used to append the attention map to the input feature map for adaptive feature refinement. In addition, to get the global feature representation with less information loss, the generalized mean pooling is introduced. To prove the efficacy of the proposed method, we have compared with two baseline method on sleep-EDF data set with different setting of the framework and input channel type, the experimental results show that the paper model has achieved significant improvements in terms of overall accuracy, Cohen's kappa, MF1, sensitivity and specificity. The performance of the proposed network is compared with that of the state-of-the-art algorithms with an overall accuracy of 83.4%, a macro F1-score of 77.3%, κ=0.77, sensitivity=77.1% and specificity=95.4%, respectively. The experimental results demonstrate the superiority of the proposed network.

Compact Global-Local Convolutional Network With Multifeature Fusion and Learning for Scene Classification in Synthetic Aperture Radar Imagery

Feature learning of convolutional neural networks (CNNs) has gained considerable attention and achieved good performance on synthetic aperture radar (SAR) image scene classification. However, the performance of the existing convolutional feature learning methods is limited for generating the distinguishable feature representations because such techniques inherently suffer from shortcomings, i.e., they do not consider the local feature distribution of deep orderless feature statistics and deep orderless multifeature learning style. To alleviate these drawbacks, we propose a compact global-local convolutional network with multifeature fusion and learning (CGML) for SAR image scene classification, which contains double branches of convolutional feature learning net (C-net) and local feature distribution learning net (L-net). L-net employs the localized and parameterized affine subspace coding layer for local feature distribution learning and captures the feature statistics of each cluster center via detailed local feature division. The standard convolutional feature map is utilized for the convolutional feature learning in C-net. Subsequently, the compact multifeature fusion and learning strategy captures the compact global second-order orderless feature representation and allows the double branches to interact with each other via the tensor sketch algorithm. Especially, the feature learning strategy of L-net is defined in affine subspace which fully characterizes the feature distribution inside each cluster space. Finally, we concatenate the outputs of the multifeature fusion and learning network, then pool and feed them into softmax loss. Based on extensive evaluations on TerraSAR-X1 and TerraSAR-X2 image scene classification datasets, CGML can yield superior performances when compared with those of several state-of-the-art networks.

Multi-Task Deep Neural Networks for Joint Sarcasm Detection and Sentiment Analysis

Sentiment analysis and sarcasm detection are specialized areas in the field of information retrieval and natural language processing. Sentiment classification is closely correlated with sarcasm detection, where people usually adopt sarcasm to highlight their negative feeling. This paper proposes a novel multi-task deep neural networks for joint sarcasm detection and sentiment analysis (MT_SS). MT_SS train both tasks jointly using bidirectional gated recurrent unit with attention network module to obtain task-specific local feature representation while using convolutional neural networks to obtain global feature representation. The experiments on two datasets show that our proposed model outperforms the state-of-the-art approaches.

Representation Learning With Dual Autoencoder for Multi-Label Classification

Multi-label classification aims to deal with the problem that an object may be associated with one or more labels, which is a more difficult task due to the complex nature of multi-label data. The crucial problem of multi-label classification is the more robust and higher-level feature representation learning, which can reduce non-helpful feature attributes from the input space prior to training. In recent years, deep learning methods based on autoencoders have achieved excellent performance in multi-label classification for the advantages of powerful representations learning ability and fast convergence speed. However, most existing autoencoder-based methods only rely on the single autoencoder model, which pose challenges for multi-label feature representations learning and fail to measure similarities between data spaces. To address this problem, in this paper, we propose a novel representation learning method with dual autoencoder for multi-label classification. Compared to the existing autoencoder-based methods, our proposed method can capture different characteristics and more abstract features from data by the serially connection of two different types of autoencoders. More specifically, firstly, the algorithm of Reconstruction Independent Component Analysis (RICA) in sparse autoencoder is trained on patches on all training and test dataset for robust global feature representations learning. Secondly, with the output of RICA, stacked autoencoder with manifold regularization (SAMR) is introduced to ameliorate the quality of multi-label features learning. Comprehensive experiments on several real-world data sets demonstrate the effectiveness of our proposed approach compared with several competing state-of-the-art methods.

Aggregated Deep Global Feature Representation for Breast Cancer Histopathology Image Classification

Convolutional neural networks (CNNs), successfully used in a great number of medical image analysis applications, have also achieved the state-of-the-art performance in breast cancer histopathology image (BCHI) classification problem recently. However, due to the large varieties among within-class images and insufficient data volume, it is still a challenge to obtain more competitive results by using deep CNN models alone. In this paper, we aim to explore the combination of CNN models with a milestone feature representation method in visual tasks, i.e., vector of locally aggregated descriptors (VLAD), for the BCHI classification, and further propose a novel aggregated deep global feature representation (ADGFR) for this problem. ADGFR adopts the deep features that are extracted from the fully connected layer to form an individual descriptor vector, and augments input images to generate different descriptors for achieving the final aggregated descriptor vector. The individual descriptor vector can effectively keep the global features of benign and malignant image, whose discriminability is further reinforced by the aggregate operation, leading to the more discriminant capability of ADGFR for BCHI. Extensive experiments on the public Break His dataset illuminate that our ADGFR can achieve the optimal classification accuracies of 95.05% at image level and 95.50% at patient level, respectively.

Learning Expression Features via Deep Residual Attention Networks for Facial Expression Recognition From Video Sequences

Facial expression recognition from video sequences is currently an interesting research topic in computer vision, pattern recognition, artificial intelligence, etc. Considering the problem of semantic gap between the extracted hand-designed features in affective videos and subjective emotions, recognizing facial expressions from video sequences is a challenging subject. To tackle this problem, this paper proposes a new method of facial expression recognition from video sequences via deep residual attention network. Firstly, due to the difference in the intensity of emotional representation of each local area in a facial image, deep residual attention networks are employed to learn high-level affective expression features for each frame of facial expression images in video sequences. The used deep residual attention networks integrate deep residual networks with a spatial attention mechanism. Then, average-pooling is performed to produce fixed-length global video-level feature representations. Finally, the global video-level feature representations are utilized as inputs of a multi-layer perceptron to conduct facial expression classification tasks in video sequences. Experimental results on two public video emotional datasets, i.e. BAUM-1s and RML, demonstrate the effectiveness of the proposed method.

Machine learning method for cosmetic product recognition: a visual searching approach

A cosmetic product recognition system is proposed in this paper. For this recognition system, we have proposed a cosmetic product database that contains image samples of forty different cosmetic items. The purpose of this recognition system is to recognize Cosmetic products with there types, brands and retailers such that to analyze a customer experience what kind of products and brands they need. This system has various applications in such as brand recognition, product recognition and also the availability of the products to the vendors. The implementation of the proposed system is divided into three components: preprocessing, feature extraction and classification. During preprocessing we have scaled and transformed the color images into gray-scaled images to speed up the process. During feature extraction, several different feature representation schemes: transformed, structural and statistical texture analysis approaches have been employed and investigated by employing the global and local feature representation schemes. Various machine learning supervised classification methods such as Logistic Regression, Linear Support Vector Machine, Adaptive k-Nearest Neighbor, Artificial Neural Network and Decision Tree classifiers have been employed to perform the classification tasks. Apart from this, we have also performed some data analytic tasks for Brand Recognition as well as Retailer Recognition and for these experimentation, we have employed some datasets from the ‘Kaggle’ website and have obtained the performance due to the above-mentioned classifiers. Finally, the performance of the cosmetic product recognition system, Brand Recognition and Retailer Recognition have been aggregated for the customer decision process in the form of the state-of-the-art for the proposed system.

Cross-modal video moment retrieval based on visual-textual relationship alignment

In recent years, increasing amounts of video resources have created a series of demands for fine retrieval of video moments, such as highlight moments in sports events and the re-creation of specific video content. In this context, research on cross-modal video segment retrieval, which attempts to output a video moment that matches the input query text, is gradually emerging. Existing solutions primarily focus on global or local feature representation for query text and video moments. However, such solutions ignore matching semantic relations contained in query text and video moments. For example, given the query text “a person is playing basketball, existing retrieval systems may incorrectly return a video moment of “a person holding a basketball without the considering the semantic relationship of “a person playing basketball. Therefore, this paper proposes a cross-modal relationship alignment framework, which we refer to as CrossGraphAlign, for cross-modal video moment retrieval. The proposed framework constructs a textual relationship graph and a visual relationship graph to model the query semantics in text and video segment relations, and then evaluates the similarity between text relations and visual relations through cross-modally aligned graph convolutional networks to help construct a more accurate video moment retrieval system. Experimental results on the publicly available cross-modal video retrieval datasets TACoS and ActivityNet Captions demonstrate that the proposed method can effectively utilize the semantic relationships to improve the recall rate in cross-modal video moment retrieval.

Transferring Deep Belief Networks for the Classification of LANDSAT8 Remote Sensing Imagery

This paper studies the structure and characteristics of LANDSAT8 OLI data, and transfers Deep Belief Networks model to OLI datasets. Firstly, the sub-sampling window is used to filter the pretraining data, which reduces the calculation cost of the hidden layer feature representation of the model. The parameters of each layer of DBN are trained forward, optimization parameters are used to make the error converge to a low value in back propagation. Finally, the activation function is used to reduce the dimension of the hidden layer of the model, and the output vector is used as the global feature representation of the image. Running DBN to complete the classification of LANDSAT8 remote sensing image in the study area, the experimental results show that the model has completed the classification and achieved high user accuracy and reliability in the case of that all types of the wetlands have similar spectral characteristics. The conclusion is that the pretrained features can be well applied to the LANDSAT8 remote sensing datasets. The accuracy is improved within the scope of cost calculation, and the local minina problem will also occur in deepening model.

Structure-Aware Feature Fusion for Unsupervised Domain Adaptation

Unsupervised domain Adaptation (UDA) aims to learn and transfer generalized features from a labelled source domain to a target domain without any annotations. Existing methods only aligning high-level representation but without exploiting the complex multi-class structure and local spatial structure. This is problematic as 1) the model is prone to negative transfer when the features from different classes are misaligned; 2) missing the local spatial structure poses a major obstacle in performing the fine-grained feature alignment. In this paper, we integrate the valuable information conveyed in classifier prediction and local feature maps into global feature representation and then perform a single mini-max game to make it domain invariant. In this way, the domain-invariant feature not only describes the holistic representation of the original image but also preserves mode-structure and fine-grained spatial structural information. The feature integration is achieved by estimating and maximizing the mutual information (MI) among the global feature, local feature and classifier prediction simultaneously. As the MI is hard to measure directly in high-dimension spaces, we adopt a new objective function that implicitly maximizes the MI via an effective sampling strategy and a discriminator design. Our STructure-Aware Feature Fusion (STAFF) network achieves the state-of-the-art performances in various UDA datasets.

Optimal Feature Transport for Cross-View Image Geo-Localization

This paper addresses the problem of cross-view image geo-localization, where the geographic location of a ground-level street-view query image is estimated by matching it against a large scale aerial map (e.g., a high-resolution satellite image). State-of-the-art deep-learning based methods tackle this problem as deep metric learning which aims to learn global feature representations of the scene seen by the two different views. Despite promising results are obtained by such deep metric learning methods, they, however, fail to exploit a crucial cue relevant for localization, namely, the spatial layout of local features. Moreover, little attention is paid to the obvious domain gap (between aerial view and ground view) in the context of cross-view localization. This paper proposes a novel Cross-View Feature Transport (CVFT) technique to explicitly establish cross-view domain transfer that facilitates feature alignment between ground and aerial images. Specifically, we implement the CVFT as network layers, which transports features from one domain to the other, leading to more meaningful feature similarity comparison. Our model is differentiable and can be learned end-to-end. Experiments on large-scale datasets have demonstrated that our method has remarkably boosted the state-of-the-art cross-view localization performance, e.g., on the CVUSA dataset, with significant improvements for top-1 recall from 40.79% to 61.43%, and for top-10 from 76.36% to 90.49%. We expect the key insight of the paper (i.e., explicitly handling domain difference via domain transport) will prove to be useful for other similar problems in computer vision as well.

Deep Learning-Based Approach for Sign Language Gesture Recognition With Efficient Hand Gesture Representation

Hand gesture recognition is an attractive research field with a wide range of applications, including video games and telesurgery techniques. Another important application of hand gesture recognition is the translation of sign language, which is a complicated structured form of hand gestures. In sign language, the fingers' configuration, the hand's orientation, and the hand's relative position to the body are the primitives of structured expressions. The importance of hand gesture recognition has increased due to the prevalence of touchless applications and the rapid growth of the hearing-impaired population. However, developing an efficient recognition system needs to overcome the challenges of hand segmentation, local hand shape representation, global body configuration representation, and gesture sequence modeling. In this paper, a novel system is proposed for dynamic hand gesture recognition using multiple deep learning architectures for hand segmentation, local and global feature representations, and sequence feature globalization and recognition. The proposed system is evaluated on a very challenging dataset, which consists of 40 dynamic hand gestures performed by 40 subjects in an uncontrolled environment. The results show that the proposed system outperforms state-of-the-art approaches, demonstrating its effectiveness.

No-reference mesh visual quality assessment via ensemble of convolutional neural networks and compact multi-linear pooling

Blind or No reference quality evaluation is a challenging issue since it is done without access to the original content. In this work, we propose a method based on deep learning for the mesh visual quality assessment without reference. For a given 3D model, we first compute its mesh saliency. Then, we extract views from the 3D mesh and the corresponding mesh saliency. After that, the views are split into small patches that are filtered using a saliency threshold. Only the salient patches are selected and used as input data. After that, three pre-trained deep convolutional neural networks are employed for feature learning: VGG, AlexNet, and ResNet. Each network is fine-tuned and produces a feature vector. The Compact Multi-linear Pooling (CMP) is used afterward to fuse the retrieved vectors into a global feature representation. Finally, fully connected layers followed by a regression module are used to estimate the quality score. Extensive experiments are executed on four mesh quality datasets and comparisons with existing methods demonstrate the effectiveness of our method in terms of correlation with subjective scores.