Visual Feature Learning Research Articles

A systematic literature review of visual feature learning: deep learning techniques, applications, challenges and future directions

A systematic literature review of visual feature learning: deep learning techniques, applications, challenges and future directions

A Multi-Group Multi-Stream attribute Attention network for fine-grained zero-shot learning

Fine-grained visual categorization in zero-shot setting is a challenging problem in the computer vision community. It requires algorithms to accurately identify fine-grained categories that do not appear during the training phase and have high visual similarity to each other. Existing methods usually address this problem by using attribute information as intermediate knowledge, which provides sufficient fine-grained characteristics of categories and can be transferred from seen categories to unseen categories. However, the learning of attribute visual features is not trivial due to the following two reasons: (i) The visual information about attributes of different types may interfere with the visual feature learning of each other. (ii) The visual characteristics of the same attribute may vary in different categories. To solve these issues, we propose a Multi-Group Multi-Stream attribute Attention network (MGMSA), which not only separates the feature learning of attributes of different types, but also isolates the learning of attribute visual features for categories with big differences in attribute appearance. This avoids the interference between uncorrelated attributes and helps to learn category-specific attribute-related visual features. This is beneficial for distinguishing fine-grained categories with subtle visual differences. Extensive experiments on benchmark datasets show that MGMSA achieves state-of-the-art performance on attribute prediction and fine-grained zero-shot learning.

Unifying Visual and Vision-Language Tracking via Contrastive Learning

Single object tracking aims to locate the target object in a video sequence according to the state specified by different modal references, including the initial bounding box (BBOX), natural language (NL), or both (NL+BBOX). Due to the gap between different modalities, most existing trackers are designed for single or partial of these reference settings and overspecialize on the specific modality. Differently, we present a unified tracker called UVLTrack, which can simultaneously handle all three reference settings (BBOX, NL, NL+BBOX) with the same parameters. The proposed UVLTrack enjoys several merits. First, we design a modality-unified feature extractor for joint visual and language feature learning and propose a multi-modal contrastive loss to align the visual and language features into a unified semantic space. Second, a modality-adaptive box head is proposed, which makes full use of the target reference to mine ever-changing scenario features dynamically from video contexts and distinguish the target in a contrastive way, enabling robust performance in different reference settings. Extensive experimental results demonstrate that UVLTrack achieves promising performance on seven visual tracking datasets, three vision-language tracking datasets, and three visual grounding datasets. Codes and models will be open-sourced at https://github.com/OpenSpaceAI/UVLTrack.

Towards Efficient and Effective Text-to-Video Retrieval with Coarse-to-Fine Visual Representation Learning

In recent years, text-to-video retrieval methods based on CLIP have experienced rapid development. The primary direction of evolution is to exploit the much wider gamut of visual and textual cues to achieve alignment. Concretely, those methods with impressive performance often design a heavy fusion block for sentence (words)-video (frames) interaction, regardless of the prohibitive computation complexity. Nevertheless, these approaches are not optimal in terms of feature utilization and retrieval efficiency. To address this issue, we adopt multi-granularity visual feature learning, ensuring the model's comprehensiveness in capturing visual content features spanning from abstract to detailed levels during the training phase. To better leverage the multi-granularity features, we devise a two-stage retrieval architecture in the retrieval phase. This solution ingeniously balances the coarse and fine granularity of retrieval content. Moreover, it also strikes a harmonious equilibrium between retrieval effectiveness and efficiency. Specifically, in training phase, we design a parameter-free text-gated interaction block (TIB) for fine-grained video representation learning and embed an extra Pearson Constraint to optimize cross-modal representation learning. In retrieval phase, we use coarse-grained video representations for fast recall of top-k candidates, which are then reranked by fine-grained video representations. Extensive experiments on four benchmarks demonstrate the efficiency and effectiveness. Notably, our method achieves comparable performance with the current state-of-the-art methods while being nearly 50 times faster.

Tabular text embedding for Vietnamese text-based person search

Vietnamese text-based person search is still a challenging problem with the limited dataset of Vietnamese descriptions. The current popular approach to this problem is Deep Neural Networks (DNNs), and recently, transformer networks have been more favored because of their outperformance over CNN and RNN networks for both vision and natural language processing tasks. However, DNN, or transformer networks, require a large amount of training data and computing time for efficient learning of visual and textual features. This brings a burden for implementing Vietnamese text-based person search by DNN, or transformer networks. Towards building a Vietnamese text-based person search system on a scarce resource dataset of Vietnamese descriptive sentences with low computing cost, in this work, we propose to apply the transformer-based architecture named TabTransformer for contextual embedding of the noun phrases chunked from the Vietnamese descriptive sentences. This is the first time the TabTransformer network has been deployed together with CNN and RNN architectures for Vietnamese text-based person search. The experimental results on a limited dataset of 3000VnPersonSearch show the better recognition accuracy of the proposed method compared to the baseline method by about 7.5% at Rank 1. In addition, the computing time of our method is more effective than the baseline method.

A novel method of battery pack energy health estimation based on visual feature learning

Accurate health estimation of massive battery packs and efficient optimization of data storage have become major technical challenges with the development of big data platforms. In this paper, multi-level energy indicators are defined to reflect the overall health state of the battery pack, and battery pack health assessment is achieved through energy estimation. A visual feature learning method is proposed to extract features from partial cell charging voltage profiles image and the relationship between features and energy indicators is constructed by the hybrid convolutional neural network. The effectiveness of the proposed method is verified on the dataset generated by the battery pack model considering cell inconsistency, and the mean absolute percentage error of each energy indicator estimation is less than 1%. Additionally, validations are carried out on simulated data with sampling noise and two cases of experimental data to verify the stability of the method. The proposed visual feature learning method provides a new idea for the data storage and health monitoring of massive battery packs on the big data platform.

Gloss Prior Guided Visual Feature Learning for Continuous Sign Language Recognition.

Continuous sign language recognition (CSLR) is to recognize the glosses in a sign language video. Enhancing the generalization ability of CSLR's visual feature extractor is a worthy area of investigation. In this paper, we model glosses as priors that help to learn more generalizable visual features. Specifically, the signer-invariant gloss feature is extracted by a pre-trained gloss BERT model. Then we design a gloss prior guidance network (GPGN). It contains a novel parallel densely-connected temporal feature extraction (PDC-TFE) module for multi-resolution visual feature extraction. The PDC-TFE captures the complex temporal patterns of the glosses. The pre-trained gloss feature guides the visual feature learning through a cross-modality matching loss. We propose to formulate the cross-modality feature matching into a regularized optimal transport problem, it can be efficiently solved by a variant of the Sinkhorn algorithm. The GPGN parameters are learned by optimizing a weighted sum of the cross-modality matching loss and CTC loss. The experiment results on German and Chinese sign language benchmarks demonstrate that the proposed GPGN achieves competitive performance. The ablation study verifies the effectiveness of several critical components of the GPGN. Furthermore, the proposed pre-trained gloss BERT model and cross-modality matching can be seamlessly integrated into other RGB-cue-based CSLR methods as plug-and-play formulations to enhance the generalization ability of the visual feature extractor.

Perceptual Visual Feature Learning With Applications in Sports Educational Image Understanding

Perceptual Visual Feature Learning With Applications in Sports Educational Image Understanding

SURG-04. FAST DETECTION OF GLIOMA INFILTRATION USING LABEL-FREE OPTICAL MICROSCOPY AND DEEP NEURAL NETWORKS

Abstract INTRODUCTION Fast and accurate detection of tumor infiltration is a central challenge in the comprehensive management of diffuse glioma patients. Despite our best real-time tumor detection strategies, such as intraoperative MRI and fluorescence-guided surgery, dense and safely resectable tumor infiltration remains at the surgical margin in over 85% of diffuse glioma patients. Patterns of treatment failure demonstrate that glioma recurrence occurs at the surgical margin in the majority of patients due to residual tumor burden. Objective Here, we present FastGlioma, an artificial intelligence (AI)-based intraoperative diagnostic system for fast (< 10 seconds) detection of diffuse glioma infiltration at microscopic resolution without the need for tumor-specific markers. Method FastGlioma is trained using large-scale, self-supervised visual feature learning on stimulated Raman histology (SRH), a rapid, label-free, optical microscopy technique, and a GPT-style whole slide SRH tumor infiltration scoring method. We pushed the performance limits of FastGlioma by investigating the trade-off between imaging speed versus accuracy for microscopic tumor detection. RESULTS In a large prospective external testing cohort of diffuse glioma patients (180 patients, 935 surgical margin specimens) who underwent intraoperative SRH imaging, we demonstrate that FastGlioma was able to detect and quantify microscopic tumor infiltration with an average AUROC of 92.1 +/- 0.0%, performing on par with three expert neuropathologists on the same task. FastGlioma is over 10X faster than previous SRH-based methods and 100X faster than conventional H&E histology. Finally, we demonstrate that FastGlioma was > 20% more accurate at detecting dense tumor infiltration when compared to radiologic features or 5-aminolevulinic fluorescence (98.0% versus 77.8% accuracy) for both IDH-mutant diffuse gliomas and IDH-wildtype glioblastomas. CONCLUSION Our results demonstrate how intraoperative optical imaging and deep neural networks can achieve fast and accurate tumor detection, unlocking the role of AI in improving the surgical management of brain tumor patients.

Still an Ineffective Method With supertrials/ERPs-Comments on "decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features".

A recent paper claims that a newly proposed method classifies EEG data recorded from subjects viewing ImageNet stimuli better than two prior methods. However, the analysis used to support that claim is based on confounded data. We repeat the analysis on a large new dataset that is free from that confound. Training and testing on aggregated supertrials derived by summing trials demonstrates that the two prior methods achieve statistically significant above-chance accuracy while the newly proposed method does not.

Learning neighbor-enhanced region representations and question-guided visual representations for visual question answering

Great strides have been made in visual question answering field (VQA) based on the application and development of deep learning in related research fields. Existing models in this field focus on the learning and fusion of visual and textual features. However, it is extremely crucial for VQA tasks to focus on the associations between image regions and use question information to enhance key features. In this paper, we propose a method for mining and integrating neighbor-enhanced region representations and question-guided visual representations. Particularly, the region feature graph is first constructed to integrate the features of all regions and the relationships between regions. Secondly, a random walk-based method is presented to acquire the neighbor-enhanced region representations, which combines the topological relationships of all region nodes in the graph. The question-guided vertical and horizontal dual attention mechanism is then proposed to enhance the region representation from the region level and the feature level, respectively. Finally, the enhanced region representation and question representation are integrated adaptively to achieve answer prediction. Convincible experiments show that our method achieves improvements and outperforms prior state-of-the-art methods on two competitive benchmarks, i.e., VQA v1 and VQA v2.

Spatio-temporal visual learning for home-based monitoring

This paper introduces a novel concept for Home-based Monitoring (HM) that enables robust analysis and understanding of activities towards improved caring and safety. Spatio-Temporal Visual Learning for HM (STVL-HM) is a novel method that learns from sensor data that is jointly represented in space and time in order to robustify the HM process. We propose a hybrid model based on a Convolution Neural Network (CNN) and Transformers. The CNN first learns the visual spatial features from various sensor data. The learned visual features are then fed into the transformer, which captures temporal information by observing the sensor status at various timestamps. STVL-HM has been tested using Kinetics-400, the real use case of human activity recognition scenario for HM data. The results reveal the clear superiority of the STVL-HM compared to the recent baseline HM solutions.

Discriminative and Robust Attribute Alignment for Zero-Shot Learning

Zero-shot learning (ZSL) aims to learn models that can recognize images of semantically related unseen categories, through transferring attribute-based knowledge learned from training data of seen classes to unseen testing data. As visual attributes play a vital role in ZSL, recent embedding-based methods usually focus on learning a compatibility function between the visual representation and the class semantic attributes. While in this work, in addition to simply learning the region embedding of different semantic attributes to maintain the generalization capability of the learned model, we further consider to improve the discrimination power of the learned visual features themselves by contrastive embedding. It exploits both the class-wise and instance-wise supervision for GZSL, under the attribute guided weakly supervised representation learning framework. To further improve the robustness of the ZSL model, we also propose to train the model under the consistency regularization constraint, through taking full advantages of self-supervised signals of the image under various perturbed augmentation situations, which could make the model robust to some occluded or un-related attribute regions. Extensive experimental results demonstrate the effectiveness of the proposed ZSL method, achieving superior performances to state-of-the-art methods on three widely-used benchmark datasets, namely CUB, SUN, and AWA2. Our source code is released at https://github.com/KORIYN/CC-ZSL.

A Discriminative Cross-Aligned Variational Autoencoder for Zero-Shot Learning.

Zero-shot learning (ZSL) aims to classify unseen samples based on the relationship between the learned visual features and semantic features. Traditional ZSL methods typically capture the underlying multimodal data structures by learning an embedding function between the visual space and the semantic space with the Euclidean metric. However, these models suffer from the hubness problem and domain bias problem, which leads to unsatisfactory performance, especially in the generalized ZSL (GZSL) task. To tackle such a problem, we formulate a discriminative cross-aligned variational autoencoder (DCA-VAE) for ZSL. The proposed model effectively utilizes a modified cross-modal-alignment variational autoencoder (VAE) to transform both visual features and semantic features obtained by the discriminative cosine metric into latent features. The key to our method is that we collect principal discriminative information from visual and semantic features to construct latent features which contain the discriminative multimodal information associated with unseen samples. Finally, the proposed model DCA-VAE is validated on six benchmarks including the large dataset ImageNet, and several experimental results demonstrate the superiority of DCA-VAE over most existing embedding or generative ZSL models on the standard ZSL and the more realistic GZSL tasks.

A facial depression recognition method based on hybrid multi-head cross attention network.

Deep-learn methods based on convolutional neural networks (CNNs) have demonstrated impressive performance in depression analysis. Nevertheless, some critical challenges need to be resolved in these methods: (1) It is still difficult for CNNs to learn long-range inductive biases in the low-level feature extraction of different facial regions because of the spatial locality. (2) It is difficult for a model with only a single attention head to concentrate on various parts of the face simultaneously, leading to less sensitivity to other important facial regions associated with depression. In the case of facial depression recognition, many of the clues come from a few areas of the face simultaneously, e.g., the mouth and eyes. To address these issues, we present an end-to-end integrated framework called Hybrid Multi-head Cross Attention Network (HMHN), which includes two stages. The first stage consists of the Grid-Wise Attention block (GWA) and Deep Feature Fusion block (DFF) for the low-level visual depression feature learning. In the second stage, we obtain the global representation by encoding high-order interactions among local features with Multi-head Cross Attention block (MAB) and Attention Fusion block (AFB). We experimented on AVEC2013 and AVEC2014 depression datasets. The results of AVEC 2013 (RMSE = 7.38, MAE = 6.05) and AVEC 2014 (RMSE = 7.60, MAE = 6.01) demonstrated the efficacy of our method and outperformed most of the state-of-the-art video-based depression recognition approaches. We proposed a deep learning hybrid model for depression recognition by capturing the higher-order interactions between the depression features of multiple facial regions, which can effectively reduce the error in depression recognition and gives great potential for clinical experiments.

Genetic hyperparameter optimization with Modified Scalable-Neighbourhood Component Analysis for breast cancer prognostication

Breast cancer is common among women resulting in mortality when left untreated. Early detection is vital so that suitable treatment could assist cancer from spreading further and save people’s life. The traditional way of detection is a time-consuming process. With the evolvement of DM (Data Mining), the healthcare industry could be benefitted in predicting the disease as it permits the physicians to determine the significant attributes for diagnosis. Though, conventional techniques have used DM-based methods to identify breast cancer, they lacked in terms of prediction rate. Moreover, parametric-Softmax classifiers have been a general option by conventional works with fixed classes, particularly when huge labelled data are present during training. Nevertheless, this turns into an issue for open set cases where new classes are encountered along with few instances to learn a generalized parametric classifier. Thus, the present study aims to implement a non-parametric strategy by optimizing the embedding of a feature rather than parametric classifiers. This research utilizes Deep CNN (Deep Convolutional Neural Network) and Inception V3 for learning visual features which preserve neighbourhood outline in semantic space relying on NCA (Neighbourhood Component Analysis) criteria. Delimited by its bottleneck, the study proposes MS-NCA (Modified Scalable-Neighbourhood Component Analysis) that relies on a non-linear objective function to perform feature fusion by optimizing the distance-learning objective due to which it gains the capability of computing inner feature products without performing mapping which increases the scalability of MS-NCA. Finally, G-HPO (Genetic-Hyper-parameter Optimization) is proposed. In this case, the new stage in the algorithm simply denotes the enhancement in the length of chromosome bringing several hyperparameters into subsequent XGBoost, NB and RF models having numerous layers for identifying the normal and affected cases of breast cancer for which optimized hyper-parameter values of RF (Random Forest), NB (Naïve Bayes), and XGBoost (eXtreme Gradient Boosting) are determined. This process helps in improvising the classification rate which is confirmed through analytical results.

Mixup Feature: A Pretext Task Self-Supervised Learning Method for Enhanced Visual Feature Learning

Mixup Feature: A Pretext Task Self-Supervised Learning Method for Enhanced Visual Feature Learning

GMM-IL: Image Classification Using Incrementally Learnt, Independent Probabilistic Models for Small Sample Sizes

When deep-learning classifiers try to learn new classes through supervised learning, they exhibit catastrophic forgetting issues. In this paper we propose the Gaussian Mixture Model - Incremental Learner (GMM-IL), a novel two-stage architecture that couples unsupervised visual feature learning with supervised probabilistic models to represent each class. The key novelty of GMM-IL is that between images and labels there is a bijective connection. New classes can be incrementally learnt using a small set of annotated images with no requirement for any previous training data. This enables the incremental addition of classes to a database, that can be indexed by visual features and reasoned over based on perception. Using Gaussian Mixture Models to represent the independent classes, we outperform a benchmark of an equivalent network with a Softmax head, obtaining increased accuracy for sample sizes smaller than 12 and increased weighted F1 score for 3 imbalanced class profiles in that sample range. This novel method enables new classes to be added to a system with only access to a few annotated images of the new class.

Confounds in the Data-Comments on "Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features".

Neuroimaging experiments in general, and EEG experiments in particular, must take care to avoid confounds. A recent TPAMI paper uses data that suffers from a serious previously reported confound. We demonstrate that their new model and analysis methods do not remedy this confound, and therefore that their claims of high accuracy and neuroscience relevance are invalid.

Domain-Adversarial-Guided Siamese Network for Unsupervised Cross-Domain 3-D Object Retrieval.

Recent advances in 3-D sensors and 3-D modeling have led to the availability of massive amounts of 3-D data. It is too onerous and time consuming to manually label a plentiful of 3-D objects in real applications. In this article, we address this issue by transferring the knowledge from the existing labeled data (e.g., the annotated 2-D images or 3-D objects) to the unlabeled 3-D objects. Specifically, we propose a domain-adversarial guided siamese network (DAGSN) for unsupervised cross-domain 3-D object retrieval (CD3DOR). It is mainly composed of three key modules: 1) siamese network-based visual feature learning; 2) mutual information (MI)-based feature enhancement; and 3) conditional domain classifier-based feature adaptation. First, we design a siamese network to encode both 3-D objects and 2-D images from two domains because of its balanced accuracy and efficiency. Besides, it can guarantee the same transformation applied to both domains, which is crucial for the positive domain shift. The core issue for the retrieval task is to improve the capability of feature abstraction, but the previous CD3DOR approaches merely focus on how to eliminate the domain shift. We solve this problem by maximizing the MI between the input 3-D object or 2-D image data and the high-level feature in the second module. To eliminate the domain shift, we design a conditional domain classifier, which can exploit multiplicative interactions between the features and predictive labels, to enforce the joint alignment in both feature level and category level. Consequently, the network can generate domain-invariant yet discriminative features for both domains, which is essential for CD3DOR. Extensive experiments on two protocols, including the cross-dataset 3-D object retrieval protocol (3-D to 3-D) on PSB/NTU, and the cross-modal 3-D object retrieval protocol (2-D to 3-D) on MI3DOR-2, demonstrate that the proposed DAGSN can significantly outperform state-of-the-art CD3DOR methods.