Fine-grained Representation Research Articles

Semi-Supervised Action Recognition From Temporal Augmentation Using Curriculum Learning

Semi-supervised learning for video action recognition is a very challenging research area. Existing state-of-the-art methods perform data augmentation on the temporality of actions, which are combined with the mainstream consistency-based semi-supervised learning framework FixMatch for action recognition. However, these approaches have the following limitations: (1) data augmentation based on video clips lacks coarse-grained and fine-grained representations of actions in temporal sequences, and the models have difficulty understanding synonymous representations of actions in different motion phases. (2) Pseudo labeling selection based on the constant thresholds lacks a “make-up curriculum” for difficult actions, that results in the low utilization of unlabeled data corresponding to difficult actions. To address the above shortcomings, we propose a semi-supervised action recognition via the temporal augmentation using curriculum learning (TACL) algorithm. Compared to previous works, TACL explores different representations of the same semantics of actions in temporal sequences for video and uses the idea of curriculum learning (CL) to reduce the difficulty of the model training process. First, for different action expressions with the same semantics, we designed the temporal action augmentation (TAA) for videos to obtain coarse-grained and fine-grained action expressions based on constant-velocity and hetero-velocity methods, respectively. Second, we construct a temporal signal to constrain the model such that fine-grained action expressions containing different movement phases have the same prediction results, and achieve action consistency learning (ACL) by combining the label and pseudo-label signals. Finally, we propose action curriculum pseudo labeling (ACPL), a loosely and strictly parallel dynamic threshold evaluation algorithm for selecting and labeling unlabeled data. We evaluate TACL on three standard public datasets: UCF101, HMDB51, and Kinetics. The combined experiments show that TACL significantly improves the accuracy of models trained on a small amount of labeled data and better evaluates the learning effects for different actions.

Infrared and visible image fusion based on Multi-State contextual hidden Markov Model

In this paper, we propose a novel multi-state contextual hidden Markov model (MCHMM) in the non-subsampled Shearlet transform (NSST) domain for image fusion. The traditional two-state hidden Markov model divides the multi-scale coefficients only into large and small states, which can lead to an inaccurate statistical model and reduce the quality of the fusion result. Our method improves upon this by developing a multi-state model and a soft context variable to provide a fine-grained representation of the high-frequency subbands, resulting in improved fusion results. Additionally, the fusion of low-frequency subbands is performed on the difference of regional energy to ensure visual quality. Our experimental results on several datasets demonstrate that the proposed method outperforms other fusion methods in both subjective and objective evaluations.

Differences in speech articulatory timing and associations with pragmatic language ability in autism

BackgroundSpeech articulation difficulties have not traditionally been considered to be a feature of Autism Spectrum Disorder (ASD). In contrast, speech prosodic differences have been widely reported in ASD, and may even be expressed in subtle form among clinically unaffected first-degree relatives, representing the expression of underlying genetic liability. Some evidence has challenged this traditional dichotomy, suggesting that differences in speech articulatory mechanisms may be evident in ASD, and potentially related to perceived prosodic differences. Clinical measurement of articulatory skills has traditionally been phoneme-based, rather than by acoustic measurement of motor control. Subtle differences in articulatory/motor control, prosodic characteristics (acoustic), and pragmatic language ability (linguistic) may each be contributors to differences perceived by listeners, but the interrelationship is unclear. In this study, we examined the articulatory aspects of this relationship, in speech samples from individuals with ASD and their parents during narration. MethodUsing Speechmark® analysis, we examined articulatory landmarks, fine-grained representations of articulatory timing as series of laryngeal and vocal-tract gestures pertaining to prosodic elements crucial for conveying pragmatic information. ResultsResults revealed articulatory timing differences in individuals with ASD but not their parents, suggesting that although potentially not influenced by broader genetic liability to ASD, subtle articulatory differences may indeed be evident in ASD as the recent literature indicates. A follow-up path analysis detected associations between articulatory timing differences and prosody, and subsequently, pragmatic language ability. ConclusionTogether, results suggest a complex relationship where subtle differences in articulatory timing may result in atypical acoustic signals, and serve as a distal mechanistic contributor to pragmatic language ability ASD.

Video Based Cocktail Causal Container for Blood Pressure Classification and Blood Glucose Prediction.

With the development of modern cameras, more physiological signals can be obtained from portable devices like smartphone. Some hemodynamically based non-invasive video processing applications have been applied for blood pressure classification and blood glucose prediction objectives for unobtrusive physiological monitoring at home. However, this approach is still under development with very few publications. In this paper, we propose an end-to-end framework, entitled cocktail causal container, to fuse multiple physiological representations and to reconstruct the correlation between frequency and temporal information during multi-task learning. Cocktail causal container processes hematologic reflex information to classify blood pressure and blood glucose. Since the learning of discriminative features from video physiological representations is quite challenging, we propose a token feature fusion block to fuse the multi-view fine-grained representations to a union discrete frequency space. A causal net is used to analyze the fused higher-order information, so that the framework can be enforced to disentangle the latent factors into the related endogenous association that corresponds to down-stream fusion information to improve the semantic interpretation. Moreover, a pair-wise temporal frequency map is developed to provide valuable insights into extraction of salient photoplethysmograph (PPG) information from fingertip videos obtained by a standard smartphone camera. Extensive comparisons have been implemented for the validation of cocktail causal container using a Clinical dataset and PPG-BP benchmark. The root mean square error of 1.329±0.167 for blood glucose prediction and precision of 0.89±0.03 for blood pressure classification are achieved in Clinical dataset.

Efficient Token-Guided Image-Text Retrieval with Consistent Multimodal Contrastive Training.

Image-text retrieval is a central problem for understanding the semantic relationship between vision and language, and serves as the basis for various visual and language tasks. Most previous works either simply learn coarse-grained representations of the overall image and text, or elaborately establish the correspondence between image regions or pixels and text words. However, the close relations between coarse- and fine-grained representations for each modality are important for image-text retrieval but almost neglected. As a result, such previous works inevitably suffer from low retrieval accuracy or heavy computational cost. In this work, we address image-text retrieval from a novel perspective by combining coarse- and fine-grained representation learning into a unified framework. This framework is consistent with human cognition, as humans simultaneously pay attention to the entire sample and regional elements to understand the semantic content. To this end, a Token-Guided Dual Transformer (TGDT) architecture which consists of two homogeneous branches for image and text modalities, respectively, is proposed for image-text retrieval. The TGDT incorporates both coarse- and fine-grained retrievals into a unified framework and beneficially leverages the advantages of both retrieval approaches. A novel training objective called Consistent Multimodal Contrastive (CMC) loss is proposed accordingly to ensure the intra- and inter-modal semantic consistencies between images and texts in the common embedding space. Equipped with a two-stage inference method based on the mixed global and local cross-modal similarity, the proposed method achieves state-of-the-art retrieval performances with extremely low inference time when compared with representative recent approaches. Code is publicly available: github.com/LCFractal/TGDT.

BloodNet: An attention-based deep network for accurate, efficient, and costless bloodstain time since deposition inference.

The time since deposition (TSD) of a bloodstain, i.e., the time of a bloodstain formation is an essential piece of biological evidence in crime scene investigation. The practical usage of some existing microscopic methods (e.g., spectroscopy or RNA analysis technology) is limited, as their performance strongly relies on high-end instrumentation and/or rigorous laboratory conditions. This paper presents a practically applicable deep learning-based method (i.e., BloodNet) for efficient, accurate, and costless TSD inference from a macroscopic view, i.e., by using easily accessible bloodstain photos. To this end, we established a benchmark database containing around 50,000 photos of bloodstains with varying TSDs. Capitalizing on such a large-scale database, BloodNet adopted attention mechanisms to learn from relatively high-resolution input images the localized fine-grained feature representations that were highly discriminative between different TSD periods. Also, the visual analysis of the learned deep networks based on the Smooth Grad-CAM tool demonstrated that our BloodNet can stably capture the unique local patterns of bloodstains with specific TSDs, suggesting the efficacy of the utilized attention mechanism in learning fine-grained representations for TSD inference. As a paired study for BloodNet, we further conducted a microscopic analysis using Raman spectroscopic data and a machine learning method based on Bayesian optimization. Although the experimental results show that such a new microscopic-level approach outperformed the state-of-the-art by a large margin, its inference accuracy is significantly lower than BloodNet, which further justifies the efficacy of deep learning techniques in the challenging task of bloodstain TSD inference. Our code is publically accessible via https://github.com/shenxiaochenn/BloodNet. Our datasets and pre-trained models can be freely accessed via https://figshare.com/articles/dataset/21291825.

THE EFFECT OF SUBJECTIVE AGING ON HEALTH AND SURVIVAL: A SYSTEMATIC REVIEW OF LONGITUDINAL DATA

Abstract Longitudinal effects of subjective age on adult health and survival are well-documented (Westerhof et al., 2014). This systematic review provides an updated and expanded evaluation of the state of the field. A systematic search in PsycInfo, Web of Science, Scopus, and PubMed resulted in 103 articles: 19 articles from the 2014 meta-analysis and 84 newer articles. Sixty-five articles focused on physical health (e.g., subjective health, objective health, and health behaviors), 41 studied mental health outcomes (e.g., mental health disorders, well-being, and quality of life), and 18 articles used mortality as the outcome. Important developments during the past years include new instruments, particularly multidimensional assessment of subjective aging, and a more fine-grained representation of health outcomes, including bio indicators. A major outcome is that most studies confirmed the longitudinal effects of subjective aging on health as reported previously, but on a much broader study portfolio.

Multi-Level Knowledge-Aware Contrastive Learning Network for Personalized Recipe Recommendation

Personalized recipe recommendation is attracting more and more attention, which can help people make choices from the exploding growth of online food information. Unlike other recommendation tasks, the target of recipe recommendation is a non-atomic item, so attribute information is especially important for the representation of recipes. However, traditional collaborative filtering or content-based recipe recommendation methods tend to focus more on user–recipe interaction information and ignore higher-order semantic and structural information. Recently, graph neural networks (GNNs)-based recommendation methods provided new ideas for recipe recommendation, but there was a problem of sparsity of supervised signals caused by the long-tailed distribution of heterogeneous graph entities. How to construct high-quality representations of users and recipes becomes a new challenge for personalized recipe recommendation. In this paper, we propose a new method, a multi-level knowledge-aware contrastive learning network (MKCLN) for personalized recipe recommendation. Compared with traditional comparative learning, we design a multi-level view to satisfy the requirement of fine-grained representation of users and recipes, and use multiple knowledge-aware aggregation methods for node fusion to finally make recommendations. Specifically, the local-level includes two views, interaction view and semantic view, which mine collaborative information and semantic information for high-quality representation of nodes. The global-level learns node embedding by capturing higher-order structural information and semantic information through a network structure view. Then, a kind of self-supervised cross-view contrastive learning is invoked to make the information of multiple views collaboratively supervise each other to learn fine-grained node embeddings. Finally, the recipes that satisfy personalized preferences are recommended to users by joint training and model prediction functions. In this study, we conduct experiments on two real recipe datasets, and the experimental results demonstrate the effectiveness and advancement of MKCLN.

Representation of naturalistic food categories in the human brain

Studies of the neural response to food pictures have identified responses in food-associated regions, such as the insula, amygdala, and orbitofrontal cortex. However, beyond this domain-specific response, little is known about how finer-grained food categories are represented in the brain. Specifically, are brain responses to food pictures driven more by objective measures of food qualities or by subjective estimates of those qualities? To address this question, we asked 487 participants on Amazon Mechanical Turk to perform an “odd-one-out” triplet task to generate a quantitative measure of the similarity of 36 food images with varying known levels of fat and sugar. We used the data from this task to generate a similarity matrix of these foods, which we then analyzed using Principal Components Analysis and K-means clustering, to identify subjective categories of these foods. Using these emergent categories and a Representational Similarity Analysis approach, we analyzed the neuroimaging data of 61 healthy subjects who viewed pictures of these foods and non-food objects during fMRI. PCA of the online data indicated most of the variance was related to participants’ estimates of how processed the foods were and their sugar content, rather than objective measures of fat, sugar, or caloric content. K-means clustering of this data revealed five emergent food groupings: sweets, fats, starches, fruits, and vegetables. Multivariate hemodynamic responses to food pictures reflected this category structure – and not objective nutritional content – in food-responsive ROIs in the brain, as pattern similarity was significantly greater within vs. between categories in the mid-insula, OFC, amygdala, and ventral striatum. A searchlight analysis also reflected this food-category structure in the responses of bilateral regions of the ventral occipito-temporal cortex. These results reveal a fine-grained representation of food categories in the brain that corresponds more to subjective assessments of food qualities, than to objective food properties.

Multi-Graph Convolutional Network for Fine-Grained and Personalized POI Recommendation

With the advent of the era of rapid information expansion, the massive data backlog that exists on the Internet has led to a serious information overload problem, which makes recommendation systems a crucial part of human life. In particular, the Point-Of-Interest (POI) recommendation system has been applied to many real-life scenarios, such as life services and autonomous driving. Specifically, the goal of POI recommendation is to recommend locations that match their personalized preferences to users. In existing POI recommendation methods, people tend to pay more attention to the impact of temporal and spatial factors of POI on users, which will alleviate the problems of data sparsity and cold start in POI recommendation. However, this tends to ignore the differences among individual users, and considering only temporal and spatial attributes does not support fine-grained POI recommendations. To solve this problem, we propose a new Fine-grained POI Recommendation With Multi-Graph Convolutional Network (FP-MGCN). This model focuses on the content representation of POIs, captures users’ personalized preferences using semantic information from user comments, and learns fine-grained representations of users and POIs through the relationships between content–content, content–POI, and POI–user. FP-MGCN employs multiple embedded propagation layers and adopts information propagation mechanisms to model the higher-order connections of different POI-related relations for enhanced representation. Fine-grained POI is finally recommended to users through the three types of propagation we designed: content–content information propagation, content–POI information propagation, and POI–user information propagation. We have conducted detailed experiments on two datasets, and the results show that FP-MGCN has advanced performance and can alleviate the data sparsity problem in POI recommendation tasks.

Self-Attentive Graph Convolution Network With Latent Group Mining and Collaborative Filtering for Personalized Recommendation

The remarkable progress of machine learning has led to some state-of-the-art algorithms in personalized recommendation. Previous recommendation algorithms generally learn users’ and items’ representations based on a user-item rating matrix. However, these methods only consider a user's own preference, but ignore the influence of the user's social circles. In this paper, we propose a novel recommendation algorithm, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Self-Attentive Graph Convolution Network with Latent Group Mining and Collaborative Filtering</i> , which consists of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Latent Group Mining</i> (LGM) module, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Collaborative Embedding</i> (CE) module and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Self-Attentive Graph Convolution</i> (SAGC) module. The LGM module analyzes users’ social circles by exploring their latent groups and generates group embedding for users and items. The CE module uses a graph embedding method to provide semantic collaborative embedding for users and items. The SAGC module fuses users’ (items’) collaborative embedding and group embedding by a self-attentive graph convolution network to learn their fine-grained representations for rating prediction. We conduct experiments on different real-world datasets, which validates that our algorithm outperforms the state-of-the-art algorithms.

Towards generalizable person re-identification with a bi-stream generative model

Generalizable person re-identification (re-ID) has attracted growing attention due to its powerful adaptation capability in the unseen data domain. However, existing solutions often neglect either crossing cameras (e.g., illumination and resolution differences) or pedestrian misalignments (e.g., viewpoint and pose discrepancies), which easily leads to poor generalization capability when adapted to the new domain. In this paper, we formulate these difficulties as: 1) Camera-Camera (CC) problem, which denotes the various human appearance changes caused by different cameras; 2) Camera-Person (CP) problem, which indicates the pedestrian misalignments caused by the same identity person under different camera viewpoints or changing pose. To solve the above issues, we propose a Bi-stream Generative Model (BGM) to learn the fine-grained representations fused with camera-invariant global feature and pedestrian-aligned local feature, which contains an encoding network and two stream decoding sub-network. Guided by original pedestrian images, one stream is employed to learn a camera-invariant global feature for the CC problem via filtering cross-camera interference factors. For the CP problem, another stream learns a pedestrian-aligned local feature for pedestrian alignment using information-complete densely semantically aligned part maps. Moreover, a part-weighted loss function is presented to reduce the influence of missing parts on pedestrian alignment. Extensive experiments demonstrate that our method outperforms the state-of-the-art methods on the large-scale generalizable re-ID benchmarks, involving domain generalization setting and cross-domain setting.

IoT-based platform for automated IEQ spatio-temporal analysis in buildings using machine learning techniques

Providing accurate information about the indoor environmental quality (IEQ) conditions inside building spaces is essential to assess the comfort levels of their occupants. These values may vary inside the same space, especially for large zones, requiring many sensors to produce a fine-grained representation of the space conditions, which increases hardware installation and maintenance costs. However, sound interpolation techniques may produce accurate values with fewer input points, reducing the number of sensors needed. This work presents a platform to automate this accurate IEQ representation based on a few sensor devices placed across a large building space. A case study is presented in a research centre in Spain using 8 wall-mounted devices and an additional moving device to train a machine learning model. The system yields accurate results for estimations at positions and times never seen before by the trained model, with relative errors between 4% and 10% for the analysed variables.

Jointly Predicting Future Content in Multiple Social Media Sites Based on Multi-task Learning

User-generated contents (UGC) in social media are the direct expression of users’ interests, preferences, and opinions. User behavior prediction based on UGC has increasingly been investigated in recent years. Compared to learning a person’s behavioral patterns in each social media site separately, jointly predicting user behavior in multiple social media sites and complementing each other (cross-site user behavior prediction) can be more accurate. However, cross-site user behavior prediction based on UGC is a challenging task due to the difficulty of cross-site data sampling, the complexity of UGC modeling, and uncertainty of knowledge sharing among different sites. For these problems, we propose a Cross-Site Multi-Task (CSMT) learning method to jointly predict user behavior in multiple social media sites. CSMT mainly derives from the hierarchical attention network and multi-task learning. Using this method, the UGC in each social media site can obtain fine-grained representations in terms of words, topics, posts, hashtags, and time slices as well as the relevances among them, and prediction tasks in different social media sites can be jointly implemented and complement each other. By utilizing two cross-site datasets sampled from Weibo, Douban, Facebook, and Twitter, we validate our method’s superiority on several classification metrics compared with existing related methods.

Boundary-Oriented Binary Building Segmentation Model With Two Scheme Learning for Aerial Images

Various deep learning-based segmentation models have been developed to segment buildings in aerial images. However, the segmentation maps predicted by the conventional convolutional neural network-based methods cannot accurately determine the shapes and boundaries of segmented buildings. In this article, to improve the prediction accuracy for the boundaries and shapes of segmented buildings in aerial images, we propose the boundary-oriented binary building segmentation model (B3SM). To construct the B3SM for boundary-enhanced semantic segmentation, we present two-scheme learning (Schemes I and II), which uses the upsampling interpolation method (USIM) as a new operator and a boundary-oriented loss function (B-Loss). In Scheme I, a raw input image is processed and transformed into a presegmented map. In Scheme II, the presegmented map from Scheme I is transformed into a more fine-grained representation. To connect these two schemes, we use the USIM operator. In addition, the novel B-Loss function is implemented in B3SM to extract the features of the boundaries of buildings effectively. To perform quantitative evaluation of the shapes and boundaries of segmented buildings generated by B3SM, we develop a new metric called the boundary-oriented intersection over union (B-IoU). After evaluating the effectiveness of two-scheme learning, USIM, and B-Loss for building segmentation, we compare the performance of B3SM to those of other state-of-the-art methods using public and custom datasets. The experimental results demonstrate that the B3SM outperforms other state-of-the-art models, resulting in more accurate shapes and boundaries for segmented buildings in aerial images.

HGA: Hierarchical Feature Extraction With Graph and Attention Mechanism for Linguistic Steganalysis

Linguistic steganalysis is an important topic in the field of information security and signal processing. In recent years, linguistic steganalysis have mainly utilized deep learning techniques and make great success. But suffer from the following major disadvantages. From the perspective of model structure, current methods only extract coarse features of the text, without focusing on the fine-grained representations. In terms of application, most of the studies only focus on single hidden scene and ignore the more realistic mixed hidden scenes which are more complex and realistic. These weaknesses limit the performance and the application of linguistic steganalysis in reality. In this paper, we propose a novel linguistic steganalysis method to overcome these weaknesses. This proposed method can extract distinguished text representation which fuses hierarchical features and perform excellently in sophisticated conditions. Firstly, we adapt gated graph neural networks as the coarse graph updater to update node representations on the graph level. Then we design a fine graph updater composed of the graph attention mechanism to focus on the highlighted nodes on the node-level. Moreover, we extract the most notable feature on the dimension-level of node by the graph channel attention module. Finally, the readout function is designed to fuse the hierarchical features and make the classification. The experimental results show that our method achieves the best results compared with the previous methods in both single hidden scene and mixed hidden scenes, which prove the effectiveness of the proposed method.

MHANet: A Multiscale Hierarchical Pansharpening Method With Adaptive Optimization

In recent years, the powerful nonlinear modeling capability of convolutional neural networks has led to an increasing number of researchers focusing on deep learning-based pansharpening methods. However, due to the diversity of remote sensing image features and the limitations of the convolution operation, the existing methods are still inadequate in restoring the spatial details of complex remote sensing scenes. Therefore, in this paper, we propose a simple and effective network for pansharpening methods. Specifically, in our hierarchical feature integration architecture, a multi-scale grouping dilated block is designed to adequately capture fine-grained representations of multi-level scale features. At the same time, we propose a spatially self-attention block to adaptively improve the feature extraction process by establishing associations between features. The above blocks are connected in a hierarchical design, with selective reuse of features between layers, and a good ability to explore new levels of features while reusing low-level features. Our experiments with the GaoFen-2 satellite dataset, WorldView-2 satellite dataset, and WorldView-3 satellite dataset show that our proposed method is highly competitive with existing excellent methods in both objective indicator evaluation and subjective visual evaluation.

Adopting Attention and Cross-Layer Features for Fine-Grained Representation

Fine-grained visual classification (FGVC) is challenging task due to discriminative feature representations. The attention-based methods show great potential for FGVC, which neglect that the deeply digging inter-layer feature relations have an impact on refining feature learning. Similarly, the associating cross-layer features methods achieve significant feature enhancement, which lost the long-distance dependencies between elements. However, most of the previous researches neglect that these two methods are mutually correlated to reinforce feature learning, which are independent of each other in related models. Thus, we adopt the respective advantages of the two methods to promote fine-gained feature representations. In this paper, we propose a novel CLNET network, which effectively applies attention mechanism and cross-layer features to obtain feature representations. Specifically, CL-NET consists of 1) adopting self-attention to capture long-rang dependencies for each element, 2) associating cross-layer features to reinforce feature learning,and 3) to cover more feature regions,we integrate attention-based operations between output and input. Experiments verify that CLNET yields new state-of-the-art performance in three widely used fine-grained benchmark datasets, including CUB-200-2011, Stanford Cars and FGVC-Aircraft. The url of our code is https://github.com/dlearing/CLNET.git.

GCDB-UNet: A novel robust cloud detection approach for remote sensing images

Cloud detection is a prerequisite in many remote sensing applications, and it has been tackled through different approaches from simple thresholding to complicated deep network training. On the other hand, existing approaches are susceptible to failures while handling thin clouds, largely because of their small sizes, sparse distributions, as well as high transparency and similarity to the non-cloud background regions. This paper presents global context dense block U-Net (GCDB-UNet), a robust cloud detection network that embeds global context dense block (GCDB) into the U-Net framework and is capable of detecting thin clouds effectively. GCDB consists of two feature extraction units for addressing the challenges in thin cloud detection, namely, a non-local self-attention unit that extracts sample correlation features by aggregating the sparsely distributed thin clouds and a squeeze excitation unit that extracts channel correlated features by differentiating their importance. In addition, a dense connection scheme is designed to exploit the multi-level fine-grained representations from the two types of extracted features and a recurrent refinement module is introduced for gradual enhancement of the predicted classification map. We also created a fully annotated cloud detection MODIS dataset that consists of 1192 training images, 80 validation images and 150 test images. Extensive experiments on Landsat8, SPARCS and MODIS datasets show that the proposed GCDB-UNet achieves superior cloud detection performance as compared with state-of-the-art methods. Our created MODIS cloud detection dataset is available at https://github.com/xiachangxue/MODIS-Dataset-for-Cloud-Detection.

Two-Branch Attention Learning for Fine-Grained Class Incremental Learning

As a long-standing research area, class incremental learning (CIL) aims to effectively learn a unified classifier along with the growth of the number of classes. Due to the small inter-class variances and large intra-class variances, fine-grained visual categorization (FGVC) as a challenging visual task has not attracted enough attention in CIL. Therefore, the localization of critical regions specialized for fine-grained object recognition plays a crucial role in FGVC. Additionally, it is important to learn fine-grained features from critical regions in fine-grained CIL for the recognition of new object classes. This paper designs a network architecture named two-branch attention learning network (TBAL-Net) for fine-grained CIL. TBAL-Net can localize critical regions and learn fine-grained feature representation by a lightweight attention module. An effective training framework is proposed for fine-grained CIL by integrating TBAL-Net into an effective CIL process. This framework is tested on three popular fine-grained object datasets, including CUB-200-2011, FGVC-Aircraft, and Stanford-Car. The comparative experimental results demonstrate that the proposed framework can achieve the state-of-the-art performance on the three fine-grained object datasets.