Long-tailed Class Research Articles

Semi-supervised learning advances species recognition for aquatic biodiversity monitoring

Aquatic biodiversity monitoring relies on species recognition from images. While deep learning (DL) streamlines the recognition process, the performance of these method is closely linked to the large-scale labeled datasets, necessitating manual processing with expert knowledge and consume substantial time, labor, and financial resources. Semi-supervised learning (SSL) offers a promising avenue to improve the performance of DL models by utilizing the extensive unlabeled samples. However, the complex collection environments and the long-tailed class imbalance of aquatic species make SSL difficult to implement effectively. To address these challenges in aquatic species recognition within the SSL scheme, we propose a Wavelet Fusion Network and the Consistency Equilibrium Loss function. The former mitigates the influence of data collection environment by fusing image information at different frequencies decomposed through wavelet transform. The latter improves the SSL scheme by refining the consistency loss function and adaptively adjusting the margin for each class. Extensive experiments are conducted on the large-scale FishNet dataset. As expected, our method improves the existing SSL scheme by up to 9.34% in overall classification accuracy. With the accumulation of image data, the improved SSL method with limited labeled data, shows the potential to advance species recognition for aquatic biodiversity monitoring and conservation.

EAT: Towards Long-Tailed Out-of-Distribution Detection

Despite recent advancements in out-of-distribution (OOD) detection, most current studies assume a class-balanced in-distribution training dataset, which is rarely the case in real-world scenarios. This paper addresses the challenging task of long-tailed OOD detection, where the in-distribution data follows a long-tailed class distribution. The main difficulty lies in distinguishing OOD data from samples belonging to the tail classes, as the ability of a classifier to detect OOD instances is not strongly correlated with its accuracy on the in-distribution classes. To overcome this issue, we propose two simple ideas: (1) Expanding the in-distribution class space by introducing multiple abstention classes. This approach allows us to build a detector with clear decision boundaries by training on OOD data using virtual labels. (2) Augmenting the context-limited tail classes by overlaying images onto the context-rich OOD data. This technique encourages the model to pay more attention to the discriminative features of the tail classes. We provide a clue for separating in-distribution and OOD data by analyzing gradient noise. Through extensive experiments, we demonstrate that our method outperforms the current state-of-the-art on various benchmark datasets. Moreover, our method can be used as an add-on for existing long-tail learning approaches, significantly enhancing their OOD detection performance. Code is available at: https://github.com/Stomach-ache/Long-Tailed-OOD-Detection.

Sharpness-Aware Model-Agnostic Long-Tailed Domain Generalization

Domain Generalization (DG) aims to improve the generalization ability of models trained on a specific group of source domains, enabling them to perform well on new, unseen target domains. Recent studies have shown that methods that converge to smooth optima can enhance the generalization performance of supervised learning tasks such as classification. In this study, we examine the impact of smoothness-enhancing formulations on domain adversarial training, which combines task loss and adversarial loss objectives. Our approach leverages the fact that converging to a smooth minimum with respect to task loss can stabilize the task loss and lead to better performance on unseen domains. Furthermore, we recognize that the distribution of objects in the real world often follows a long-tailed class distribution, resulting in a mismatch between machine learning models and our expectations of their performance on all classes of datasets with long-tailed class distributions. To address this issue, we consider the domain generalization problem from the perspective of the long-tail distribution and propose using the maximum square loss to balance different classes which can improve model generalizability. Our method's effectiveness is demonstrated through comparisons with state-of-the-art methods on various domain generalization datasets. Code: https://github.com/bamboosir920/SAMALTDG.

Learning Multi-Expert Distribution Calibration for Long-Tailed Video Classification

Most existing state-of-the-art video classification methods assume that the training data obey a uniform distribution. However, video data in the real world typically exhibit an imbalanced long-tailed class distribution, resulting in a model bias towards head class and relatively low performance on tail class. While the current long-tailed classification methods usually focus on image classification, adapting them to video data is not a trivial extension. We propose an end-to-end multi-expert distribution calibration method to address these challenges based on two-level distribution information. The method jointly considers the distribution of samples in each class (intra-class distribution) and the overall distribution of diverse data (inter-class distribution) to solve the issue of imbalanced data under long-tailed distribution. By modeling the two-level distribution information, the model can jointly consider the head classes and the tail classes and significantly transfer the knowledge from the head classes to improve the performance of the tail classes. Extensive experiments verify that our method achieves state-of-the-art performance on the long-tailed video classification task.

Adaptive Feature Learning for Unbiased Scene Graph Generation.

Scene Graph Generation (SGG) aims to detect all objects and identify their pairwise relationships in the scene. Recently, tremendous progress has been made in exploring better context relationship representations. Previous work mainly focuses on contextual information aggregation and uses de-biasing strategies on samples to eliminate the preference for head predicates. However, there remain challenges caused by indeterminate feature training. Overlooking the label confusion problem in feature training easily results in a messy feature distribution among the confused categories, thereby affecting the prediction of predicates. To alleviate the aforementioned problem, in this paper, we focus on enhancing predicate representation learning. Firstly, we propose a novel Adaptive Message Passing (AMP) network to dynamically conduct information propagation among neighbors. AMP provides discriminating representations for neighbor nodes under the view of de-noising and adaptive aggregation. Furthermore, we construct a feature-assisted training paradigm alongside the predicate classification branch, guiding predicate feature learning to the corresponding feature space. Moreover, to alleviate biased prediction caused by the long-tailed class distribution and the interference of confused labels, we design a Bi-level Curriculum learning scheme (BiC). The BiC separately considers the training from the feature learning and de-biasing levels, preserving discriminating representations of different predicates while resisting biased predictions. Results on multiple SGG datasets show that our proposed method AMP-BiC has superior comprehensive performance, demonstrating its effectiveness.

A few-shot disease diagnosis decision making model based on meta-learning for general practice

BackgroundDiagnostic errors have become the biggest threat to the safety of patients in primary health care. General practitioners, as the “gatekeepers” of primary health care, have a responsibility to accurately diagnose patients. However, many general practitioners have insufficient knowledge and clinical experience in some diseases. Clinical decision making tools need to be developed to effectively improve the diagnostic process in primary health care. The long-tailed class distributions of medical datasets are challenging for many popular decision making models based on deep learning, which have difficulty predicting few-shot diseases. Meta-learning is a new strategy for solving few-shot problems. Methods and materialsIn this study, a few-shot disease diagnosis decision making model based on a model-agnostic meta-learning algorithm (FSDD-MAML) is proposed. The MAML algorithm is applied in a knowledge graph-based disease diagnosis model to find the optimal model parameters. Moreover, FSDD-MAML can learn learning rates for all modules of the knowledge graph-based disease diagnosis model. For n-way, k-shot learning tasks, the inner loop of FSDD-MAML performs multiple gradient update steps to learn internal features in disease classification tasks using n×k examples, and the outer loop of FSDD-MAML optimizes the meta-objective to find the associated optimal parameters and learning rates. FSDD-MAML is compared with the original knowledge graph-based disease diagnosis model and other meta-learning algorithms based on an abdominal disease dataset. ResultMeta-learning algorithms can greatly improve the performance of models in top-1 evaluation compared with top-3, top-5, and top-10 evaluations. The proposed decision making model FSDD-MAML outperforms all the other models, with a precision@1 of 90.02 %. We achieve state-of-the-art performance in the diagnosis of all diseases, and the prediction performance for few-shot diseases is greatly improved. For the two groups with the fewest examples of diseases, FSDD-MAML achieves relative increases in precision@1 of 29.13 % and 21.63 % compared with the original knowledge graph-based disease diagnosis model. In addition, we analyze the reasoning process of several few-shot disease predictions and provide an explanation for the results. ConclusionThe decision making model based on meta-learning proposed in this paper can support the rapid diagnosis of diseases in general practice and is especially capable of helping general practitioners diagnose few-shot diseases. This study is of profound significance for the exploration and application of meta-learning to few-shot disease assessment in general practice.

DBN-Mix: Training dual branch network using bilateral mixup augmentation for long-tailed visual recognition

There is growing interest in the challenging visual perception task of learning from long-tailed class distributions. The extreme class imbalance in the training dataset biases the model to prefer recognizing majority class data over minority class data. Furthermore, the lack of diversity in minority class samples makes it difficult to find a good representation. In this paper, we propose an effective data augmentation method, referred to as bilateral mixup augmentation, which can improve the performance of long-tailed visual recognition. The bilateral mixup augmentation combines two samples generated by a uniform sampler and a re-balanced sampler and augments the training dataset to enhance the representation learning for minority classes. We also reduce the classifier bias using class-wise temperature scaling, which scales the logits differently per class in the training phase. We apply both ideas to the dual-branch network (DBN) framework, presenting a new model, named dual-branch network with bilateral mixup (DBN-Mix). Experiments on popular long-tailed visual recognition datasets show that DBN-Mix improves performance significantly over baseline and that the proposed method achieves state-of-the-art performance in some categories of benchmarks.

Deep Long-Tailed Learning: A Survey.

Deep long-tailed learning, one of the most challenging problems in visual recognition, aims to train well-performing deep models from a large number of images that follow a long-tailed class distribution. In the last decade, deep learning has emerged as a powerful recognition model for learning high-quality image representations and has led to remarkable breakthroughs in generic visual recognition. However, long-tailed class imbalance, a common problem in practical visual recognition tasks, often limits the practicality of deep network based recognition models in real-world applications, since they can be easily biased towards dominant classes and perform poorly on tail classes. To address this problem, a large number of studies have been conducted in recent years, making promising progress in the field of deep long-tailed learning. Considering the rapid evolution of this field, this article aims to provide a comprehensive survey on recent advances in deep long-tailed learning. To be specific, we group existing deep long-tailed learning studies into three main categories (i.e., class re-balancing, information augmentation and module improvement), and review these methods following this taxonomy in detail. Afterward, we empirically analyze several state-of-the-art methods by evaluating to what extent they address the issue of class imbalance via a newly proposed evaluation metric, i.e., relative accuracy. We conclude the survey by highlighting important applications of deep long-tailed learning and identifying several promising directions for future research.

Myo1f has an essential role in γδT intraepithelial lymphocyte adhesion and migration.

γδT intraepithelial lymphocyte represents up to 60% of the small intestine intraepithelial compartment. They are highly migrating cells and constantly interact with the epithelial cell layer and lamina propria cells. This migratory phenotype is related to the homeostasis of the small intestine, the control of bacterial and parasitic infections, and the epithelial shedding induced by LPS. Here, we demonstrate that Myo1f participates in the adhesion and migration of intraepithelial lymphocytes. Using long-tailed class I myosins KO mice, we identified the requirement of Myo1f for their migration to the small intestine intraepithelial compartment. The absence of Myo1f affects intraepithelial lymphocytes' homing due to reduced CCR9 and α4β7 surface expression. In vitro, we confirm that adhesion to integrin ligands and CCL25-dependent and independent migration of intraepithelial lymphocytes are Myo1f-dependent. Mechanistically, Myo1f deficiency prevents correct chemokine receptor and integrin polarization, leading to reduced tyrosine phosphorylation which could impact in signal transduction. Overall, we demonstrate that Myo1f has an essential role in the adhesion and migration in γδT intraepithelial lymphocytes.

Identifying bird species by their calls in Soundscapes

In many real data science problems, it is common to encounter a domain mismatch between the training and testing datasets, which means that solutions designed for one may not transfer well to the other due to their differences. An example of such was in the BirdCLEF2021 Kaggle competition, where participants had to identify all bird species that could be heard in audio recordings. Thus, multi-label classifiers, capable of coping with domain mismatch, were required. In addition, classifiers needed to be resilient to a long-tailed (imbalanced) class distribution and weak labels. Throughout the competition, a diverse range of solutions based on convolutional neural networks were proposed. However, it is unclear how different solution components contribute to overall performance. In this work, we contextualise the problem with respect to the previously existing literature, analysing and discussing the choices made by the different participants. We also propose a modular solution architecture to empirically quantify the effects of different architectures. The results of this study provide insights into which components worked well for this challenge.

MUS-CDB: Mixed Uncertainty Sampling With Class Distribution Balancing for Active Annotation in Aerial Object Detection

Recent aerial object detection models rely on a large amount of labeled training data, which requires unaffordable manual labeling costs in large aerial scenes with dense objects. Active learning effectively reduces the data labeling cost by selectively querying the informative and representative unlabelled samples. However, existing active learning methods are mainly with class-balanced settings and image-based querying for generic object detection tasks, which are less applicable to aerial object detection scenarios due to the long-tailed class distribution and dense small objects in aerial scenes. In this paper, we propose a novel active learning method for cost-effective aerial object detection. Specifically, both object-level and image-level informativeness are considered in the object selection to refrain from redundant and myopic querying. Besides, an easy-to-use class-balancing criterion is incorporated to favor the minority objects to alleviate the long-tailed class distribution problem in model training. We further devise a training loss to mine the latent knowledge in the unlabeled image regions. Extensive experiments are conducted on the DOTA-v1.0 and DOTA-v2.0 benchmarks to validate the effectiveness of the proposed method. For the ReDet, KLD, and SASM detectors on the DOTA-v2.0 dataset, the results show that our proposed MUS-CDB method can save nearly 75% of the labeling cost while achieving comparable performance to other active learning methods in terms of mAP. Code is publicly online.

Establishing a citywide street tree inventory with street view images and computer vision techniques

Trees in urban areas have diverse ecological, social, and health benefits. The establishment of up-to-date and accurate street-tree inventories that list the species and locations of individual street trees is critical to urban tree management and tree-planting campaigns. However, street-tree inventories are incomplete or lacking altogether in most cities. This is partly because conventional field assessment is laborious or expensive. In this study, we developed and validated a novel and cost-effective method to establish a city-wide tree inventory based on computer vision and freely available street view images (SVIs). Tree information such as species, height, crown diameter, and geographical coordinates at the individual tree level can be assessed. Based on an object detection model, we adopted a species-based loss function to address the challenges of long-tailed class distribution of species, which is caused by imbalance among sample size of different tree species and can lead to poor performance of the model. Compared with other research in urban tree species recognition, the modified model shows a higher accuracy. In order to calculate quantitative features of street trees, we employed a deep learning algorithm, which is pretrained on stereo dataset and validated on Google Street View images, to estimate the depth of each pixel in SVIs. Furthermore, as a demonstration, we established the citywide tree inventory and conducted tree diversity analysis for Jinan, China. Compared with new developed area, the old town has more street trees and more diverse tree species which can improve biodiversity and walkability. We also found that plane trees, which can cause allergic reactions, are dominant in northern new developed urban area.

Real-Time Activities of Daily Living Recognition Under Long-Tailed Class Distribution

Real-time/online Activities of Daily Living (ADL) recognition is an enabling technology for various ubiquitous computing applications. Most of the existing ADL recognition systems consider class balanced data, while the real-life datasets are significantly imbalanced because they follow the long-tailed class distribution. Such imbalanced training data can result in a biased system optimized to favor the majority classes while failing to recognize the minority classes. This paper proposes an online system that recognizes the ADL while considering the long-tailed class distribution. The system first generates hand-crafted and high-level features by using conventional learning and deep learning, which cover the advantage of both technologies to recognize the ADL. Next, the system uses an ensemble technique to concatenate the generated features. Finally, the system minimizes a loss function, which is a linear combination of focal loss for addressing the long-tailed class distribution problem and center loss for enhancing the discriminative power of the deeply learned features. We conduct several experiments on real-life long-tailed datasets to verify the accuracy of the system.

Long-Tailed Graph Representation Learning via Dual Cost-Sensitive Graph Convolutional Network

Deep learning algorithms have seen a massive rise in popularity for remote sensing over the past few years. Recently, studies on applying deep learning techniques to graph data in remote sensing (e.g., public transport networks) have been conducted. In graph node classification tasks, traditional graph neural network (GNN) models assume that different types of misclassifications have an equal loss and thus seek to maximize the posterior probability of the sample nodes under labeled classes. The graph data used in realistic scenarios tend to follow unbalanced long-tailed class distributions, where a few majority classes contain most of the vertices and the minority classes contain only a small number of nodes, making it difficult for the GNN to accurately predict the minority class samples owing to the classification tendency of the majority classes. In this paper, we propose a dual cost-sensitive graph convolutional network (DCSGCN) model. The DCSGCN is a two-tower model containing two subnetworks that compute the posterior probability and the misclassification cost. The model uses the cost as ”complementary information” in a prediction to correct the posterior probability under the perspective of minimal risk. Furthermore, we propose a new method for computing the node cost labels based on topological graph information and the node class distribution. The results of extensive experiments demonstrate that DCSGCN outperformed other competitive baselines on different real-world imbalanced long-tailed graphs.

Paying attention for adjacent areas: Learning discriminative features for large-scale 3D scene segmentation

Despite recent improvements in analyzing large-scale 3D point clouds, several problems still exist: (a) segmentation models suffer from intra-class inconsistency and inter-class indistinction; (b) the existing methods ignore the inherent long-tailed class distribution of real-world 3D data. These problems result in unsatisfactory semantic segmentation predictions, especially in object adjacent areas. To handle these problems, this paper proposes a novel Adjacent areas Refinement Network (ARNet). Specifically, an Adjacent areas Refinement (AR) module is designed, which consists of two parallel attention blocks. Besides, our proposed attention blocks can process a large number of points (N∼105) with a slight increase in the computational complexity and time cost. Additionally, to deal with the inherent long-tailed class distribution in real-world 3D data, imbalance adjustment loss and occupancy regression loss are introduced. Based on this, the proposed network can handle the classification of both majority and minority classes, which is essential in distinguishing the ambiguous parts in large-scale 3D scenes. The proposed AR module and the loss functions can be easily integrated into the cutting-edge backbone networks, contributing to better performance in modeling semantic inter-dependencies and significantly improving the accuracy of the state-of-the-art semantic segmentation methods on indoor and outdoor scenes.

DropLoss for Long-Tail Instance Segmentation

Long-tailed class distributions are prevalent among the practical applications of object detection and instance segmentation. Prior work in long-tail instance segmentation addresses the imbalance of losses between rare and frequent categories by reducing the penalty for a model incorrectly predicting a rare class label. We demonstrate that the rare categories are heavily suppressed by correct background predictions, which reduces the probability for all foreground categories with equal weight. Due to the relative infrequency of rare categories, this leads to an imbalance that biases towards predicting more frequent categories. Based on this insight, we develop DropLoss -- a novel adaptive loss to compensate for this imbalance without a trade-off between rare and frequent categories. With this loss, we show state-of-the-art mAP across rare, common, and frequent categories on the LVIS dataset. Codes are available at https://github.com/timy90022/DropLoss.

Asymptotic estimates for finite-time ruin probability in a discrete-time risk model with dependence structures and CMC simulations

Consider a discrete-time risk model with dependence structures, where claim sizes are assumed to follow a one-sided linear process whose innovations further obey a so-called bivariate upper tail independence. The stochastic discount factors follow a stationary causal process. Then, the insurer is said to be exposed to a stochastic economic environment that contains two kinds of risks, i.e. the insurance risk and financial risk. The two kinds of risks form a sequence of independent and identically distributed random pairs which are copies of a random pair with a common bivariate Sarmanov dependent distribution. When the distributions of the innovations belong to the intersection of the dominated-variation class and the long-tailed class, we derive some asymptotic formulas for the finite-time ruin probability. We also get conservative asymptotic bounds when the distributions of the innovations belong to the regular variation class. Finally, we verify our results through a Crude Monte Carlo simulation.

Myo1e modulates the recruitment of activated B cells to inguinal lymph nodes.

The inclusion of lymphocytes in high endothelial venules and their migration to the lymph nodes are critical steps in the immune response. Cell migration is regulated by the actin cytoskeleton and myosins. Myo1e is a long-tailed class I myosin and is highly expressed in B cells, which have not been studied in the context of cell migration. By using intravital microscopy in an in vivo model and performing in vitro experiments, we studied the relevance of Myo1e for the adhesion and inclusion of activated B cells in high endothelial venules. We observed reduced expression of integrins and F-actin in the membrane protrusions of B lymphocytes, which might be explained by deficiencies in vesicular trafficking. Interestingly, the lack of Myo1e reduced the phosphorylation of focal adhesion kinase (FAK; also known as PTK2), AKT (also known as AKT1) and RAC-1, disturbing the FAK-PI3K-RAC-1 signaling pathway. Taken together, our results indicate a critical role of Myo1e in the mechanism of B-cell adhesion and migration.

Human myosin 1e tail but not motor domain replaces fission yeast Myo1 domains to support myosin-I function during endocytosis

In both unicellular and multicellular organisms, long-tailed class I myosins function in clathrin-mediated endocytosis. Myosin 1e (Myo1e) in vertebrates and Myo1 in fission yeast have similar domain organization, yet whether these proteins or their individual protein domains are functionally interchangeable remains unknown. In an effort to assess functional conservation of class I myosins, we tested whether human Myo1e could replace Myo1 in fission yeast Schizosaccharomyces pombe and found that it was unable to substitute for yeast Myo1. To determine if any individual protein domain is responsible for the inability of Myo1e to function in yeast, we created human-yeast myosin-I chimeras. By functionally testing these chimeric myosins in vivo, we concluded that the Myo1e motor domain is unable to function in yeast, even when combined with the yeast Myo1 tail and a full complement of yeast regulatory light chains. Conversely, the Myo1e tail, when attached to the yeast Myo1 motor domain, supports localization to endocytic actin patches and partially rescues the endocytosis defect in myo1Δ cells. Further dissection showed that both the TH1 and TH2-SH3 domains in the human Myo1e tail are required for localization and function of chimeric myosin-I at endocytic sites. Overall, this study provides insights into the role of individual myosin-I domains, expands the utility of fission yeast as a simple model system to study the effects of disease-associated MYO1E mutations, and supports a model of co-evolution between a myosin motor and its actin track.

Class I myosin Myo1e regulates TLR4-triggered macrophage spreading, chemokine release, and antigen presentation via MHC class II.

TLR-mediated recognition of microbial danger induces substantial changes in macrophage migration, adherence, and phagocytosis. Recently, we described the LPS-regulated phosphorylation of many cytoskeleton-associated proteins by phosphoproteomics. The functional role of these cytoskeletal and motor proteins in innate immune cell responses is largely unexplored. Here, we first identified both long-tailed class I myosins Myo1e and Myo1f as important contributors to LPS-triggered macrophage spreading. Mouse bone marrow-derived macrophages and DCs deficient in Myo1e selectively secreted increased amounts of the chemokine CCL2. In addition, the cell surface expression of MHC class II (MHC-II) on both cell types was reduced in the absence of Myo1e. However, transcriptional changes in CCL2 and MHC-II were not observed in the absence of Myo1e, indicating that Myo1e regulates specific intracellular transport processes. The capacity of macrophages and DCs lacking Myo1e to stimulate antigen-specific CD4(+) T-cell proliferation was impaired, consistent with the reduced MHC-II surface protein levels. Surprisingly, in Myo1e-deficient DCs, the proteolytic cleavage of endocytosed antigen was also increased. Together, our results provide evidence for a non-redundant function of the motor protein Myo1e in the regulation of TLR4-controlled, cytoskeleton-associated functional properties of macrophages and DCs, and in induction of a full MHC-II-restricted adaptive immune response.