Multi-label Classification Framework Research Articles

Cross-modality semantic guidance for multi-label image classification

Multi-label image classification aims to predict a set of labels that are present in an image. The key challenge of multi-label image classification lies in two aspects: modeling label correlations and utilizing spatial information. However, the existing approaches mainly calculate the correlation between labels according to co-occurrence among them. While the result is easily affected by the label noise and occasional co-occurrences. In addition, some works try to model the correlation between labels and spatial features, but the correlation among labels is not fully considered to model the spatial relationships among features. To address the above issues, we propose a novel cross-modality semantic guidance-based framework for multi-label image classification, namely CMSG. First, we design a semantic-guided attention (SGA) module, which applies the label correlation matrix to guide the learning of class-specific features, which implicitly models semantic correlations among labels. Second, we design a spatial-aware attention (SAA) module to extract high-level semantic-aware spatial features based on class-specific features obtained from the SGA module. The experiments carried out on three benchmark datasets demonstrate that our proposed method outperforms existing state-of-the-art algorithms on multi-label image classification.

Language-Guided Transformer for Federated Multi-Label Classification

Federated Learning (FL) is an emerging paradigm that enables multiple users to collaboratively train a robust model in a privacy-preserving manner without sharing their private data. Most existing approaches of FL only consider traditional single-label image classification, ignoring the impact when transferring the task to multi-label image classification. Nevertheless, it is still challenging for FL to deal with user heterogeneity in their local data distribution in the real-world FL scenario, and this issue becomes even more severe in multi-label image classification. Inspired by the recent success of Transformers in centralized settings, we propose a novel FL framework for multi-label classification. Since partial label correlation may be observed by local clients during training, direct aggregation of locally updated models would not produce satisfactory performances. Thus, we propose a novel FL framework of Language-Guided Transformer (FedLGT) to tackle this challenging task, which aims to exploit and transfer knowledge across different clients for learning a robust global model. Through extensive experiments on various multi-label datasets (e.g., FLAIR, MS-COCO, etc.), we show that our FedLGT is able to achieve satisfactory performance and outperforms standard FL techniques under multi-label FL scenarios. Code is available at https://github.com/Jack24658735/FedLGT.

Attention-Induced Embedding Imputation for Incomplete Multi-View Partial Multi-Label Classification

As a combination of emerging multi-view learning methods and traditional multi-label classification tasks, multi-view multi-label classification has shown broad application prospects. The diverse semantic information contained in heterogeneous data effectively enables the further development of multi-label classification. However, the widespread incompleteness problem on multi-view features and labels greatly hinders the practical application of multi-view multi-label classification. Therefore, in this paper, we propose an attention-induced missing instances imputation technique to enhance the generalization ability of the model. Different from existing incomplete multi-view completion methods, we attempt to approximate the latent features of missing instances in embedding space according to cross-view joint attention, instead of recovering missing views in kernel space or original feature space. Accordingly, multi-view completed features are dynamically weighted by the confidence derived from joint attention in the late fusion phase. In addition, we propose a multi-view multi-label classification framework based on label-semantic feature learning, utilizing the statistical weak label correlation matrix and graph attention network to guide the learning process of label-specific features. Finally, our model is compatible with missing multi-view and partial multi-label data simultaneously and extensive experiments on five datasets confirm the advancement and effectiveness of our embedding imputation method and multi-view multi-label classification model.

Enhanced multi-label cardiology diagnosis with channel-wise recurrent fusion

The timely detection of abnormal electrocardiogram (ECG) signals is vital for preventing heart disease. However, traditional automated cardiology diagnostic methods have the limitation of being unable to simultaneously identify multiple diseases in a segment of ECG signals, and do not consider the potential correlations between the 12-lead ECG signals. To address these issues, this paper presents a novel network architecture, denoted as Branched Convolution and Channel Fusion Network (BCCF-Net), designed for the multi-label diagnosis of ECG cardiology to achieve simultaneous identification of multiple diseases. Among them, the BCCF-Net incorporates the Channel-wise Recurrent Fusion (CRF) network, which is designed to enhance the ability to explore potential correlation information between 12 leads. Furthermore, the utilization of the squeeze and excitation (SE) attention mechanism maximizes the potential of the convolutional neural network (CNN). In order to efficiently capture complex patterns in space and time across various scales, the multi branch convolution (MBC) module has been developed. Through extensive experiments on two public datasets with seven subtasks, the efficacy and robustness of the proposed ECG multi-label classification framework have been comprehensively evaluated. The results demonstrate the superior performance of the BCCF-Net compared to other state-of-the-art algorithms. The developed framework holds practical application in clinical settings, allowing for the refined diagnosis of cardiac arrhythmias through ECG signal analysis.

Multi-Label Contrastive Learning for Abstract Visual Reasoning.

For a long time, the ability to solve abstract reasoning tasks was considered one of the hallmarks of human intelligence. Recent advances in the application of deep learning (DL) methods led to surpassing human abstract reasoning performance, specifically in the most popular type of such problems-Raven's progressive matrices (RPMs). While the efficacy of DL systems is indeed impressive, the way they approach the RPMs is very different from that of humans. State-of-the-art systems solving RPMs rely on massive pattern-based training and sometimes on exploiting biases in the dataset, whereas humans concentrate on the identification of the rules/concepts underlying the RPM to be solved. Motivated by this cognitive difference, this work aims at combining DL with the human way of solving RPMs. Specifically, we cast the problem of solving RPMs into a multilabel classification framework where each RPM is viewed as a multilabel data point, with labels determined by the set of abstract rules underlying the RPM. For efficient training of the system, we present a generalization of the noise contrastive estimation algorithm to the case of multilabel samples and a new sparse rule encoding scheme for RPMs. The proposed approach is evaluated on the two most popular benchmark datasets [I-RAVEN and procedurally generated matrices (PGM)] and on both of them demonstrate an advantage over the state-of-the-art results.

Transformer Driven Matching Selection Mechanism for Multi-Label Image Classification

Graph Matching has recently emerged as an attractive technique applied to various computer vision tasks. Graph Matching based multi-label image classification, in particular, treats each image as a bag of instances and reformulates the classification task as an instance-label matching selection problem, achieving state-of-the-art results on diverse benchmarks. However, the generalization and scalability of such learned model cannot be well guaranteed due to its manually predetermined graph structure and high-dimension embedding of dense connections between instances and labels. To address these limitations, in this work, we propose a novel Transformer Driven Matching Selection framework for Multi-Label Image Classification (C-TMS), where instance structural relationships, class-wise global dependencies, and the co-occurrence possibility of varying instance-label assignments are simultaneously taken into consideration in a unified and adaptive manner. Moreover, the parallelization capability of the Transformer enables efficient computation, making our model scalable to large-scale datasets. Specifically, we first represent instances and labels as nodes in the visual space and label space respectively, and then compute the hidden representation of each node in its individual space, by attending a self-attention strategy over its entire neighborhood. Subsequently, the cross-attention is adopted to excavate the correct assignments between instances and labels, and further interprets how classifying each label depends on the instances within an image and its interaction with other labels. Finally, an asymmetric focal loss is designed to optimize the instance-label correspondence, and read out image-level category confidences. Extensive experiments conducted on various multi-label image datasets demonstrate the superiority of our proposed method.

Geriatric depression and anxiety screening via deep learning using activity tracking and sleep data.

Geriatric depression and anxiety have been identified as mood disorders commonly associated with the onset of dementia. Currently, the diagnosis of geriatric depression and anxiety relies on self-reported assessments for primary screening purposes, which is uncomfortable for older adults and can be prone to misreporting. When a more precise diagnosis is needed, additional methods such as in-depth interviews or functional magnetic resonance imaging are used. However, these methods can not only be time-consuming and costly but also require systematic and cost-effective approaches. The main objective of this study was to investigate the feasibility of training an end-to-end deep learning (DL) model by directly inputting time-series activity tracking and sleep data obtained from consumer-grade wrist-worn activity trackers to identify comorbid depression and anxiety. To enhance accuracy, the input of the DL model consisted of step counts and sleep stages as time series data, along with minimal depression and anxiety assessment scores as non-time-series data. The basic structure of the DL model was designed to process mixed-input data and perform multi-label-based classification for depression and anxiety. Various DL models, including the convolutional neural network (CNN) and long short-term memory (LSTM), were applied to process the time-series data, and model selection was conducted by comparing the performances of the hyperparameters. This study achieved significant results in the multi-label classification of depression and anxiety, with a Hamming loss score of 0.0946 in the Residual Network (ResNet), by applying a mixed-input DL model based on activity tracking data. The comparison of hyper-parameter performance and the development of various DL models, such as CNN, LSTM, and ResNet contributed to the optimization of time series data processing and achievement of meaningful results. This study can be considered as the first to develop a mixed-input DL model based on activity tracking data for the multi-label identification of late-life depression and anxiety. The findings of the study demonstrate the feasibility and potential of using consumer-grade wrist-worn activity trackers in conjunction with DL models to improve the identification of comorbid mental health conditions in older adults. The study also established a multi-label classification framework for identifying the complex symptoms of depression and anxiety.

Expanding analytical capabilities in intrusion detection through ensemble-based multi-label classification

Intrusion detection systems are primarily designed to flag security breaches upon their occurrence. These systems operate under the assumption of single-label data, where each instance is assigned to a single category. However, when dealing with complex data, such as malware triage, the information provided by the IDS is limited. Consequently, additional analysis becomes necessary, leading to delays and incurring additional computational costs. Existing solutions to this problem typically merge these steps by considering a unified, but large, label set encompassing both intrusion and analytical labels, which adversely affects efficiency and performance. To address these challenges, this paper presents a novel framework for multi-label classification by employing an ensemble of sequential models that preserve the original label sets during training. Each model focuses on learning the distribution specifically related to its assigned set of labels, independent of the other label sets. To capture the relationship between different sets of labels, the parameters of each trained model initialize the subsequent model, ensuring that information from unrelated label sets does not interfere with the learning objective. Consequently, the proposed method enhances prediction performance without increasing computational complexity. To evaluate the effectiveness of our approach, we conduct experiments on a real-world dataset related to intrusion detection. The results clearly demonstrate the effectiveness of our proposed method in handling multi-label classification tasks.

A Multitask Dynamic Graph Attention Autoencoder for Imbalanced Multilabel Time Series Classification.

Graph learning is widely applied to process various complex data structures (e.g., time series) in different domains. Due to multidimensional observations and the requirement for accurate data representation, time series are usually represented in the form of multilabels. Accurately classifying multilabel time series can provide support for personalized predictions and risk assessments. It requires effectively capturing complex label relevance and overcoming imbalanced label distributions of multilabel time series. However, the existing methods are unable to model label relevance for multilabel time series or fail to fully exploit it. In addition, the existing multilabel classification balancing strategies suffer from limitations, such as disregarding label relevance, information loss, and sampling bias. This article proposes a dynamic graph attention autoencoder-based multitask (DGAAE-MT) learning framework for multilabel time series classification. It can fully and accurately model label relevance for each instance by using a dynamic graph attention-based graph autoencoder to improve multilabel classification accuracy. DGAAE-MT employs a dual-sampling strategy and cooperative training approach to improve the classification accuracy of low-frequency classes while maintaining the classification accuracy of high-frequency and mid-frequency classes. It avoids information loss and sampling bias. DGAAE-MT achieves a mean average precision (mAP) of 0.955 and an F1 score of 0.978 on a mixed medical time series dataset. It outperforms state-of-the-art works in the past two years.

Multi-Label Radar Compound Jamming Signal Recognition Using Complex-Valued CNN with Jamming Class Representation Fusion

In the complex battlefield electromagnetic environment, multiple jamming signals can enter the radar receiver simultaneously due to the development of jammers and modulation technology. The received compound jamming signals aggravate the difficulty of recognition and subsequent counter-countermeasure. In the face of strong overlapping signals and unseen jamming signal combinations, the performance of existing recognition methods usually seriously degrades. In this paper, an end-to-end multi-label classification framework combining a complex-valued convolutional neural network (CV-CNN) and jamming class representations is proposed to automatically recognize the jamming signal components of compound jamming signals. A basic multi-label CV-CNN (ML-CV-CNN) is first designed to directly process time–domain complex signals and fully retain jamming signal information. Then, the jamming class representations are generated using prototype clustering implemented by learning vector quantization, and they are fused with the ML-CV-CNN using class decoupling implemented by the attention mechanism to construct a multi-label class representation CV-CNN (ML-CR-CV-CNN), which can better learn the class-related features required for recognition. Finally, an adaptive threshold calibration is adopted to obtain optimal recognition results by multi-threshold discrimination. Simulation results verify that the proposed method has superior recognition performance, which is reflected in the strong robustness to the varying jamming-to-noise ratio (JNR) and power ratio, faster convergence speed with high JNRs, and better generalization for unseen jamming signal combinations.

Contrastively enforcing distinctiveness for multi-label image classification

Recently, as an effective way of learning latent representations, contrastive learning has been increasingly popular and successful in various domains. The success of contrastive learning in single-label classifications motivates us to leverage this learning framework to enhance distinctiveness for better performance in multi-label image classification. In this paper, we show that a direct application of contrastive learning can hardly improve in multi-label cases. Accordingly, we propose a novel framework for multi-label classification with contrastive learning in a fully supervised setting, which learns multiple representations of an image under the context of different labels. This introduces a simple yet intuitive adaption of contrastive learning into our model to boost its performance in multi-label image classification. Extensive experiments on four benchmark datasets show that the proposed framework achieves state-of-the-art performance in the comparison with the advanced methods in multi-label classification.

Ensembled Feature based Multi-Label ECG Arrhythmia Classification.

Arrhythmia is one of the major contributors to sudden cardiac death. Compared to single-label arrhythmia classification, multi-label classification provides the advantage of simultaneous detection of different cardiac ailments. Existing multi-label approaches employ deep learning models, which have several limitations, e.g., data dependency, overfitting, complexity, and lack of interpretability. In this work, we propose a multi-label classification framework with ensembled hand-crafted features added with label powerset, which better preserves the correlation between classes. Experimentation with CPSC 2018 database shows that the proposed model provides accuracy and F1-score of 88.52% and 88.45% for single-label data and 88.11% and 86.32% for multi-label data, respectively. The proposed method improves individual class performance for some important diseases compared to the existing multi-label state-of-the-art methods.Clinical relevance- Early detection of life-threatening multi-label ECG patterns consisting of atrial fibrillation, atrioventricular block, and right bundle branch block with high accuracy can reduce the sudden cardiac death rate.

Multi-task learning neural framework for categorizing sexism

Sexism, a form of oppression based on one’s sex, manifests itself in numerous ways and causes enormous suffering. In view of the growing number of experiences of sexism reported online, automatically classifying these recollections can aid in the battle against sexism by allowing gender studies researchers and government officials involved in policymaking to conduct more effective analyses. This paper investigates the 23-class fine-grained, multi-label classification of accounts (reports) of sexism. Moreover, we propose a knowledge-based cascaded multi-task framework for fine-grained multi-label sexism classification. We leverage several supporting tasks, including homogeneous and heterogeneous auxiliary tasks. Homogeneous tasks are set up without incurring any manual labeling cost and heterogeneous tasks are set up which have a high correlation with the accounts of sexism. Unlabeled accounts of sexism are utilized through unsupervised learning to help construct our multi-task setup. In addition, we incorporate a knowledge module within the framework to infuse external knowledge features into the learning process. Further, we investigate transfer learning that employs weakly labeled accounts of sexism and transfers the learning to the multi-label sexism classification. We also devise objective functions that exploit label correlations in the training data explicitly. Multiple proposed multi-task methods outperform the state-of-the-art multi-label sexism classification across five standard metrics.

Weak-PMLC: A large-scale framework for multi-label policy classification based on extremely weak supervision

With the development of e-government, multiple local governments in China are developing Internet-based open policy platforms, and these online platforms need to automatically classify policies. Current policy classification methods are usually supervised models that require massive amounts of annotated policies, which can be expensive and difficult to obtain in practice. To alleviate the burden of human experts annotating a large number of policies, we propose a large-scale framework (Weak-PMLC) for multi-label policy classification based on extremely weak supervision, which does not rely on any labeled documents and uses only the label names of each category. Specifically, we first pre-train a language model (LM) on a given dataset to extend the LM from general to domain-specific. We then utilize the domain-specific LM to generate seed words semantically related to label names. Finally, following the category-related seed words, we generate massive pseudo-labeled policies as training data for high-performance supervised models. To verify the effectiveness of our proposed method, we created two new human-labeled datasets containing about 56k and 37k policies, respectively. We also define 59 label names, which are several key feature words used to summarize all collected policies. We show that Weak-PMLC achieves around 90% F1-scores on these two datasets and improves performance by 4% over state-of-the-art weakly supervised methods. Further experiments show that the proposed Weak-PMLC is even comparable to some supervised models.

A novel soft attention-based multi-modal deep learning framework for multi-label skin lesion classification

A novel soft attention-based multi-modal deep learning framework for multi-label skin lesion classification

Graph Attention Transformer Network for Multi-label Image Classification

Multi-label classification aims to recognize multiple objects or attributes from images. The key to solving this issue relies on effectively characterizing the inter-label correlations or dependencies, which bring the prevailing graph neural network. However, current methods often use the co-occurrence probability of labels based on the training set as the adjacency matrix to model this correlation, which is greatly limited by the dataset and affects the model’s generalization ability. This article proposes a Graph Attention Transformer Network, a general framework for multi-label image classification by mining rich and effective label correlation. First, we use the cosine similarity value of the pre-trained label word embedding as the initial correlation matrix, which can represent richer semantic information than the co-occurrence one. Subsequently, we propose the graph attention transformer layer to transfer this adjacency matrix to adapt to the current domain. Our extensive experiments have demonstrated that our proposed methods can achieve highly competitive performance on three datasets.

Aligning Image Semantics and Label Concepts for Image Multi-Label Classification

Image multi-label classification task is mainly to correctly predict multiple object categories in the images. To capture the correlation between labels, graph convolution network based methods have to manually count the label co-occurrence probability from training data to construct a pre-defined graph as the input of graph network, which is inflexible and may degrade model generalizability. Moreover, most of the current methods cannot effectively align the learned salient object features with the label concepts, so that the predicted results of model may not be consistent with the image content. Therefore, how to learn the salient semantic features of images and capture the correlation between labels, and then effectively align them is one of the key to improve the performance of image multi-label classification task. To this end, we propose a novel image multi-label classification framework which aims to align I mage S emantics with L abel C oncepts ( ISLC ). Specifically, we propose a residual encoder to learn salient object features in the images, and exploit the self-attention layer in aligned decoder to automatically capture the correlation between labels. Then, we leverage the cross-attention layers in aligned decoder to align image semantic features with label concepts, so as to make the labels predicted by model more consistent with image content. Finally, the output features of the last layer of residual encoder and aligned decoder are fused to obtain the final output feature for classification. The proposed ISLC model achieves good performance on various prevalent multi-label image datasets such as MS-COCO 2014, PASCAL VOC 2007, VG-500, and NUS-WIDE with 87.2%, 96.9%, 39.4%, and 64.2%, respectively.

Characterization of Moving Sound Sources Direction-of-Arrival Estimation Using Different Deep Learning Architectures

Sound source localization is an important task for several applications and the use of deep learning for this task has recently become a popular research topic. While a number of previous works has focused on static sound sources, in this paper we evaluate the performance of a deep learning classification system for localization of moving sound sources. In particular, we evaluate the effect of key parameters at the levels of feature extraction (e.g., STFT parameters) and model training (e.g., neural network architectures). We evaluate the performance of different settings in terms of precision and F-score, in a multi-class multi-label classification framework. In our previous work for localization of moving sound sources, we investigated feedforward neural networks under different acoustic conditions and STFT parameters, and showed that the presence of some reverberation in the training dataset can help in achieving better detection for the direction of arrival of the sources. In this paper, we extend the work to show that (1) window size does not affect the performance of static sources but highly affects the performance of moving sources, (2) sequence length has a significant effect on the performance of recurrent architectures, and (3) a temporal convolutional neural network can outperform both recurrent and feedforward networks for moving sound sources.

Multi-label disaster text classification via supervised contrastive learning for social media data

Social media is a crucial way to release information in a timely manner during disasters, which provides help to people who suffer from disasters. In this disaster information, each message may contain multiple labels. The single-label classification method cannot be adapted to a multi-label classification problem. In addition, there is a lack of representative baseline datasets due to the differences between disaster data sources. We propose a supervised contrastive learning based multi-label classification framework as a general framework for data processing and model training. Specifically, it learns features of disaster data and identifies different disaster type information, then trains a multi-label classification model of disaster texts in different dimensions. The results of empirical studies on three disaster text classification datasets show that this method can effectively improve the accuracy of the model and the representation ability of semantic information.

Systemic distortion analysis with deep distortion directed image quality assessment models

An image processing system may bring distortions to the images going through it due to its inherent attributes or defects. Identifying the distortion types in a system is important to solve the system defects and improve its service quality. In this paper, we propose a novel algorithm to analyse the systemic distortions with multiple deep distortion directed image quality assessment models. Specifically, We pre-train several image quality assessment models that target to different distortions respectively. We use multiple images as anchor to test the system. We use the pre-trained IQA models to evaluate the anchor images that have gone through the system, and explore the features about the relationship between distortion types and quality scores. To analyse multiple distortions in a system, we embed dictionary learning into the multi-label classification framework. The experimental results demonstrate that our algorithm has good effectiveness, robustness and extensibility.