Multi-label Classification Task Research Articles

Advancing language models through domain knowledge integration: a comprehensive approach to training, evaluation, and optimization of social scientific neural word embeddings

This article proposes a comprehensive strategy for training, evaluating, and optimizing domain-specific word2vec-based word embeddings, using social science literature as an example. Our primary objectives are: (1) to train the embeddings utilizing a corpus of social science text, (2) to test their performance against domain-unspecific embeddings using our developed intrinsic and extrinsic evaluation strategy, and (3) to enhance their performance even further by using domain knowledge. As an integral part of this approach, we present SociRel-461, a domain-knowledge dictionary designed for the intrinsic evaluation and subsequent refinement of social science word embeddings. Using a dataset of 100,000 full-text scientific articles in sociology, we train multiple vector space models, which we then benchmark against a larger, pre-trained general language embedding model as part of our extrinsic evaluation. Furthermore, we developed a transfer learning multi-label classification task for extrinsic evaluation. Our findings reveal that domain-specific embeddings outperform their domain-unspecific counterparts in both intrinsic and extrinsic evaluations. We also investigated the retrofitting post-processing method to enhance domain-unspecific embeddings with the domain knowledge embedded in SociRel-461. While retrofitting does not enhance our domain-specific vector space models, it significantly improves the performance of the domain-unspecific embeddings. This highlights the potential of retrofitting for the transfer of domain knowledge to domain-unspecific embeddings. Our results emphasize the importance of utilizing domain-specific word embeddings for better performance in domain specific transfer learning tasks, as they outperform conventional embeddings trained on everyday language.

Imbalance-Robust Multi-Label Self-Adjusting kNN

In the task of multi-label classification in data streams, instances arriving in real time need to be associated with multiple labels simultaneously. Various methods based on the k Nearest Neighbors algorithm have been proposed to address this task. However, these methods face limitations when dealing with imbalanced data streams, a problem that has received limited attention in existing works. To approach this gap, this paper introduces the Imbalance-Robust Multi-Label Self-Adjusting kNN (IRMLSAkNN), designed to tackle multi-label imbalanced data streams. IRMLSAkNN’s strength relies on maintaining relevant instances with imbalance labels by using a discarding mechanism that considers the imbalance ratio per label. On the other hand, it evaluates subwindows with an imbalance-aware measure to discard older instances that are lacking performance. We conducted statistical experiments on 32 benchmark data streams, evaluating IRMLSAkNN against eight multi-label classification algorithms using common accuracy-aware and imbalance-aware measures. The obtained results demonstrate that IRMLSAkNN consistently outperforms these algorithms in terms of predictive capacity and time cost across various levels of imbalance.

Comparison between parameter-efficient techniques and full fine-tuning: A case study on multilingual news article classification.

Adapters and Low-Rank Adaptation (LoRA) are parameter-efficient fine-tuning techniques designed to make the training of language models more efficient. Previous results demonstrated that these methods can even improve performance on some classification tasks. This paper complements existing research by investigating how these techniques influence classification performance and computation costs compared to full fine-tuning. We focus specifically on multilingual text classification tasks (genre, framing, and persuasion techniques detection; with different input lengths, number of predicted classes and classification difficulty), some of which have limited training data. In addition, we conduct in-depth analyses of their efficacy across different training scenarios (training on the original multilingual data; on the translations into English; and on a subset of English-only data) and different languages. Our findings provide valuable insights into the applicability of parameter-efficient fine-tuning techniques, particularly for multilabel classification and non-parallel multilingual tasks which are aimed at analysing input texts of varying length.

Business text classification with imbalanced data and moderately large label spaces for digital transformation

Digital transformation refers to an organization’s use of digital technology to improve its products, services, and operations, aligning them with evolving business requirements. To demonstrate this transformative process, we present a real-life case study where a company seeks to automate the classification of their textual data rather than relying on manual methods. Transitioning to automated classification involves deploying machine learning models, which rely on pre-labeled datasets for training and making predictions on new data. However, upon receiving the dataset from the company, we faced challenges due to the imbalanced distribution of labels and moderately large label spaces. To tackle text classification with such a business dataset, we evaluated four distinct methods for multi-label text classification: fine-tuned Bidirectional Encoder Representations from Transformers (BERT), Binary Relevance, Classifier Chains, and Label Powerset. The results revealed that fine-tuned BERT significantly outperformed the other methods across key metrics like Accuracy, F1-score, Precision, and Recall. Binary Relevance also displayed competence in handling the dataset effectively, while Classifier Chains and Label Powerset exhibited comparatively less impressive performance. These findings highlight the remarkable effectiveness of fine-tuned BERT model and the Binary Relevance classifier in multi-label text classification tasks, particularly when dealing with imbalanced training datasets and moderately large label spaces. This positions them as valuable assets for businesses aiming to automate data classification in the digital transformation era.

Structure Mapping Generative Adversarial Network for Multi-View Information Mapping Pattern Mining.

Multi-view learning is dedicated to integrating information from different views and improving the generalization performance of models. However, in most current works, learning under different views has significant independency, overlooking common information mapping patterns that exist between these views. This paper proposes a Structure Mapping Generative adversarial network (SM-GAN) framework, which utilizes the consistency and complementarity of multi-view data from the innovative perspective of information mapping. Specifically, based on network-structured multi-view data, a structural information mapping model is proposed to capture hierarchical interaction patterns among views. Subsequently, three different types of graph convolutional operations are designed in SM-GAN based on the model. Compared with regular GAN, we add a structural information mapping module between the encoder and decoder wthin the generator, completing the structural information mapping from the micro-view to the macro-view. This paper conducted sufficient validation experiments using public imaging genetics data in Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset. It is shown that SM-GAN outperforms baseline and advanced methods in multi-label classification and evolution prediction tasks.

Mutual Balancing in State-Object Components for Compositional Zero-Shot Learning

Compositional Zero-Shot Learning (CZSL) aims to recognize unseen compositions from seen states and objects. The disparity between the manually labeled semantic information and its actual visual features causes a significant imbalance of visual deviation in the distribution of various object classes and state classes, which is ignored by existing methods. To ameliorate these issues, we consider the CZSL task as an unbalanced multi-label classification task and propose a novel method called MUtual balancing in STate-object components (MUST) for CZSL, which provides a balancing inductive bias for the model. In particular, we split the classification of the composition classes into two consecutive processes to analyze the entanglement of the two components to get additional knowledge in advance, which reflects the degree of visual deviation between the two components. We use the knowledge gained to modify the model’s training process in order to generate more distinct class borders for classes with significant visual deviations. Extensive experiments demonstrate that our approach significantly outperforms the state-of-the-art on MIT-States, UT-Zappos, and C-GQA when combined with the basic CZSL frameworks, and it can improve various CZSL frameworks. Our code is available at https://github.com/LanchJL/MUST.

Taming the Sigmoid Bottleneck: Provably Argmaxable Sparse Multi-Label Classification

Sigmoid output layers are widely used in multi-label classification (MLC) tasks, in which multiple labels can be assigned to any input. In many practical MLC tasks, the number of possible labels is in the thousands, often exceeding the number of input features and resulting in a low-rank output layer. In multi-class classification, it is known that such a low-rank output layer is a bottleneck that can result in unargmaxable classes: classes which cannot be predicted for any input. In this paper, we show that for MLC tasks, the analogous sigmoid bottleneck results in exponentially many unargmaxable label combinations. We explain how to detect these unargmaxable outputs and demonstrate their presence in three widely used MLC datasets. We then show that they can be prevented in practice by introducing a Discrete Fourier Transform (DFT) output layer, which guarantees that all sparse label combinations with up to k active labels are argmaxable. Our DFT layer trains faster and is more parameter efficient, matching the F1@k score of a sigmoid layer while using up to 50% fewer trainable parameters. Our code is publicly available at https://github.com/andreasgrv/sigmoid-bottleneck.

Attention-Induced Embedding Imputation for Incomplete Multi-View Partial Multi-Label Classification

As a combination of emerging multi-view learning methods and traditional multi-label classification tasks, multi-view multi-label classification has shown broad application prospects. The diverse semantic information contained in heterogeneous data effectively enables the further development of multi-label classification. However, the widespread incompleteness problem on multi-view features and labels greatly hinders the practical application of multi-view multi-label classification. Therefore, in this paper, we propose an attention-induced missing instances imputation technique to enhance the generalization ability of the model. Different from existing incomplete multi-view completion methods, we attempt to approximate the latent features of missing instances in embedding space according to cross-view joint attention, instead of recovering missing views in kernel space or original feature space. Accordingly, multi-view completed features are dynamically weighted by the confidence derived from joint attention in the late fusion phase. In addition, we propose a multi-view multi-label classification framework based on label-semantic feature learning, utilizing the statistical weak label correlation matrix and graph attention network to guide the learning process of label-specific features. Finally, our model is compatible with missing multi-view and partial multi-label data simultaneously and extensive experiments on five datasets confirm the advancement and effectiveness of our embedding imputation method and multi-view multi-label classification model.

Covering variable precision fuzzy rough sets based on overlap functions and the application to multi-label classification

Fuzzy rough sets (FRSs) play a significant role in the field of data analysis, and one of the common methods for constructing FRSs is the use of the fuzzy logic operators. To further extend FRSs theory to more diverse information backgrounds, this article proposes a covering variable precision fuzzy rough set model based on overlap functions and fuzzy β-neighbourhood operators (OCVPFRS). Some necessary properties of OCVPFRS have also been studied in this work. Furthermore, multi-label classification is a prevalent task in the realm of machine learning. Each object (sample or instance) in multi-label data is associated with various labels (classes), and there are numerous features or attributes that need to be taken into account within the attribute space. To enhance various performance metrics in the multi-label classification task, attribute reduction is an essential pre-processing step. Therefore, according to overlap functions and fuzzy rough sets’ excellent work on applications: such as image processing and multi-criteria decision-making, we establish an attribute reduction method suitable for multi-label data based on OCVPFRS. Through a series of experiments and comparative analysis with existing multi-label attribute reduction methods, the effectiveness and superiority of the proposed method have been verified.

MMDL-Net: Multi-Band Multi-Label Remote Sensing Image Classification Model

High-resolution remote sensing imagery comprises spatial structure features of multispectral bands varying in scale, color, and shape. These heterogeneous geographical features introduce grave challenges to the fine segmentation required for classification applications in remote sensing imagery, where direct application of traditional image classification models fails to deliver optimal results. To overcome these challenges, a multispectral, multi-label model, MMDL-Net, has been developed. This model is integrated with the multi-label BigEarthNet dataset, primarily employed for land cover classification research in remote sensing imagery, with each image composed of 13 spectral bands and spatial resolutions of 10 m, 20 m, and 60 m. To effectively utilize the information across these bands, a multispectral stacking module has been introduced to concatenate this spectral information. To proficiently process three distinct large-scale remote sensing image datasets, a multi-label classification module has been incorporated for training and inference. To better learn and represent the intricate features within the images, a twin-number residual structure has been proposed. The results demonstrate that the MMDL-Net model achieves a top accuracy of 83.52% and an F1 score of 77.97%, surpassing other deep learning models and conventional methods, thereby exhibiting exceptional performance in the task of multispectral multi-label classification of remote sensing imagery.

Combining convolutional neural networks and spatial-channel “squeeze and excitation” block for multiple-label image classification

<span lang="EN-US">In emergency rooms and intensive care units, catheters and tubes are used to keep critically ill patients alive. Appropriate catheter or tube insertion is crucial to avoiding serious complications. Such issues can be rectified if they are identified early. Chest X-rays are commonly used to assess catheter placement. Convolutional neural networks (</span><span lang="IN">CNN</span><span lang="EN-US">)</span><span lang="EN-US"> have recently been found to enhance multi-label classification tasks on chest X-rays images. Furthermore, attention modules have shown the effect of enhancing spatial encoding on network feature maps. This research analyzed the experiments of each CNN model with different attention blocks. Resnet200D with batch normalization and spatial-channel squeeze and excitation block (BN+scSE) is the best architecture for multiple-label image classification on a chest X-rays dataset from National Institutes of Health Clinical Center (NIH) with multiple catheters and tubes. Then came EfficientNetB5 with BN+scSE and Inception_v3 with spatial squeeze and channel excitation block, respectively.</span>

Correlation concept-cognitive learning model for multi-label classification

As a cognitive process, concept-cognitive learning (CCL) emphasizes the structured expression of data through systematic cognition and understanding, to obtain valuable information in the data. Although concept-cognitive learning has achieved good results in single-label classification tasks, it has not yet been applied to multi-label learning. The difficulty is that the existing concept-cognitive learning fails to effectively associate and utilize the structural relationships in the feature concepts and the multi-label concepts. In addition, in the face of complex multi-label datasets, it is difficult to obtain effective concepts for classification tasks. To solve these problems, this paper proposes a correlation concept-cognitive learning method and applies it to the multi-label classification (MLC) task. Moreover, the relationship between the feature concepts and the multi-label concepts is established by extent. On this basis, we comprehensively consider the extent relevancy and intent relevancy, and learn correlation concepts. In order to improve the classification precision, we construct correlation concept spaces to obtain the representation of effective concepts. Finally, we conducted experimental evaluations on ten datasets to illustrate the effectivity and advantages of the proposed approach.

MULTI-LABEL CLASSIFICATION OF THORACIC DISEASES USING STRUCTURED DEEP LEARNING FRAMEWORK

The current medical scenario indicates that thoracic diseases are the primary cause of illnesses of human beings worldwide. COVID-19 outbreak results in the generation of tremendous Chest X-Ray (CXR) and Computed Tomography (CT) image data archives because CXR and CT imaging are the foremost effective screening tools that characterize patterns and features of pathologies present in the lungs. After careful observation, these chest radiographs show that images are often labeled with more than one pathology, that extends disease diagnosis problem to tedious multi-label classification task. So, this paper proposes a modified mish activation function and compound loss-based Separable Convolution Neural Network (SCNN) model that accomplishes image-level detection of multiple pathologies and COVID-19 from multi-modality chest radiographs. We investigate the power of separable convolution in the CNN network that interprets spatial and depth-wise dimensions of each pixel in CXR and CT images. A modified mish called as Swmish, a non-monotonic activation function, is introduced that preserves negative gradient flow of input and capture fine-grained detail of multiple regions. Moreover, a compound loss function of dice coefficient and binary cross entropy is applied that strongly optimizes the model during training. The generality of the proposed SCNN model is confirmed after conducting comprehensive experiments on multi-label CXR and COVID19-CT datasets. The proposed model gains an AUC score of 0.91 and loss of 0.13 on the multi-label CXR dataset and achieves a testing accuracy of 0.97, AUC score of 0.99 and a loss of 0.11 on the COVID19-CT dataset. These results show that the proposed model achieves highest evaluation metrics in classifying binary and multi-label thoracic abnormalities from multi-modality chest radiographs.

A Bayesian network learning method for sparse and unbalanced data with GNN-based multilabel classification application

Creating a well-defined Bayesian network (BN) is helpful for developing effective graph neural networks (GNNs) that exploit annotated labels in multilabel classification (MLC) tasks. Obtaining correct BNs can be challenging when the labels are highly unbalanced and sparse. This study proposes a novel scoring method to address data imbalance and sparsity by introducing an imbalanced weight term. Comparison results show that the BNs learned from the proposed scoring method are simpler and more accurate than the well-known methods. The obtained BNs based on the proposed method are used to construct GNNs for MLC tasks based on three datasets. The findings suggest that the adjacency matrices based on moral graphs of BNs exhibit more excellent stability than those derived from mathematical approaches. By selecting the parameter thoughtfully, GNNs based on the proposed BDsp scores showcase up to 12% classification performance improvement compared to GNNs based on existing Bayesian scores.

Gridless Underdetermined DOA Estimation for Mobile Agents with Limited Snapshots Based on Deep Convolutional Generative Adversarial Network

Deep learning techniques have made certain breakthroughs in direction-of-arrival (DOA) estimation in recent years. However, most of the current deep-learning-based DOA estimation methods view the direction finding problem as a grid-based multi-label classification task and require multiple samplings with a uniform linear array (ULA), which leads to grid mismatch issues and difficulty in ensuring accurate DOA estimation with insufficient sampling and in underdetermined scenarios. In order to solve these challenges, we propose a new DOA estimation method based on a deep convolutional generative adversarial network (DCGAN) with a coprime array. By employing virtual interpolation, the difference co-array derived from the coprime array is extended to a virtual ULA with more degrees of freedom (DOFs). Then, combining with the Hermitian and Toeplitz prior knowledge, the covariance matrix is retrieved by the DCGAN. A backtracking method is employed to ensure that the reconstructed covariance matrix has a low-rank characteristic. We performed DOA estimation using the MUSIC algorithm. Simulation results demonstrate that the proposed method can not only distinguish more sources than the number of physical sensors but can also quickly and accurately solve DOA, especially with limited snapshots, which is suitable for fast estimation in mobile agent localization.

Dimensionality Reduction for Hierarchical Multi-Label Classification: A Systematic Mapping Study

Hierarchical multi-label classification problems typically deal with datasets with many attributes and labels, which can negatively impact the classifier performance. The application of dimensionality reduction methods can significantly improve the performance of classifiers. Dimensionality reduction can be performed by feature extraction or feature selection, according to the problem domain and datasets characteristics. This work carried out a systematic literature mapping to identify the approaches and techniques of dimensionality reduction that have been used in hierarchical multi-label classification tasks. Searches were performed on 7 important databases for the Computer Science field. From a list of 184 retrieved papers, 12 were selected for analysis, from which it was possible to determine a general overview of studies conducted from 2010 to 2022. It was identified that feature selection was the most frequent reduction method, with filter approach standing out. In addition, it was detected that most of the works used tree hierarchical structure. As its main outcome, this paper presents the state of the art of dimensionality reduction problem for hierarchical multi-label classification, indicating trends and research issues in the field.

A Formalization of Multilabel Classification in Terms of Lattice Theory and Information Theory: Concerning Datasets

Multilabel classification is a recently conceptualized task in machine learning. Contrary to most of the research that has so far focused on classification machinery, we take a data-centric approach and provide an integrative framework that blends qualitative and quantitative descriptions of multilabel data sources. By combining lattice theory, in the form of formal concept analysis, and entropy triangles, obtained from information theory, we explain from first principles the fundamental issues of multilabel datasets such as the dependencies of the labels, their imbalances, or the effects of the presence of hapaxes. This allows us to provide guidelines for resampling and new data collection and their relationship with broad modelling approaches. We have empirically validated our framework using 56 open datasets, challenging previous characterizations that prove that our formalization brings useful insights into the task of multilabel classification. Further work will consider the extension of this formalization to understand the relationship between the data sources, the classification methods, and ways to assess their performance.

Expanding analytical capabilities in intrusion detection through ensemble-based multi-label classification

Intrusion detection systems are primarily designed to flag security breaches upon their occurrence. These systems operate under the assumption of single-label data, where each instance is assigned to a single category. However, when dealing with complex data, such as malware triage, the information provided by the IDS is limited. Consequently, additional analysis becomes necessary, leading to delays and incurring additional computational costs. Existing solutions to this problem typically merge these steps by considering a unified, but large, label set encompassing both intrusion and analytical labels, which adversely affects efficiency and performance. To address these challenges, this paper presents a novel framework for multi-label classification by employing an ensemble of sequential models that preserve the original label sets during training. Each model focuses on learning the distribution specifically related to its assigned set of labels, independent of the other label sets. To capture the relationship between different sets of labels, the parameters of each trained model initialize the subsequent model, ensuring that information from unrelated label sets does not interfere with the learning objective. Consequently, the proposed method enhances prediction performance without increasing computational complexity. To evaluate the effectiveness of our approach, we conduct experiments on a real-world dataset related to intrusion detection. The results clearly demonstrate the effectiveness of our proposed method in handling multi-label classification tasks.

NLP-enabled Recommendation of Hashtags for Covid based Tweets using Hybrid BERT-LSTM Model

Hashtags have become a new trend to summarize the feelings, sentiments, emotions, swinging moods, food tastes and much more. It also represents various entities like places, families and friends. It is a way to search and categorize various stuff on social media sites. With the increase in the hashtagging, there is a need to automate it, leading to the term “Hashtag Recommendation”. Also, there are plenty of posts on social media sites which remain untagged. These untagged posts get filtered out while searching and categorizing the data using a label. Such posts do not make any contribution to any helpful insight and remain abandoned. But, if the user of such posts is recommended by labels according to his post, then he may choose one or more of them, thus making the posts labelled. For such cases Hashtag recommendation comes into the picture. Although much research work has been done on Hashtag Recommendation using traditional Deep Learning approaches, not much work has been done using NLP based Bert Embedding. In this paper, we have proposed a model, BELHASH, Bert Embedding based LSTM for Hashtag Recommendation. This task is considered as a Multilabel Classification task as the hashtags are one-hot encoded into multiple binary vectors of zeros and ones using MultiLabelBinarizer. This model has been evaluated on Covid 19 tweets. We have achieved 0.72 accuracy, 0.7 Precision, 0.66 Recall and 0.67 F1-Score. This is the first paper of hashtag recommendation to the best of our knowledge combining Bert embedding with LSTM model and achieving the state of the arts results.

Multi-Label Text Classification Model Based on Multi-Level Constraint Augmentation and Label Association Attention

In the multi-label text classification task, a text usually corresponds to multiple label categories, and the labels have correlation and hierarchical structure. However, when the label hierarchy is unknown, the number of various labels is not balanced, which makes it difficult for the model to classify low-frequency labels. In addition, labels have semantic similarities that make it difficult for the model to distinguish between them. In this article, we propose a multi-label text classification model based on multi-level constraint augmentation and label association attention. Compared with traditional methods, our method has two contributions: (1) In order to alleviate the problem of unbalanced number of different label categories and ensure the rationality of sample generation, we propose a data augmentation method based on multi-level constraints. In the process of sample generation, this method uses historical generation information, sample original text information, and sample topic to constrain the generated text. (2) In order to make the model recognize the associated labels accurately, we propose an interaction mechanism based on label association attention and filter gate. This method combines text information and label weight information. At the same time, our classification model considers the important weights of text sentences and effectively utilizes the co-occurrence relationship between labels. Experimental results on three benchmark datasets show that our model outperforms state-of-the-art methods on all main evaluation metrics, especially on low-frequency label prediction with sparse samples.