Pairwise Learning Research Articles

A unified multimodal classification framework based on deep metric learning

Multimodal classification algorithms play an essential role in multimodal machine learning, aiming to categorize distinct data points by analyzing data characteristics from multiple modalities. Extensive research has been conducted on distilling multimodal attributes and devising specialized fusion strategies for targeted classification tasks. Nevertheless, current algorithms mainly concentrate on a specific classification task and process data about the corresponding modalities. To address these limitations, we propose a unified multimodal classification framework proficient in handling diverse multimodal classification tasks and processing data from disparate modalities. UMCF is task-independent, and its unimodal feature extraction module can be adaptively substituted to accommodate data from diverse modalities. Moreover, we construct the multimodal learning scheme based on deep metric learning to mine latent characteristics within multimodal data. Specifically, we design the metric-based triplet learning to extract the intra-modal relationships within each modality and the contrastive pairwise learning to capture the inter-modal relationships across various modalities. Extensive experiments on two multimodal classification tasks, fake news detection and sentiment analysis, demonstrate that UMCF can extract multimodal data features and achieve superior classification performance than task-specific benchmarks. UMCF outperforms the best fake news detection baselines by 2.3% on average regarding F1 scores.

Towards sharper excess risk bounds for differentially private pairwise learning

Pairwise learning is a vital part of machine learning. It depends on pairs of training instances, and is naturally fit for modeling relationships between samples. However, as a data driven paradigm, it faces huge privacy issues. Differential privacy (DP) is a useful tool to protect the privacy of machine learning, but corresponding excess population risk bounds are loose in existing DP pairwise learning analysis. In this paper, we propose a gradient perturbation algorithm for pairwise learning to get better risk bounds under Polyak–Łojasiewicz condition, including both convex and non-convex cases. Specifically, for the theoretical risk bound in expectation, previous best results are of rates O(pn2ϵ2+1n) and O(pnϵ+1n) under strongly convex condition and convex conditions, respectively. In this paper, we use the on-average stability and achieve an O(min{pn1.5ϵ+p1.5n2.5ϵ3,pn2ϵ2+1n}) bound, significantly improving previous bounds. For the high probability risk bound, previous best results are analyzed by the uniform stability, and O(βnU+pnϵ) excess population risk bounds are achieved under strongly convex or convex conditions, where βnU is the traditional pairwise uniform stability parameter, it is large since it considers the worst case of the loss sensitivity. In this paper, we propose the pairwise locally elastic stability and improve the high probability bound to O(βEn+pnϵ), in which the pairwise locally elastic stability parameter βE≪βnU because it considers the average sensitivity of the pairwise loss function.

Multi-objective evolutionary multi-tasking band selection algorithm for hyperspectral image classification

Hyperspectral images (HSI) contain a great number of bands, which enable better characterization of features. However, the huge dimension and information volume brought by the abundant bands may give rise to a negative influence on the efficiency of subsequent processing on hyperspectral images. Band selection (BS) is a commonly adopted to reduce the dimension of HSIs. Different from the previous work, in this paper, hyperspectral band selection problem is formulated as a multi-objective optimization problem, where the band distribution uniformity among the selected bands and inter-class separation distance from a few labeled samples are optimized simultaneously. To fully exploit the relation between the band subsets with different sizes, we construct a multi-objective evolutionary multi-tasking algorithm for hyperspectral band selection (namely MEMT-HBS) to achieve the selected band subsets for all the selected band sizes in one run. To implement MEMT-HBS, the intra-task pairwise learning based solution generation strategy is suggested to evolve the population for each task to achieve high-quality offspring whose selected band size is restricted to a fixed scope. The inter-task band coverage based knowledge transferring strategy is utilized to choose useful individuals from adjacent tasks to further enhance the performance of current task. Compared with the state-of-the-art semi-supervised and unsupervised BS algorithms, empirical results on different standard hyperspectral datasets show that our proposed MEMT-HBS can determine the superior band subset which has a higher image classification accuracy over the comparison algorithms.

DP-MP: a novel cross-subject fatigue detection framework with DANN-based prototypical representation and mix-up pairwise learning.

Electroencephalography (EEG) is widely recognized as an effective method for detecting fatigue. However, practical applications of EEG for fatigue detection in real-world scenarios are often challenging, particularly in cases involving subjects not included in the training datasets, owing to bio-individual differences and noisy labels. This study aims to develop an effective framework for cross-subject fatigue detection by addressing these challenges. In this study, we propose a novel framework, termed DP-MP, for cross-subject fatigue detection, which utilizes a Domain-Adversarial Neural Network (DANN)-based prototypical representation in conjunction with Mix-up pairwise learning. Our proposed DP-MP framework aims to mitigate the impact of bio-individual differences by encoding fatigue-related semantic structures within EEG signals and exploring shared fatigue prototype features across individuals. Notably, to the best of our knowledge, this work is the first to conceptualize fatigue detection as a pairwise learning task, thereby effectively reducing the interference from noisy labels. Furthermore, we propose the Mix-up pairwise learning (MixPa) approach in the field of fatigue detection, which broadens the advantages of pairwise learning by introducing more diverse and informative relationships among samples. Cross-subject experiments were conducted on two benchmark databases, SEED-VIG and FTEF, achieving state-of-the-art performance with average accuracies of 88.14% and 97.41%, respectively. These promising results demonstrate our model's effectiveness and excellent generalization capability. This is the first time EEG-based fatigue detection has been conceptualized as a pairwise learning task, offering a novel perspective to this field. Moreover, our proposed DP-MP framework effectively tackles the challenges of bio-individual differences and noisy labels in the fatigue detection field and demonstrates superior performance. Our work provides valuable insights for future research, promoting the application of brain-computer interfaces for fatigue detection in real-world scenarios.&#xD.

BiSpec Pairwise AI: guiding the selection of bispecific antibody target combinations with pairwise learning and GPT augmentation

PurposeBispecific antibodies (BsAbs), capable of targeting two antigens simultaneously, represent a significant advancement by employing dual mechanisms of action for tumor suppression. However, how to pair targets to develop effective and safe bispecific drugs is a major challenge for pharmaceutical companies.MethodsUsing machine learning models, we refined the biological characteristics of currently approved or in clinical development BsAbs and analyzed hundreds of membrane proteins as bispecific targets to predict the likelihood of successful drug development for various target combinations. Moreover, to enhance the interpretability of prediction results in bispecific target combination, we combined machine learning models with Large Language Models (LLMs). Through a Retrieval-Augmented Generation (RAG) approach, we supplement each pair of bispecific targets’ machine learning prediction with important features and rationales, generating interpretable analytical reports.ResultsIn this study, the XGBoost model with pairwise learning was employed to predict the druggability of BsAbs. By analyzing extensive data on BsAbs and designing features from perspectives such as target activity, safety, cell type specificity, pathway mechanism, and gene embedding representation, our model is able to predict target combinations of BsAbs with high market potential. Specifically, we integrated XGBoost with the GPT model to discuss the efficacy of each bispecific target pair, thereby aiding the decision-making for drug developers.ConclusionThe novelty of this study lies in the integration of machine learning and GPT techniques to provide a novel framework for the design of BsAbs drugs. This holistic approach not only improves prediction accuracy, but also enhances the interpretability and innovativeness of drug design.

A Novel Progressive Enhancement of Low Light Raw Images

Low-light imaging on mobile devices is often difficult due to the issue of not enough light passing through the small aperture, resulting in poor quality images. Most previous work on low-resolution images has focused on a single task, such as illumination, color correction, or noise removal; Noise removal function based on short and long images of the camera model. This technique is less efficient and general in real-world environments that require special camera integration and restoration. In this paper, we propose a lighting system that can integrate illumination variation, color correction, and noise removal to solve this problem. Considering the difficulty of obtaining model-specific data and the maximum content of the resulting image, we created two branches: the coefficient estimation branch and the integration branch. While the computer estimator operates in the sparsely resolved space and estimates the coefficients developed by pairwise learning, the collaborative system operates in the fully resolved space, providing step-by-step integration and denoising. Compared to existing methods, our framework does not need to remember as much information when switching to another camera model, which reduces the effort required to benefit our usage pattern. Through extensive testing, we demonstrate its great potential in real-world low-light applications.

PR-PL: A Novel Prototypical Representation Based Pairwise Learning Framework for Emotion Recognition Using EEG Signals

Affective brain-computer interface based on electroencephalography (EEG) is an important branch in the field of affective computing. However, the individual differences in EEG emotional data and the noisy labeling problem in the subjective feedback seriously limit the effectiveness and generalizability of existing models. To tackle these two critical issues, we propose a novel transfer learning framework with Prototypical Representation based Pairwise Learning ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PR-PL</i> ). The discriminative and generalized EEG features are learned for emotion revealing across individuals and the emotion recognition task is formulated as pairwise learning for improving the model tolerance to the noisy labels. More specifically, a prototypical learning is developed to encode the inherent emotion-related semantic structure of EEG data and align the individuals' EEG features to a shared common feature space under consideration of the feature separability of both source and target domains. Based on the aligned feature representations, pairwise learning with an adaptive pseudo labeling method is introduced to encode the proximity relationships among samples and alleviate the label noises effect on modeling. Extensive results on two benchmark databases (SEED and SEED-IV) under four different cross-validation evaluation protocols validate the model reliability and stability across subjects and sessions. Compared to the literature, the average enhancement of emotion recognition across four different evaluation protocols is 2.04% (SEED) and 2.58% (SEED-IV). The source code is available at <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><uri>https://github.com/KAZABANA/PR-PL</uri></i> .

Differentially private stochastic gradient descent with low-noise

Modern machine learning algorithms aim to extract fine-grained information from data to provide accurate predictions, which often conflicts with the goal of privacy protection. This paper addresses the practical and theoretical importance of developing privacy-preserving machine learning algorithms that ensure good performance while preserving privacy. In this paper, we focus on the privacy and utility (measured by excess risk bounds) performances of differentially private stochastic gradient descent (SGD) algorithms in the setting of stochastic convex optimization. Specifically, we examine the pointwise problem in the low-noise setting for which we derive sharper excess risk bounds for the differentially private SGD algorithm. In the pairwise learning setting, we propose a simple differentially private SGD algorithm based on gradient perturbation. Furthermore, we develop novel utility bounds for the proposed algorithm, proving that it achieves optimal excess risk rates even for non-smooth losses. Notably, we establish fast learning rates for privacy-preserving pairwise learning under the low-noise condition, which is the first of its kind.

Limited Memory Online Gradient Descent for Kernelized Pairwise Learning with Dynamic Averaging

Pairwise learning, an important domain within machine learning, addresses loss functions defined on pairs of training examples, including those in metric learning and AUC maximization. Acknowledging the quadratic growth in computation complexity accompanying pairwise loss as the sample size grows, researchers have turned to online gradient descent (OGD) methods for enhanced scalability. Recently, an OGD algorithm emerged, employing gradient computation involving prior and most recent examples, a step that effectively reduces algorithmic complexity to O(T), with T being the number of received examples. This approach, however, confines itself to linear models while assuming the independence of example arrivals. We introduce a lightweight OGD algorithm that does not require the independence of examples and generalizes to kernel pairwise learning. Our algorithm builds the gradient based on a random example and a moving average representing the past data, which results in a sub-linear regret bound with a complexity of O(T). Furthermore, through the integration of O(√T logT) random Fourier features, the complexity of kernel calculations is effectively minimized. Several experiments with real-world datasets show that the proposed technique outperforms kernel and linear algorithms in offline and online scenarios.

Mitigating Confounding Bias in Practical Recommender Systems With Partially Inaccessible Exposure Status.

To improve user experience, recommender systems have been widely used on many online platforms. In these systems, recommendation models are typically learned from positive/negative feedback that are collected automatically. Notably, recommender systems are a little different from general supervised learning tasks. In recommender systems, there are some factors (e.g., previous recommendation models or operation strategies of a online platform) that determine which items can be exposed to each individual user. Normally, the previous exposure results are not only relevant to the instances' features (i.e., user or item), but also affect their feedback ratings, thus leading to confounding bias in the recommendation models. To mitigate this bias, researchers have already provided a variety of strategies. However, there are still two issues that are underappreciated: 1) previous debiased RS approaches cannot effectively capture recommendation-specific, exposure-specific and their common knowledge simultaneously; 2) the true exposure results of the user-item pairs are partially inaccessible, so there would be some noises if we use their observability to approximate it as existing approaches. Motivated by this, we develop a novel debiasing recommendation approach. More specifically, we first propose a mutual information-based counterfactual learning framework based on the causal relationship among the instance features, exposure status, and ratings. This framework can 1) capture recommendation-specific, exposure-specific and their common knowledge by explicitly modeling the relationship among the causal factors, and 2) achieve robustness towards partially inaccessible exposure results by a pairwise learning strategy. Under such a framework, we implement an optimizable loss function with theoretical analysis. By minimizing this loss, we expect to obtain an unbiased recommendation model that reflects the users' real interests. Meanwhile, we also prove that our loss function has robustness towards the partial inaccessibility of the exposure status. Finally, extensive experiments on public datasets manifest the superiority of our proposed method in boosting the recommendation performance.

New Scalable and Efficient Online Pairwise Learning Algorithm.

Pairwise learning is an important machine-learning topic with many practical applications. An online algorithm is the first choice for processing streaming data and is preferred for handling large-scale pairwise learning problems. However, existing online pairwise learning algorithms are not scalable and efficient enough for large-scale high-dimensional data, because they were designed based on singly stochastic gradients. To address this challenging problem, in this article, we propose a dynamic doubly stochastic gradient algorithm (D2SG) for online pairwise learning. Especially, only the time and space complexities of O (d) are needed for incorporating a new sample, where d is the dimensionality of data. This means that our D2SG is much faster and more scalable than the existing online pairwise learning algorithms while the statistical accuracy can be guaranteed through our rigorous theoretical analysis under standard assumptions. The experimental results on a variety of real-world datasets not only confirm the theoretical result of our new D2SG algorithm, but also show that D2SG has better efficiency and scalability than the existing online pairwise learning algorithms.

MSDSE: Predicting drug-side effects based on multi-scale features and deep multi-structure neural network

Unexpected side effects may accompany the research stage and post-marketing of drugs. These accidents lead to drug development failure and even endanger patients' health. Thus, it is essential to recognize the unknown drug-side effects. Most existing methods in silico find the answer from the association network or similarity network of drugs while ignoring the drug-intrinsic attributes. The limitation is that they can only handle drugs in the maturation stage.To be suitable for early drug-side effect screening, we conceive a multi-structural deep learning framework, MSDSE, which synthetically considers the multi-scale features derived from the drug. MSDSE can jointly learn SMILES sequence-based word embedding, substructure-based molecular fingerprint, and chemical structure-based graph embedding. In the preprocessing stage of MSDSE, we project all features to the abstract space with the same dimension. MSDSE builds a bi-level channel strategy, including a convolutional neural network module with an Inception structure and a multi-head Self-Attention module, to learn and integrate multi-modal features from local to global perspectives. Finally, MSDSE regards the prediction of drug-side effects as pair-wise learning and outputs the pair-wise probability of drug-side effects through the inner product operation. MSDSE is evaluated and analyzed on benchmark datasets and performs optimally compared to other baseline models. We also set up the ablation study to explain the rationality of the feature approach and model structure. Moreover, we select model partial prediction results for the case study to reveal actual capability. The original data are available at http://github.com/yuliyi/MSDSE.

Deep pairwise learning for user preferences via dual graph attention model in location-based social networks

Deep pairwise learning for user preferences via dual graph attention model in location-based social networks

Pairwise Guided Multilayer Cross-Fusion Network for Bird Image Recognition

Bird identification is the first step in collecting data on bird diversity and abundance, which also helps research on bird distribution and population measurements. Most research has built end-to-end training models for bird detection task via CNNs or attentive models, but many perform unsatisfactorily in fine-grained bird recognition. Bird recognition tasks are highly influenced by factors, including the similar appearance of different subcategories, diverse bird postures, and other interference factors such as tree branches and leaves from the background. To tackle this challenge, we propose the Progressive Cross-Union Network (PC-Net) to capture more subtle parts with low-level attention maps. Based on cross-layer information exchange and pairwise learning, the proposed method uses two modules to improve feature representation and localization. First, it utilizes low- and high-level information for cross-layer feature fusion, which enables the network to extract more comprehensive and discriminative features. Second, the network incorporates deep semantic localization to identify and enhance the most relevant regions in the images. In addition, the network is designed with a semantic guidance loss to improve its generalization for variable bird poses. The PC-Net was evaluated on an extensively used birds dataset (CUB-200-2011), which contains 200 birds subcategories. The results demonstrate that the PC-Net achieved an impressive recognition accuracy of 89.2%, thereby outperforming maintained methods in bird subcategory identification. We also achieved competitive results on two other datasets with data on cars and airplanes. The results indicated that the PC-Net improves the accuracy of diverse bird recognition, as well as other fine-grained recognition scenarios.

Learning Rates for Nonconvex Pairwise Learning

Pairwise learning is receiving increasing attention since it covers many important machine learning tasks, e.g., metric learning, AUC maximization, and ranking. Investigating the generalization behavior of pairwise learning is thus of great significance. However, existing generalization analysis mainly focuses on the convex objective functions, leaving the nonconvex pairwise learning far less explored. Moreover, the current learning rates of pairwise learning are mostly of slower order. Motivated by these problems, we study the generalization performance of nonconvex pairwise learning and provide improved learning rates. Specifically, we develop different uniform convergence of gradients for pairwise learning under different assumptions, based on which we characterize empirical risk minimizer, gradient descent, and stochastic gradient descent. We first establish learning rates for these algorithms in a general nonconvex setting, where the analysis sheds insights on the trade-off between optimization and generalization and the role of early-stopping. We then derive faster learning rates of order O(1/n) for nonconvex pairwise learning with a gradient dominance curvature condition, where n is the sample size. Provided that the optimal population risk is small, we further improve the learning rates to O(1/n2), which, to the best of our knowledge, are the first O(1/n2) rates for pairwise learning.

Stability-Based Generalization Analysis for Mixtures of Pointwise and Pairwise Learning

Stability-Based Generalization Analysis for Mixtures of Pointwise and Pairwise Learning

A pairwise ranking estimation model for surrogate-assisted evolutionary algorithms

Surrogate-assisted evolutionary algorithms (SAEAs) have attracted considerable attention for reducing the computation time required by an EA on computationally expensive optimization problems. In such algorithms, a surrogate model estimates the solution evaluation with a low computing cost and is used to obtain promising solutions to which the accurate evaluation with an expensive computation cost is then applied. This study proposes a novel pairwise ranking surrogate model called the Extreme Learning-machine-based DirectRanker (ELDR). ELDR integrates two machine learning models: extreme learning machine (ELM) and DirectRanker (DR). ELM is a single-layer neural network capable of fast learning, whereas DR uses pairwise learning to rank using a neural network developed mainly for information retrieval. To investigate the effectiveness of the proposed surrogate model, this study first examined the estimation accuracy of ELDR. Subsequently, ELDR was incorporated into a state-of-the-art SAEA and compared with existing SAEAs on well-known real-valued optimization benchmark problems. The experimental results revealed that ELDR has a high estimation accuracy even on high-dimensional problems with a small amount of training data. In addition, the SAEA using ELDR exhibited a high search performance compared with other existing SAEAs, especially on high-dimensional problems.

Generalization Analysis of Pairwise Learning for Ranking with Deep Neural Networks.

Pairwise learning is widely employed in ranking, similarity and metric learning, area under the curb maximization, and many other learning tasks involving sample pairs. Pairwise learning with deep neural networks was considered for ranking, but enough theoretical understanding about this topic is lacking. In this letter, we apply symmetric deep neural networks to pairwise learning for ranking with a hinge loss ϕh and carry out generalization analysis for this algorithm. A key step in our analysis is to characterize a function that minimizes the risk. This motivates us to first find the minimizer of ϕh-risk and then design our two-part deep neural networks with shared weights, which induces the antisymmetric property of the networks. We present convergence rates of the approximation error in terms of function smoothness and a noise condition and give an excess generalization error bound by means of properties of the hypothesis space generated by deep neural networks. Our analysis is based on tools from U-statistics and approximation theory.

Preciser comparison: Augmented multi-layer dynamic contrastive strategy for text2text question classification

Text2text question classification (TQC), as a particular application case of question classification (QC), is of great practical value. Traditional QC methods usually label categories of questions using one or multiple keywords provided by users. In contrast, in TQC, each question in natural language is automatically categorized into pre-designed standard question classes, which are coded in the form of short text. Because of this unique characteristic, TQC relies on a specifically designed framework and should be trained and validated based on customized experimental datasets. Previous TQC-related work mainly utilized textual similarity-matching methods. However, no effective pairwise learning paradigm has been proposed in TQC to model correlations between input text and classes; and the influence of distance metrics and loss function in TQC has not been investigated. In this work, we propose a novel and comprehensive strategy, Augmented Dynamic Multi-layer Contrastive (ADMC), to resolve the challenge of TQC. Our framework consists of (1) an optional data augmentation module, (2) one stage for dynamic negative sampling, and (3) one stage for precise matching. The comprehensive TQC framework with ADMC strategy in this work resolves data imbalance and explores distance metrics learning via multiple augmentation options and dynamic negative sampling based on multi-layer contrastive learning. To compensate for the shortage of public datasets for this task, we collected two real-world datasets and adaptively expanded three existing public datasets, which will be available after data masking. The results show that our ADMC outperformed other baseline methods investigated in this paper. The codes are available at https://github.com/WJULYW/ADMC.

Content-based medical image retrieval with opponent class adaptive margin loss

The increasing utilization of medical imaging technology with digital storage capabilities has facilitated the compilation of large-scale data repositories. Fast access to image samples with similar appearance to suspected cases in these repositories can help establish a consulting system for healthcare professionals, and improve diagnostic procedures while minimizing processing delays. However, manual querying of large repositories is labor intensive. Content-based image retrieval (CBIR) offers an automated solution based on quantitative assessment of image similarity based on image features in a latent space. Since conventional methods based on hand-crafted features typically show poor generalization performance, learning-based CBIR methods have received attention recently. A common framework in this domain involves classifier-guided models that are trained to detect different image classes. Similarity assessments are then performed on the features captured by the intermediate stages of the trained models. While classifier-guided methods are powerful in inter-class discrimination, they are suboptimally sensitive to within-class differences in image features. An alternative framework instead performs task-agnostic training to learn an embedding space that enforces the representational discriminability of images. Within this representational-learning framework, a powerful method is triplet-wise learning that addresses the deficiencies of point-wise and pair-wise learning in characterizing the similarity relationships between image classes. However, the traditional triplet loss enforces separation between only a subset of image samples within the triplet via a manually-set constant margin value, so it can lead to suboptimal segregation of opponent classes and limited generalization performance. To address these limitations, we introduce a triplet-learning method for automated querying of medical image repositories based on a novel Opponent Class Adaptive Margin (OCAM) loss. To maintain optimally discriminative representations, OCAM considers relationships among all image pairs within the triplet and utilizes an adaptive margin value that is automatically selected per dataset and during the course of training iterations. CBIR performance of OCAM is compared against state-of-the-art loss functions for representational learning on three public databases (gastrointestinal disease, skin lesion, lung disease). On average, OCAM shows an mAP performance of 86.30% in the KVASIR dataset, 70.30% in the ISIC 2019 dataset, and 85.57% in the X-RAY dataset. Comprehensive experiments in each application domain demonstrate the superior performance of OCAM against competing triplet-wise methods at 1.52%, classifier-guided methods at 2.29%, and non-triplet representational-learning methods at 4.56%.