Noise Contrastive Estimation Research Articles

VTPL: Visual and text prompt learning for visual-language models

Visual-language (V-L) models have achieved remarkable success in learning combined visual–textual representations from large web datasets. Prompt learning, as a solution for downstream tasks, can address the forgetting of knowledge associated with fine-tuning. However, current methods focus on a single modality and fail to fully use multimodal information. This paper aims to address these limitations by proposing a novel approach called visual and text prompt learning (VTPL) to train the model and enhance both visual and text prompts. Visual prompts align visual features with text features, whereas text prompts enrich the semantic information of the text. Additionally, this paper introduces a poly-1 information noise contrastive estimation (InfoNCE) loss and a center loss to increase the interclass distance and decrease the intraclass distance. Experiments on 11 image datasets show that VTPL outperforms state-of-the-art methods, achieving 1.61%, 1.63%, 1.99%, 2.42%, and 2.87% performance boosts over CoOp for 1, 2, 4, 8, and 16 shots, respectively.

Cluster-Based Pairwise Contrastive Loss for Noise-Robust Speech Recognition.

This paper addresses a joint training approach applied to a pipeline comprising speech enhancement (SE) and automatic speech recognition (ASR) models, where an acoustic tokenizer is included in the pipeline to leverage the linguistic information from the ASR model to the SE model. The acoustic tokenizer takes the outputs of the ASR encoder and provides a pseudo-label through K-means clustering. To transfer the linguistic information, represented by pseudo-labels, from the acoustic tokenizer to the SE model, a cluster-based pairwise contrastive (CBPC) loss function is proposed, which is a self-supervised contrastive loss function, and combined with an information noise contrastive estimation (infoNCE) loss function. This combined loss function prevents the SE model from overfitting to outlier samples and represents the pronunciation variability in samples with the same pseudo-label. The effectiveness of the proposed CBPC loss function is evaluated on a noisy LibriSpeech dataset by measuring both the speech quality scores and the word error rate (WER). The experimental results reveal that the proposed joint training approach using the described CBPC loss function achieves a lower WER than the conventional joint training approaches. In addition, it is demonstrated that the speech quality scores of the SE model trained using the proposed training approach are higher than those of the standalone-SE model and SE models trained using conventional joint training approaches. An ablation study is also conducted to investigate the effects of different combinations of loss functions on the speech quality scores and WER. Here, it is revealed that the proposed CBPC loss function combined with infoNCE contributes to a reduced WER and an increase in most of the speech quality scores.

Medical nearest-word embedding technique implemented using an unsupervised machine learning approach for Bengali language

Abstract The rapid growth of natural language processing (NLP) applications, such as text summarization, speech recognition, information extraction, and machine translation, has led to the development of structured query language (SQL) for extracting information from structured data. However, due to limited resources, converting Natural Language (NL) queries to SQL in Bengali is challenging. This article proposes an unsupervised machine learning model to find semantically Bengali closed words that can generate SQL from NL queries in Bengali. The main objective of the proposed system is to provide support in the creation of patient-oriented explanations and educational resources by simplifying intricate medical terminology. The major findings of the proposed system are as follows: The use of machine translation in the field of medicine facilitates the dissemination of healthcare information to a diverse international audience and improves the performance of entity recognition tasks, including the identification of medical conditions, drugs, or procedures within clinical notes or electronic health data. This system allows a naive user to extract health-related information from a healthcare-structured database without any knowledge of SQL. The system accepts a query and generates a response according to the query in Bengali language. Query tokenization and stop word removal are carried out in the preprocessing stage, and unsupervised machine learning techniques are implemented to process the input query sentence. Tokenized words are converted into vectors using the skip-gram model, with noise-contrastive estimation (NCE) applied to discriminate between actual and irrelevant words. Stochastic gradient descent (SGD) optimizes the model by randomly choosing a small amount of data from the dataset and using cosine similarity to measure closer words. The semantically closer words are found using an unsupervised learning method to generate the SQL.

Mitigating the Impact of False Negative in Dense Retrieval with Contrastive Confidence Regularization

In open-domain Question Answering (QA), dense text retrieval is crucial for finding relevant passages to generate answers. Typically, contrastive learning is used to train a retrieval model, which maps passages and queries to the same semantic space, making similar ones closer and dissimilar ones further apart. However, training such a system is challenging due to the false negative problem, where relevant passages may be missed during data annotation. Hard negative sampling, commonly used to improve contrastive learning, can introduce more noise in training. This is because hard negatives are those close to a given query, and thus more likely to be false negatives. To address this, we propose a novel contrastive confidence regularizer for Noise Contrastive Estimation (NCE) loss, a commonly used contrastive loss. Our analysis shows that the regularizer helps make the dense retrieval model more robust against false negatives with a theoretical guarantee. Additionally, we propose a model-agnostic method to filter out noisy negative passages in the dataset, improving any downstream dense retrieval models. Through experiments on three datasets, we demonstrate that our method achieves better retrieval performance in comparison to existing state-of-the-art dense retrieval systems.

Multi-Label Contrastive Learning for Abstract Visual Reasoning.

For a long time, the ability to solve abstract reasoning tasks was considered one of the hallmarks of human intelligence. Recent advances in the application of deep learning (DL) methods led to surpassing human abstract reasoning performance, specifically in the most popular type of such problems-Raven's progressive matrices (RPMs). While the efficacy of DL systems is indeed impressive, the way they approach the RPMs is very different from that of humans. State-of-the-art systems solving RPMs rely on massive pattern-based training and sometimes on exploiting biases in the dataset, whereas humans concentrate on the identification of the rules/concepts underlying the RPM to be solved. Motivated by this cognitive difference, this work aims at combining DL with the human way of solving RPMs. Specifically, we cast the problem of solving RPMs into a multilabel classification framework where each RPM is viewed as a multilabel data point, with labels determined by the set of abstract rules underlying the RPM. For efficient training of the system, we present a generalization of the noise contrastive estimation algorithm to the case of multilabel samples and a new sparse rule encoding scheme for RPMs. The proposed approach is evaluated on the two most popular benchmark datasets [I-RAVEN and procedurally generated matrices (PGM)] and on both of them demonstrate an advantage over the state-of-the-art results.

Asymmetrical Attention Networks Fused Autoencoder for Debiased Recommendation

Popularity bias is a massive challenge for autoencoder-based models, which decreases the level of personalization and hurts the fairness of recommendations. User reviews reflect their preferences and help mitigate bias or unfairness in the recommendation. However, most existing works typically incorporate user (item) reviews into a long document and then use the same module to process the document in parallel. Actually, the set of user reviews is completely different from the set of item reviews. User reviews are heterogeneous in that they reflect a variety of items purchased by users, while item reviews are only related to the item itself and are thus typically homogeneous. In this article, a novel asymmetric attention network fused with autoencoders is proposed, which jointly learns representations from the user and item reviews and implicit feedback to perform recommendations. Specifically, we design an asymmetric attentive module to capture rich representations from user and item reviews, respectively, which solves data sparsity and explainable problems. Furthermore, to further address popularity bias, we apply a noise-contrastive estimation objective to learn high-quality “de-popularity” embedding via the decoder structure. A series of extensive experiments are conducted on four benchmark datasets to show that leveraging user review information can eliminate popularity bias and improve performance compared to various state-of-the-art recommendation techniques.

Preserving Structural Consistency in the Generation of Synthetic CT in Pelvic MR-Only Radiation Treatment Planning

MR-based synthetic CT (sCT) generation is necessary for MR-only radiotherapy to assist in radiation dose calculation, owing to no electronic density information in MR images. This study investigated the feasibility of synthesizing CT images from magnetic resonance (MR) images using generation antagonism networks (GANs) for MR radiotherapy of rectal cancer. Meanwhile, the transformer module and the contrast learning loss were introduced to improve the sCT. The data set used in this study was the T2-weighted MR and CT image data of 108 patients with rectal cancer. Three-fold cross-validation was performed on all data sets. The transformer module was introduced into the plain CycleGAN, and the improved Patch Noise Contrastive Estimation (PatchNCE) loss was used as the loss function. The improved PatchNCE loss maintained the structural consistency of the MR and the synthetic CT by ensuring the consistency of the distribution of image patches on the MR-sCT image pair. The 2.5D images were taken as the input of our model, which refers to taking two consecutive adjacent layers in a specific layer. The CT-to-sCT image similarity was evaluated by metrics of mean absolute error (MAE), peak signal-to-noise ratio (PSNR), and Structure Similarity Index Measure (SSIM). The sCT dosimetric accuracy was verified against CT-based dose distributions for the photon plan. Relative dose differences in the planning target volume and organs at risk were computed. The evaluation indicators of sCT images generated by our model were superior to the plain CycleGAN in the results of the three-fold cross-validation. MAE, PSNR and SSIM of our model were 42.850HU, 26.486 and 0.988, respectively, which were superior to 47.129HU, 25.167 and 0.978 of the plain CycleGAN. In addition, sCT generated by our model exhibited good continuity in the axial direction compared with plain CycleGAN. Furthermore, most of the relative differences in the DVH indicators were less than 1%. The accuracy of sCT can be effectively improved by introducing a transformer module and comparative learning loss function. Moreover, all dosimetric differences were within clinically acceptable criteria for photon radiotherapy, demonstrating the feasibility of the MRI-only workflow for patients with rectal cancer.

Similarity contrastive estimation for image and video soft contrastive self-supervised learning

Contrastive representation learning has proven to be an effective self-supervised learning method for images and videos. Most successful approaches are based on Noise Contrastive Estimation (NCE) and use different views of an instance as positives that should be contrasted with other instances, called negatives, that are considered as noise. However, several instances in a dataset are drawn from the same distribution and share underlying semantic information. A good data representation should contain relations between the instances, or semantic similarity and dissimilarity, that contrastive learning harms by considering all negatives as noise. To circumvent this issue, we propose a novel formulation of contrastive learning using semantic similarity between instances called Similarity Contrastive Estimation (SCE). Our training objective is a soft contrastive one that brings the positives closer and estimates a continuous distribution to push or pull negative instances based on their learned similarities. We validate empirically our approach on both image and video representation learning. We show that SCE performs competitively with the state of the art on the ImageNet linear evaluation protocol for fewer pretraining epochs and that it generalizes to several downstream image tasks. We also show that SCE reaches state-of-the-art results for pretraining video representation and that the learned representation can generalize to video downstream tasks. Source code is available here: https://github.com/juliendenize/eztorch.

A knowledge-based learning framework for self-supervised pre-training towards enhanced recognition of biomedical microscopy images

Self-supervised pre-training has become the priory choice to establish reliable neural networks for automated recognition of massive biomedical microscopy images, which are routinely annotation-free, without semantics, and without guarantee of quality. Note that this paradigm is still at its infancy and limited by closely related open issues: (1) how to learn robust representations in an unsupervised manner from unlabeled biomedical microscopy images of low diversity in samples? and (2) how to obtain the most significant representations demanded by a high-quality segmentation? Aiming at these issues, this study proposes a knowledge-based learning framework (TOWER) towards enhanced recognition of biomedical microscopy images, which works in three phases by synergizing contrastive learning and generative learning methods: (1) Sample Space Diversification: Reconstructive proxy tasks have been enabled to embed a priori knowledge with context highlighted to diversify the expanded sample space; (2) Enhanced Representation Learning: Informative noise-contrastive estimation loss regularizes the encoder to enhance representation learning of annotation-free images; (3) Correlated Optimization: Optimization operations in pre-training the encoder and the decoder have been correlated via image restoration from proxy tasks, targeting the need for semantic segmentation. Experiments have been conducted on public datasets of biomedical microscopy images against the state-of-the-art counterparts (e.g., SimCLR and BYOL), and results demonstrate that: TOWER statistically excels in all self-supervised methods, achieving a Dice improvement of 1.38 percentage points over SimCLR. TOWER also has potential in multi-modality medical image analysis and enables label-efficient semi-supervised learning, e.g., reducing the annotation cost by up to 99% in pathological classification.

Smoothed noise contrastive mutual information neural estimation

Information Noise Contrastive Estimation (InfoNCE) is a popular neural estimator of mutual information (MI). While InfoNCE has demonstrated impressive results in representation learning, the estimation can be significantly off. While the original estimator is known to underestimate the MI due to the logn upper bound, where n is the sample size, we show that some subsequent fix can cause the MI estimate to overshoot apparently without any bound. We propose a novel MI variational estimator, smoothed InfoNCE, that resolves the issues by smoothing out the contrastive estimation. Experiments on high-dimensional Gaussian data confirm that the proposed estimate can break the logn curse without overshooting.

Prescriptive analytics for a maritime routing problem

Port state control (PSC) serves as the final defense against substandard ships in maritime transportation. The port state control officer (PSCO) routing problem involves selecting ships for inspection and determining the inspection sequence for available PSCOs, aiming to identify the highest number of deficiencies. Port authorities face this problem daily, making decisions without prior knowledge of ship conditions. Traditionally, a predict-then-optimize framework is employed, but its machine learning (ML) models' loss function fails to account for the impact of predictions on the downstream optimization problem, potentially resulting in suboptimal decisions. We adopt a decision-focused learning framework, integrating the PSCO routing problem into the ML models' training process. However, as the PSCO routing problem is NP-hard and plugging it into the training process of ML models requires that it be solved numerous times, computational complexity and scalability present significant challenges. To address these issues, we first convert the PSCO routing problem into a compact model using undominated inspection templates, enhancing the model's solution efficiency. Next, we employ a family of surrogate loss functions based on noise-contrastive estimation (NCE) for the ML model, requiring a solution pool treating suboptimal solutions as noise samples. This pool represents a convex hull of feasible solutions, avoiding frequent reoptimizations during the ML model's training process. Through computational experiments, we compare the predictive and prescriptive qualities of both the two-stage framework and the decision-focused learning framework under varying instance sizes. Our findings suggest that accurate predictions do not guarantee good decisions; the decision-focused learning framework's performance may depend on the optimization problem size and the training dataset size; and using a solution pool containing noise samples strikes a balance between training efficiency and decision performance.

Hierarchical Contrastive Learning for Temporal Point Processes

As an important sequential model, the temporal point process (TPP) plays a central role in real-world sequence modeling and analysis, whose learning is often based on the maximum likelihood estimation (MLE). However, due to imperfect observations, such as incomplete and sparse sequences that are common in practice, the MLE of TPP models often suffers from overfitting and leads to unsatisfactory generalization power. In this work, we develop a novel hierarchical contrastive (HCL) learning method for temporal point processes, which provides a new regularizer of MLE. In principle, our HCL considers the noise contrastive estimation (NCE) problem at the event-level and at the sequence-level jointly. Given a sequence, the event-level NCE maximizes the probability of each observed event given its history while penalizing the conditional probabilities of the unobserved events. At the same time, we generate positive and negative event sequences from the observed sequence and maximize the discrepancy between their likelihoods through the sequence-level NCE. Instead of using time-consuming simulation methods, we generate the positive and negative sequences via a simple but efficient model-guided thinning process. Experimental results show that the MLE method assisted by the HCL regularizer outperforms classic MLE and other contrastive learning methods in learning various TPP models consistently. The code is available at https://github.com/qingmeiwangdaily/HCL_TPP.

Contrastive blind denoising autoencoder for real time denoising of industrial IoT sensor data

In an industrial IoT setting, ensuring the quality of sensor data is a must when data-driven algorithms operate on the upper layers of the control system. Unfortunately, the common place in industrial facilities is to find sensor time series heavily corrupted by noise and outliers. In this work, a purely data-driven self-supervised learning-based approach based on a blind denoising autoencoder is proposed for real time denoising of industrial sensor data. The term \textit{blind} stresses that no prior knowledge about the noise is required for denoising, in contrast to typical denoising autoencoders where prior knowledge about the noise is required. Blind denoising is achieved by using a noise contrastive estimation (NCE) regularization on the latent space of the autoencoder, which not only helps to denoise but also induces a meaningful and smooth latent space. Experimental evaluation in both a simulated system and a real industrial process shows that the proposed technique outperforms classical denoising methods.

Pace-Adaptive and Noise-Resistant Contrastive Learning for Multimodal Feature Fusion

Multimodal feature fusion aims to draw complementary information from different modalities to achieve better performance. Contrastive learning is effective at discriminating coexisting semantic features (positive) from irrelative ones (negative) in multimodal signals. However, positive and negative pairs learn at separate rates, which undermines the overall performance of multimodal contrastive learning (MCL). Moreover, the learned representation model is not robust, as MCL utilizes supervision signals from potentially noisy modalities. To address these issues, a novel multimodal contrastive learning objective, Pace-adaptive and Noise-resistant Noise-Contrastive Estimation (PN-NCE), is proposed for multimodal fusion by directly using unimodal features. PN-NCE encourages the positive and negative pairs reaching to their optimal similarity scores adaptively and shows less susceptibility to noisy inputs during training. A theoretical analysis is performed on its robustness. Maximizing modality invariance information in the fused representation is expected to benefit the overall performance and therefore, an estimator that measures the difference between the fused representation and its unimodal representations is integrated into MCL to obtain a more modality-invariant fusion output. The proposed method is model-agnostic and can be adapted to various multimodal tasks. It also bears less performance degradation when reducing the number of training samples at the linear probing stage. With different networks and modality inputs from three multimodal datasets, experimental results show that PN-NCE achieves consistent enhancements compared with previous state-of-the-art approaches.

Active anomaly detection based on deep one-class classification

Active learning has been utilized as an efficient tool in building anomaly detection models by leveraging expert feedback. In an active learning framework, a model queries samples to be labeled by experts and re-trains the model with the labeled data samples. It unburdens in obtaining annotated datasets while improving anomaly detection performance. However, most of the existing studies focus on helping experts identify as many abnormal data samples as possible, which is a sub-optimal approach for one-class classification-based deep anomaly detection. In this paper, we tackle two essential problems of active learning for Deep SVDD: query strategy and semi-supervised learning method. First, rather than solely identifying anomalies, our query strategy selects uncertain samples according to an adaptive boundary. Second, we apply noise contrastive estimation in training a one-class classification model to incorporate both labeled normal and abnormal data effectively. We analyze that the proposed query strategy and semi-supervised loss individually improve an active learning process of anomaly detection and further improve when combined together on seven anomaly detection datasets.

AdaNS: Adaptive negative sampling for unsupervised graph representation learning

Recently, unsupervised graph representation learning has attracted considerable attention through effectively encoding graph-structured data without semantic annotations. To accelerate its training, noise contrastive estimation (NCE) samples uniformly negative examples to fit an unnormalized graph model. However, this uniform sampling strategy may easily lead to slow convergence, even the vanishing gradient problem. In this paper, we theoretically show that sampling those hard negatives close to the current anchor can relieve the above difficulties. With this finding, we then propose an Adaptive Negative Sampling strategy, namely AdaNS, which efficiently samples the hard negatives from the mixing distribution regarding the dimensional elements of the current node representation. Experiments show that our AdaNS sampling strategy applied on top of representative unsupervised models, e.g., DeepWalk, GraphSAGE, can outperform the existing negative sampling strategies in the tasks of node classification and visualization. This also further demonstrates that sampling those hard negatives can bring performance improvements for learning the node representations.

GraVIS: Grouping Augmented Views From Independent Sources for Dermatology Analysis.

Self-supervised representation learning has been extremely successful in medical image analysis, as it requires no human annotations to provide transferable representations for downstream tasks. Recent self-supervised learning methods are dominated by noise-contrastive estimation (NCE, also known as contrastive learning), which aims to learn invariant visual representations by contrasting one homogeneous image pair with a large number of heterogeneous image pairs in each training step. Nonetheless, NCE-based approaches still suffer from one major problem that is one homogeneous pair is not enough to extract robust and invariant semantic information. Inspired by the archetypical triplet loss, we propose GraVIS, which is specifically optimized for learning self-supervised features from dermatology images, to group homogeneous dermatology images while separating heterogeneous ones. In addition, a hardness-aware attention is introduced and incorporated to address the importance of homogeneous image views with similar appearance instead of those dissimilar homogeneous ones. GraVIS significantly outperforms its transfer learning and self-supervised learning counterparts in both lesion segmentation and disease classification tasks, sometimes by 5 percents under extremely limited supervision. More importantly, when equipped with the pre-trained weights provided by GraVIS, a single model could achieve better results than winners that heavily rely on ensemble strategies in the well-known ISIC 2017 challenge. Code is available at https://bit.ly/3xiFyjx.

Contrastive Learning of Coarse-Grained Force Fields.

Coarse-grained models have proven helpful for simulating complex systems over long time scales to provide molecular insights into various processes. Methodologies for systematic parametrization of the underlying energy function or force field that describes the interactions among different components of the system are of great interest for ensuring simulation accuracy. We present a new method, potential contrasting, to enable efficient learning of force fields that can accurately reproduce the conformational distribution produced with all-atom simulations. Potential contrasting generalizes the noise contrastive estimation method with umbrella sampling to better learn the complex energy landscape of molecular systems. When applied to the Trp-cage protein, we found that the technique produces force fields that thoroughly capture the thermodynamics of the folding process despite the use of only α-carbons in the coarse-grained model. We further showed that potential contrasting could be applied over large data sets that combine the conformational ensembles of many proteins to improve force field transferability. We anticipate potential contrasting as a powerful tool for building general-purpose coarse-grained force fields.

Dynamic Multiscale Feature Fusion Method for Underwater Target Recognition

The feature information of small-scale targets is seriously missing under the interference of complex underwater terrain and light refraction. Moreover, the unbalanced distribution of underwater target samples can also affect the accuracy of spatial semantic feature extraction. Aiming at the above problems, this paper proposes a dynamic multiscale feature fusion method for underwater target recognition. Firstly, this paper uses multiscale info noise contrastive estimation (MS-InfoNCE) loss to extract the significant features of the target at 4 scales. Secondly, the method learns the spatial semantic features of the target through a dynamic conditional probability matrix. Finally, this paper designs different feature fusion mechanisms for different scale targets, dynamically fusing multiscale significant features and spatial semantic features to recognize underwater weak targets. The experimental results show that the recognition accuracy of the proposed algorithm is 1.38% higher than that of the existing algorithm when recognizing underwater distorted targets.

Ranking Info Noise Contrastive Estimation: Boosting Contrastive Learning via Ranked Positives

This paper introduces Ranking Info Noise Contrastive Estimation (RINCE), a new member in the family of InfoNCE losses that preserves a ranked ordering of positive samples. In contrast to the standard InfoNCE loss, which requires a strict binary separation of the training pairs into similar and dissimilar samples, RINCE can exploit information about a similarity ranking for learning a corresponding embedding space. We show that the proposed loss function learns favorable embeddings compared to the standard InfoNCE whenever at least noisy ranking information can be obtained or when the definition of positives and negatives is blurry. We demonstrate this for a supervised classification task with additional superclass labels and noisy similarity scores. Furthermore, we show that RINCE can also be applied to unsupervised training with experiments on unsupervised representation learning from videos. In particular, the embedding yields higher classification accuracy, retrieval rates and performs better on out-of-distribution detection than the standard InfoNCE loss.