Negative Pairs Research Articles

Contrastive Unsupervised Representation Learning With Optimize-Selected Training Samples.

Contrastive unsupervised representation learning (CURL) is a technique that seeks to learn feature sets from unlabeled data. It has found widespread and successful application in unsupervised feature learning, with the design of positive and negative pairs serving as the type of data samples. While CURL has seen empirical successes in recent years, there is still room for improvement in terms of the pair data generation process. This includes tasks such as combining and re-filtering samples, or implementing transformations among positive/negative pairs. We refer to this as the sample selection process. In this article, we introduce an optimized pair-data sample selection method for CURL. This method efficiently ensures that the two types of sampled data (similar pair and dissimilar pair) do not belong to the same class. We provide a theoretical analysis to demonstrate why our proposed method enhances learning performance by analyzing its error probability. Furthermore, we extend our proof into PAC-Bayes generalization to illustrate how our method tightens the bounds provided in previous literature. Our numerical experiments on text/image datasets show that our method achieves competitive accuracy with good generalization bounds.

Dual-Stream Contrastive Learning for Compositional Zero-Shot Recognition

Compositional Zero-Shot learning (CZSL) requires recognizing unseen attribute-object compositions using observed visual primitives attributes and objects in a training set, which is a critical capacity for learning systems because the long tail of new combinations dominates the distribution in the real world. However, CZSL is a challenging problem because learning systems tend to learn the dependencies between objects and attributes, which is not conducive to composition classification, and incorrect dependencies will mislead the classification of new combinations of known attributes and objects. This paper primarily introduces a novel yet effective dual-stream contrastive learning method with two main objectives: making the learned representations discriminative and transferring knowledge more efficiently from seen to unseen compositions. Specifically, we generate positive and negative pairs based on the similarity of different concepts (attributes and objects), independently capturing the discriminative representations of concepts. Meanwhile, unlike existing contrastive methods that select negative samples randomly, we construct confusable compositional representations as the negatives to explore the intrinsic relevance between attributes and objects, which can improve the generalization from seen to unseen compositions. Experimental results on two benchmarks show that the proposed method outperforms the state-of-the-arts.

COFT-AD: COntrastive Fine-Tuning for Few-Shot Anomaly Detection.

Existing approaches towards anomaly detection (AD) often rely on a substantial amount of anomaly-free data to train representation and density models. However, large anomaly-free datasets may not always be available before the inference stage; in which case an anomaly detection model must be trained with only a handful of normal samples, a.k.a. few-shot anomaly detection (FSAD). In this paper, we propose a novel methodology to address the challenge of FSAD which incorporates two important techniques. Firstly, we employ a model pre-trained on a large source dataset to initialize model weights. Secondly, to ameliorate the covariate shift between source and target domains, we adopt contrastive training to fine-tune on the few-shot target domain data. To learn suitable representations for the downstream AD task, we additionally incorporate cross-instance positive pairs to encourage a tight cluster of the normal samples, and negative pairs for better separation between normal and synthesized negative samples. We evaluate few-shot anomaly detection on 3 controlled AD tasks and 4 real-world AD tasks to demonstrate the effectiveness of the proposed method.

Clustering Enhanced Multiplex Graph Contrastive Representation Learning.

Multiplex graph representation learning has attracted considerable attention due to its powerful capacity to depict multiple relation types between nodes. Previous methods generally learn representations of each relation-based subgraph and then aggregate them into final representations. Despite the enormous success, they commonly encounter two challenges: 1) the latent community structure is overlooked and 2) consistent and complementary information across relation types remains largely unexplored. To address these issues, we propose a clustering-enhanced multiplex graph contrastive representation learning model (CEMR). In CEMR, by formulating each relation type as a view, we propose a multiview graph clustering framework to discover the potential community structure, which promotes representations to incorporate global semantic correlations. Moreover, under the proposed multiview clustering framework, we develop cross-view contrastive learning and cross-view cosupervision modules to explore consistent and complementary information in different views, respectively. Specifically, the cross-view contrastive learning module equipped with a novel negative pairs selecting mechanism enables the view-specific representations to extract common knowledge across views. The cross-view cosupervision module exploits the high-confidence complementary information in one view to guide low-confidence clustering in other views by contrastive learning. Comprehensive experiments on four datasets confirm the superiority of our CEMR when compared to the state-of-the-art rivals.

Nonparametric Clustering-Guided Cross-View Contrastive Learning for Partially View-Aligned Representation Learning.

With the increasing availability of multi-view data, multi-view representation learning has emerged as a prominent research area. However, collecting strictly view-aligned data is usually expensive, and learning from both aligned and unaligned data can be more practicable. Therefore, Partially View-aligned Representation Learning (PVRL) has recently attracted increasing attention. After aligning multi-view representations based on their semantic similarity, the aligned representations can be utilized to facilitate downstream tasks, such as clustering. However, existing methods may be constrained by the following limitations: 1) They learn semantic relations across views using the known correspondences, which is incomplete and the existence of false negative pairs (FNP) can significantly impact the learning effectiveness; 2) Existing strategies for alleviating the impact of FNP are too intuitive and lack a theoretical explanation of their applicable conditions; 3) They attempt to find FNP based on distance in the common space and fail to explore semantic relations between multi-view data. In this paper, we propose a Nonparametric Clustering-guided Cross-view Contrastive Learning (NC3L) for PVRL, in order to address the above issues. Firstly, we propose to estimate the similarity matrix between multi-view data in the marginal cross-view contrastive loss to approximate the similarity matrix of supervised contrastive learning (CL). Secondly, we establish the theoretical foundation for our proposed method by analyzing the error bounds of the loss function and its derivatives between our method and supervised CL. Thirdly, we propose a Deep Variational Nonparametric Clustering (DeepVNC) by designing a deep reparameterized variational inference for Dirichlet process Gaussian mixture models to construct cluster-level similarity between multi-view data and discover FNP. Additionally, we propose a reparameterization trick to improve the robustness and the performance of our proposed CL method. Extensive experiments on four widely used benchmark datasets show the superiority of our proposed method compared with state-of-the-art methods.

An adversarial contrastive autoencoder for robust multivariate time series anomaly detection

Multivariate time series (MTS), whose patterns change dynamically, often have complex temporal and dimensional dependence. Most existing reconstruction-based MTS anomaly detection methods only learn the point-wise information while ignoring the overall trend of time series, resulting in their incompetence in extracting high-level semantic information. Although a few contrastive learning-based approaches have been proposed recently to solve this problem, they forcibly increase the difference between the features of normal data, leading to the loss of useful information. This paper proposes an adversarial contrastive autoencoder (ACAE) for MTS anomaly detection. ACAE conducts feature combination and decomposition as the contrastive learning proxy task, which introduces adversarial training to learn the transformation-invariant representation of data, achieving a robust representation of MTS. Firstly, ACAE constructs positive and negative sample pairs through the multi-scale timestamp mask and random sampling. Secondly, the features of the original samples are combined with those of the positive and negative samples to generate the positive and negative composite features. Finally, ACAE trains the encoder and discriminator to decompose the negative composite features cooperatively to decrease the similarity between the features of negative pairs. In contrast, it adversarially decomposes the positive composite features to increase the similarity between the features of positive pairs. Experimental results show that ACAE outperforms 14 state-of-the-art baselines on five real-world datasets from different fields.

ABUS tumor segmentation via decouple contrastive knowledge distillation

Objective. In recent years, deep learning-based methods have become the mainstream for medical image segmentation. Accurate segmentation of automated breast ultrasound (ABUS) tumor plays an essential role in computer-aided diagnosis. Existing deep learning models typically require a large number of computations and parameters. Approach. Aiming at this problem, we propose a novel knowledge distillation method for ABUS tumor segmentation. The tumor or non-tumor regions from different cases tend to have similar representations in the feature space. Based on this, we propose to decouple features into positive (tumor) and negative (non-tumor) pairs and design a decoupled contrastive learning method. The contrastive loss is utilized to force the student network to mimic the tumor or non-tumor features of the teacher network. In addition, we designed a ranking loss function based on ranking the distance metric in the feature space to address the problem of hard-negative mining in medical image segmentation. Main results. The effectiveness of our knowledge distillation method is evaluated on the private ABUS dataset and a public hippocampus dataset. The experimental results demonstrate that our proposed method achieves state-of-the-art performance in ABUS tumor segmentation. Notably, after distilling knowledge from the teacher network (3D U-Net), the Dice similarity coefficient (DSC) of the student network (small 3D U-Net) is improved by 7%. Moreover, the DSC of the student network (3D HR-Net) reaches 0.780, which is very close to that of the teacher network, while their parameters are only 6.8% and 12.1% of 3D U-Net, respectively. Significance. This research introduces a novel knowledge distillation method for ABUS tumor segmentation, significantly reducing computational demands while achieving state-of-the-art performance. The method promises enhanced accuracy and feasibility for computer-aided diagnosis in diverse imaging scenarios.

Contrastive variational information bottleneck for aspect-based sentiment analysis

Deep learning techniques have dominated the literature on aspect-based sentiment analysis (ABSA), achieving state-of-the-art performance. However, deep models generally suffer from spurious correlations between input features and output labels, which significantly hurts the robustness and generalization capability. In this paper, we propose to reduce spurious correlations for ABSA, via a novel Contrastive Variational Information Bottleneck framework (called CVIB). The proposed CVIB framework is composed of an original network and a self-pruned network, and these two networks are optimized simultaneously via contrastive learning. Concretely, we employ the Variational Information Bottleneck (VIB) principle to learn an informative and compressed network (self-pruned network) from the original network, which discards the superfluous patterns or spurious correlations between input features and prediction labels. Then, self-pruning contrastive learning is devised to pull together semantically similar positive pairs and push away dissimilar pairs, where the representations of the anchor learned by the original and self-pruned networks respectively are regarded as a positive pair while the representations of two different sentences within a mini-batch are treated as a negative pair. To verify the effectiveness of our CVIB method, we conduct extensive experiments on five benchmark ABSA datasets. The experimental results show that our approach achieves better performance than the strong competitors in terms of overall prediction performance, robustness, and generalization.

Deep Contrastive Clustering via Hard positive sample Debiased

Deep graph clustering aims to reveal the underlying information of the graph and provide accurate embedding for the node clustering task, in which contrastive learning plays an important role. However, the commonly used contrastive loss function incorrectly classifies elements outside the diagonal of the cross view as negative samples, which contain a large number of positive sample pairs. In order to overcome the above problems, we propose a new deep graph contrastive clustering method, which combines hard positive sample debiasing and sample pair weighting, and improves the recognition ability of the network by removing the potential positive sample pairs and hard sample pair weighting in the negative sample pair of the loss function. More specifically, we developed a symmetric graph neural network to encode node representations. Using two sets of node representations, the correctness of negative cases is increased by clustering to generate high-confidence pseudo-label pairs for labels and confidence. The similarity distribution differences are weighted by adapting to different dataset samples to improve the sample recognition ability. To verify the efficacy of our DCHD, we compare it to existing state-of-the-art methods for node clustering tasks on six real-world datasets. Overall, the experimental results show that our proposed method outperforms current state-of-the-art graph clustering methods.

Contrastive representation learning in recommendation systems--The investigation of the performance of the self-supervised learning in large-scale recommendation systems

Self-supervised learning (SSL) has been proposed in machine learning projects for its convenience of reducing self-labeled datasets in recent years. However, the implementation of SSL in large-scale recommendation systems has lagged behind the evolution because of their scarce and tailed characteristics. In 2021, an article proposed the use of SSL in recommendation systems to pursue an improvement in the performance of recommender models. This article is built on this previous investigation and aims to further explore the role of SSL in recommendation systems and to investigate an improvement of the models efficiency. To answer research questions, this paper tests three models with different numbers of towers to discover the best performance of the use of SSL in recommender models. Consequently, it is found that implementing SSL on the item side only (two-tower DNNs) produced the best result. Then, when constructing the two-tower DNNs model, this article examines different numbers of negative pairs to change the InfoNCE loss to investigate a tradeoff between the number of positive and negative samples in the performance of the model. As a result, it turns out to be a weak correlation between this ratio and the performance; Hence, it is concluded that the change of the number of positive and negative samples would not necessarily affect the two-tower DNNs model. In our experiential stage, this paper uses a real-world dataset with 100k training samples to testify and compare our results.

Contrastive Self-Supervised Clustering for Specific Emitter Identification

Specific Emitter Identification (SEI) is crucial for attacking and defending Internet of Things (IoT) devices in untrusted scenarios or battlefield environments. However, existing SEI methods usually require annotation information, which is often unavailable in non-cooperative communications and untrusted scenarios. In this paper, we propose a signal contrastive self-supervised clustering (SCSC) method for unsupervised SEI applications. First, we propose SCSC with 1D fingerprint pyramid feature extractor (1D-FPFE) for obtaining hierarchical subtle features of emitter signals. Then, we propose a bit-pulse selection (BPS) strategy and several signal data augmentation methods. By constructing signal positive and negative instance pairs through data augmentation, our approach generates cluster preference representations in a contrastive self-supervised learning manner. Extensive experimental results based on communication burst emitter dataset show that SCSC achieves an accuracy improvement of about 26% over the current best communication signal clustering algorithm. Moreover, SCSC also exhibits good performance and generalization for 30 emitter clustering and few-shot unlabeled signal clustering.

Progressive Hard Negative Masking: From Global Uniformity to Local Tolerance

Unsupervised contrastive learning has recently become increasingly popular due to its amazing performance without the need for costly annotations. However, indiscriminate sampling of negative pairs is accompanied by the uniformity-tolerance dilemma, which is especially serious in node-level graph contrastive learning due to the smoothing property of graph convolutional operators. Previous negative mining strategies that either overly emphasize hard negatives or rely on precise distribution estimation can make minor improvements or even degrade the performance in such a case. In this paper, we investigate the role of hard negatives in the uniformity-tolerance dilemma and propose a novel contrastive objective with a progressive hard negative masking scheme. The proposed objective, as an asymptotically-tightened lower bound of mutual information, is theoretically and empirically demonstrated to be capable of allowing higher local tolerance and stronger contrastive effects, thus leading to higher-quality embedding distributions and considerable performance improvement in downstream node classification tasks.

A creative wide-frequency and large-amplitude vibration isolator design method based on magnetic negative stiffness and displacement amplification mechanism

Negative stiffness vibration isolation devices have attracted significant attention and research interest for their ability to enhance the low-frequency vibration isolation performance of passive vibration isolators. However, their narrow linear negative stiffness region has hindered their broader application. To address this limitation, a novel design method for vibration isolators is proposed, integrating the permanent magnetic negative stiffness device (NSD) with the displacement amplification mechanism (DAM). The paper begins by presenting a specific vibration isolator design method and analyzing the kinematic relationship. Next, a single-degree-of-freedom (SDOF) nonlinear equivalent model of the vibration isolator is established using the pseudo-rigid-body model (PRBM) approach. Numerical simulations are then conducted to obtain the transmissibility of the isolator. Furthermore, two isolators—one with the displacement amplification effect and another without—are manufactured, and experimental investigations are carried out. The experimental results confirm that the incorporation of the displacement amplification mechanism effectively expands the linear negative stiffness region of the negative stiffness magnet pair, enabling wide-frequency and large-amplitude vibration isolation. Notably, these experimental findings align well with the simulation results. In summary, this paper proposes a creative design method for wide frequency and large amplitude vibration isolators, and proves it through theoretical analysis and experimental validation.

Boosting semi-supervised learning with Contrastive Complementary Labeling

Semi-supervised learning (SSL) approaches have achieved great success in leveraging a large amount of unlabeled data to learn deep models. Among them, one popular approach is pseudo-labeling which generates pseudo labels only for those unlabeled data with high-confidence predictions. As for the low-confidence ones, existing methods often simply discard them because these unreliable pseudo labels may mislead the model. Unlike existing methods, we highlight that these low-confidence data can be still beneficial to the training process. Specifically, although we cannot determine which class a low-confidence sample belongs to, we can assume that this sample should be very unlikely to belong to those classes with the lowest probabilities (often called complementary classes/labels). Inspired by this, we propose a novel Contrastive Complementary Labeling (CCL) method that constructs a large number of reliable negative pairs based on the complementary labels and adopts contrastive learning to make use of all the unlabeled data. Extensive experiments demonstrate that CCL significantly improves the performance on top of existing advanced methods and is particularly effective under the label-scarce settings. For example, CCL yields an improvement of 2.43% over FixMatch on CIFAR-10 only with 40 labeled data.

Self-Supervised Clustering Models Based on BYOL Network Structure

Contrastive-based clustering models usually rely on a large number of negative pairs to capture uniform representations, which requires a large batch size and high computational complexity. In contrast, some self-supervised methods perform non-contrastive learning to capture discriminative representations only with positive pairs, but suffer from the collapse of clustering. To solve these issues, a novel end-to-end self-supervised clustering model is proposed in this paper. The basic self-supervised learning network is first modified, followed by the incorporation of a Softmax layer to obtain cluster assignments as data representation. Then, adversarial learning on the cluster assignments is integrated into the methods to further enhance discrimination across different clusters and mitigate the collapse between clusters. To further encourage clustering-oriented guidance, a new cluster-level discrimination is assembled to promote clustering performance by measuring the self-correlation between the learned cluster assignments. Experimental results on real-world datasets exhibit better performance of the proposed model compared with the existing deep clustering methods.

GLRP: Global and local contrastive learning based on relative position for medical image segmentation on cardiac MRI

AbstractContrastive learning, as an unsupervised technique, is widely employed in image segmentation to enhance segmentation performance even when working with small labeled datasets. However, generating positive and negative data pairs for medical image segmentation poses a challenge due to the presence of similar tissues and organs across different slices in datasets. To tackle this issue, we propose a novel contrastive learning strategy that leverages the relative position differences between image slices. Additionally, we combine global and local features to address this problem effectively. In order to enhance segmentation accuracy and reduce isolated mis‐segmented regions, we employ a two‐dimensional fully connected conditional random field for iterative optimization of the segmentation results. With only 10 labeled samples, our proposed method is able to achieve average dice scores of 0.876 and 0.899 on the public and private dataset heart segmentation tasks, surpassing the PCL method's 0.801 and 0.852. Experimental results on both public and private MRI datasets demonstrate that our proposed method yields significant improvements in medical segmentation tasks with limited annotated samples, outperforming existing semi‐supervised and self‐supervised techniques.

An Iterative Semi-supervised Approach with Pixel-wise Contrastive Loss for Road Extraction in Aerial Images

Extracting roads in aerial images has numerous applications in artificial intelligence and multimedia computing, including traffic pattern analysis and parking space planning. Learning deep neural networks, though very successful, demand vast amounts of high-quality annotations, of which acquisition is time-consuming and expensive. In this work, we propose a semi-supervised approach for image-based road extraction in which only a small set of labeled images are available for training to address this challenge. We design a pixel-wise contrastive loss to self-supervise the network training to utilize the large corpus of unlabeled images. The key idea is to identify pairs of overlapping image regions (positive) or non-overlapping image regions (negative) and encourage the network to make similar outputs for positive pairs or dissimilar outputs for negative pairs. We also develop a negative sampling strategy to filter false-negative samples during the process. An iterative procedure is introduced to apply the network over raw images to generate pseudo-labels, filter and select high-quality labels with the proposed contrastive loss, and retrain the network with the enlarged training dataset. We repeat these iterative steps until convergence. We validate the effectiveness of the proposed methods by performing extensive experiments on the public SpaceNet3 and DeepGlobe Road datasets. Results show that our proposed method achieves state-of-the-art results on public image segmentation benchmarks and significantly outperforms other semi-supervised methods.

Task-oriented contrastive learning for unsupervised domain adaptation

Unsupervised domain adaptation aims to transfer the knowledge learned from the labeled source domain to the unlabeled target domain, thereby improving the classification performance of the target domain. Recent methods use contrastive learning to optimize this task, however, these methods only focus on contrastive learning for aspects of domain alignment, which do not actively serve the classification task, resulting in suboptimal solution. To this end, we propose Task-oriented Contrastive Learning for Unsupervised Domain Adaptation (TOCL), which addresses the inconsistency problem of optimizing contrastive learning and classification tasks. We perform feature weighting on the source domain images and extract the task-related features containing semantic information after data augmentation as the source domain positive and negative sample pairs for contrastive learning. At the same time, a delimitation discriminator is introduced to maximize the output difference between two types of random data-augmented samples of the target domain to detect data-augmented samples that are not conducive to classification, and then minimize the difference through the feature generator to generate effective data-augmented features (good for classification). Our extensive experiments on several public datasets show that TOCL has good adaptability and effectiveness, which improves the accuracy of classification results.

Joint unsupervised contrastive learning and robust GMM for text clustering

Text clustering aims to organize a vast collection of documents into meaningful and coherent clusters, thereby facilitating the extraction of valuable insights. While current frameworks for text clustering try to minimize the anisotropy of pre-trained language models through contrastive learning of text embeddings, the approach of treating in-batch samples as negatives is suboptimal. The K-means algorithm offers a way to sample both hard negatives and false negatives. However, relying solely on a single measure of semantic similarity between distributions and using coarse-grained weighting for negative pairs may potentially limit performance. Furthermore, considering the very similar distribution in text clusters due to rich semantics, the Mahalanobis distance-based Gaussian Mixture Model (GMM) is prone to falling into local optima due to one Gaussian model, having a smaller weight, may gradually merging into another during the parameter evaluation by the EM algorithm. To tackle these challenges, we propose a model named JourTC: Joint unsupervised contrastive learning and robust GMM for Text Clustering. In the contrastive learning phase, hard negatives, potential false negatives, and their corresponding global similarity-aware weights are determined through posterior probabilities derived from a Robust GMM (RGMM). This RGMM utilizes the entropy of each individual Gaussian model as a metric and adaptively adjusts the posterior probabilities of samples based on the Gaussian models with both maximum and minimum entropy to diminish the influence of low-entropy Gaussian models. Extensive experiments have shown that JourTC can be seamlessly integrated into existing text clustering frameworks, leading to a notable improvement in accuracy. Our code is publicly available.11http://github.com/nickhcx/JourTC.

Dual-stream-based dense local features contrastive learning for semi-supervised medical image segmentation

Most existing semi-supervised methods lack the ability to optimize feature distribution, which leads to poor inter-class separability and suboptimal decision boundaries during prediction. Additionally, the contours of targets in medical images are often ambiguous, making it challenging to identify contour features and potentially leading to segmentation errors. To address these issues, we propose a novel semi-supervised segmentation framework called Dual-stream-based Dense Local Features Contrastive Learning. Specifically, our approach employs a prediction stream to generate a predicted segmentation map and Signed Distance Map, which mutually reinforce each other to enhance the exploitation of contour features. We also introduce a feature stream that incorporates a Dense Local Features Contrastive Learning module, consisting of Sampling by Shape Info (SSI) and Dense Local Features Contrast (DLFC). The SSI module utilizes local shape information to create a Balanced Coefficient, which guides the sampling of positive and negative pairs between dense local features. The DLFC module effectively enhances intra-class compactness and inter-class separability by utilizing contrastive learning and we additionally introduce consistency regularization to promote the efficiency of contrastive learning. Extensive experiments on medical image segmentation benchmark datasets demonstrate our approach achieves significant improvements against existing state-of-the-art methods.