Loss Of Contrast Research Articles

PICL: Physics informed contrastive learning for partial differential equations

Neural operators have recently grown in popularity as Partial Differential Equation (PDE) surrogate models. Learning solution functionals, rather than functions, has proven to be a powerful approach to calculate fast, accurate solutions to complex PDEs. While much work has been performed evaluating neural operator performance on a wide variety of surrogate modeling tasks, these works normally evaluate performance on a single equation at a time. In this work, we develop a novel contrastive pretraining framework utilizing generalized contrastive loss that improves neural operator generalization across multiple governing equations simultaneously. Governing equation coefficients are used to measure ground-truth similarity between systems. A combination of physics-informed system evolution and latent-space model output is anchored to input data and used in our distance function. We find that physics-informed contrastive pretraining improves accuracy for the Fourier neural operator in fixed-future and autoregressive rollout tasks for the 1D and 2D heat, Burgers’, and linear advection equations.

Contrastive learning for neural fingerprinting from limited neuroimaging data

IntroductionNeural fingerprinting is a technique used to identify individuals based on their unique brain activity patterns. While deep learning techniques have been demonstrated to outperform traditional correlation-based methods, they often require retraining to accommodate new subjects. Furthermore, the limited availability of samples in neuroscience research can impede the quick adoption of deep learning methods, presenting a challenge for their broader application in neural fingerprinting.MethodsThis study addresses these challenges by using contrastive learning to eliminate the need for retraining with new subjects and developing a data augmentation methodology to enhance model robustness in limited sample size conditions. We utilized the LEMON dataset, comprising 3 Tesla MRI and resting-state fMRI scans from 138 subjects, to compute functional connectivity as a baseline for fingerprinting performance based on correlation metrics. We adapted a recent deep learning model by incorporating data augmentation with short random temporal segments for training and reformulated the fingerprinting task as a contrastive problem, comparing the efficacy of contrastive triplet loss against conventional cross-entropy loss.ResultsThe results of this study confirm that deep learning methods can significantly improve fingerprinting performance over correlation-based methods, achieving an accuracy of about 98% in identifying a single subject out of 138 subjects utilizing 39 different functional connectivity profiles.DiscussionThe contrastive method showed added value in the “leave subject out” scenario, demonstrating flexibility comparable to correlation-based methods and robustness across different data sizes. These findings suggest that contrastive learning and data augmentation offer a scalable solution for neural fingerprinting, even with limited sample sizes.

HRCUNet: Hierarchical Region Contrastive Learning for Segmentation of Breast Tumors in DCE‐MRI

ABSTRACTSegmenting breast tumors from dynamic contrast‐enhanced magnetic resonance images is a critical step in the early detection and diagnosis of breast cancer. However, this task becomes significantly more challenging due to the diverse shapes and sizes of tumors, which make it difficult to establish a unified perception field for modeling them. Moreover, tumor regions are often subtle or imperceptible during early detection, exacerbating the issue of extreme class imbalance. This imbalance can lead to biased training and challenge accurately segmenting tumor regions from the predominant normal tissues. To address these issues, we propose a hierarchical region contrastive learning approach for breast tumor segmentation. Our approach introduces a novel hierarchical region contrastive learning loss function that addresses the class imbalance problem. This loss function encourages the model to create a clear separation between feature embeddings by maximizing the inter‐class margin and minimizing the intra‐class distance across different levels of the feature space. In addition, we design a novel Attention‐based 3D Multi‐scale Feature Fusion Residual Module to explore more granular multi‐scale representations to improve the feature learning ability of tumors. Extensive experiments on two breast DCE‐MRI datasets demonstrate that the proposed algorithm is more competitive against several state‐of‐the‐art approaches under different segmentation metrics.

Residual trio feature network for efficient super-resolution

Deep learning-based approaches have demonstrated impressive performance in single-image super-resolution (SISR). Efficient super-resolution compromises the reconstructed image’s quality to have fewer parameters and Flops. Ensured efficiency in image reconstruction and improved reconstruction quality of the model are significant challenges. This paper proposes a trio branch module (TBM) based on structural reparameterization. TBM achieves equivalence transformation through structural reparameterization operations, which use a complex network structure in the training phase and convert it to a more lightweight structure in the inference, achieving efficient inference while maintaining accuracy. Based on the TBM, we further design a lightweight version of the enhanced spatial attention mini (ESA-mini) and the residual trio feature block (RTFB). Moreover, the multiple RTFBs are combined to construct the residual trio network (RTFN). Finally, we introduce a localized contrast loss for better applicability to the super-resolution task, which enhances the reconstruction quality of the super-resolution model. Experiments show that the RTFN framework proposed in this paper outperforms other state-of-the-art efficient super-resolution methods in terms of inference speed and reconstruction quality.

Photometric Redshift Estimation of Quasars by a Cross-modal Contrast Learning Method

Estimating photometric redshifts (photo-z) of quasars is crucial for measuring cosmic distances and monitoring cosmic evolution. While numerous point estimation methods have successfully determined photo-z, they often struggle with the inherently ill-posed nature of the problem and frequently overlook significant morphological features in the probability density functions (pdfs) of photo-z, such as calibration and sharpness. To address these challenges, we introduce a cross-modal contrastive learning probabilistic model that employs adversarial training, contrastive loss functions, and a mixture density network to estimate the pdf of photo-z. This method facilitates the conversion between multiband photometric data attributes, such as magnitude and color, and photometric image features, while extracting features invariant across modalities. We utilize the continuous ranked probability score (CRPS) and the probability integral transform (PIT) as metrics to assess the quality of the pdf. Our approach demonstrates robust performance across various survey bands, image qualities, and redshift distributions. Specifically, in a comprehensive data set from the Sloan Digital Sky Survey and the Wide-field Infrared Survey Explorer (WISE) survey, our probabilistic model achieved a CRPS of 0.1187. Additionally, in a combined data set from SkyMapper and WISE, it reached a CRPS of 0.0035. Our probabilistic model also produced well-calibrated PIT histograms for both data sets, indicating nearly uniform distributions. We further tested our approach in classification tasks within the SkyMapper data set. Despite the absence of u, v, and g bands, it effectively distinguished between quasars, galaxies, and stars with an accuracy of 98.96%. This versatile method can be extended to other scenarios, such as analyzing extended sources like galaxies, across different surveys and varying redshift distributions.

Contrastive learning-enabled digital twin framework for fault diagnosis of rolling bearing

Abstract Rolling bearings are essential components in various industrial machines, and their failures can lead to significant downtime and maintenance costs. Traditional data-driven fault diagnosis methods often require extensive fault datasets for training, which may not always be available in critical industrial scenarios, limiting their practicality. Digital twins, virtual representations of physical entities reflecting their operational conditions, offer a promising solution for the fault diagnosis of rolling bearings with limited fault data. In this paper, we propose a novel digital twin-driven framework to address the challenge of limited training data in rolling bearing fault diagnosis. Firstly, a virtual bearing simulation model is used to generate the simulated data. Subsequently, a transformer-based network is introduced to learn the discrepancy features from the raw data. Then, a Maximum Mean Discrepancy loss and a supervised contrastive learning loss for raw and augmentation data are established to achieve global domain alignment and instance-based domain alignment. Finally, an unsupervised contrastive learning loss for the augmentation data of the target domain is established to further improve the diagnostic performance. In five cross-domain fault diagnosis tasks representing real industrial scenarios set in this study, the proposed method achieved an average diagnostic accuracy of 84.40%, exceeding two existing advanced domain adaptation methods by more than 10%. The experimental results demonstrate that the proposed method achieves high diagnostic performance in real industrial scenarios where labeled data is lacking. This shows its significant benefits for monitoring the condition of critical bearings.

Few‐Shot Contrastive Learning‐Based Multi‐Round Dialogue Intent Classification Method

ABSTRACTTraditional text classification models face challenges in handling long texts and understanding topic transitions in dialogue scenarios, leading to suboptimal performance in automatic speech recognition (ASR)‐based multi‐round dialogue intent classification. In this article, we propose a few‐shot contrastive learning‐based multi‐round dialogue intent classification method. First, the ASR texts are partitioned, and role‐based features are extracted using a Transformer encoder. Second, refined sample pairs are forward‐propagated, adversarial samples are generated by perturbing word embedding matrices and contrastive loss is applied to positive sample pairs. Then, positive sample pairs are input into a multi‐round reasoning module to learn semantic clues from the entire scenario through multiple dialogues, obtain reasoning features, input them into a classifier to obtain classification results, and calculate multi‐task loss. Finally, a prototype update module (PUM) is introduced to rectify the biased prototypes by using gated recurrent unit (GRU) to update the prototypes stored in the memory bank and few‐shot learning (FSL) task. Experimental evaluations demonstrate that the proposed method outperforms state‐of‐the‐art methods on two public datasets (DailyDialog and CM) and a private real‐world dataset.

Attack-Defending Contrastive Learning for Volumetric Medical Image Zero-Watermarking

Zero-watermarking is an emerging distortion-free copyright protection method for volumetric medical images. However, achieving both robustness against various malicious attacks and distinguishability between individual images remains challenging. In this paper, we propose a novel attack-defending contrastive learning zero-watermarking scheme (ADCL-ZW) to tackle the above challenge using deep learning-based representations. In our approach, we design an attack-defending data enrichment mechanism to enhance the watermarking robustness by generating a large number of image samples under various watermarking attacks. Subsequently, features for both watermarking distinguishability and robustness are enhanced through application of a contrastive loss. In particular, we implement a dual-stream Siamese network architecture to effectively handle both signal attacks and geometric attacks in order to enhance the wartermarking performance. Experimental results demonstrate that ADCL-ZW achieves stronger watermarking robustness and a better tradeoff between watermarking robustness and distinguishability compared with state-of-the art zero-watermarking methods. One of the highlighted metrics the false negative rate of ADCL-ZW achieves 0.01 when a fixed false positive rate is set to 1%, which is more than 13.3 times better than the benchmark methods.

Multi-view graph contrastive representation learning for bundle recommendation

Bundle recommendation can recommend a collection of associated items that can be consumed together to a user rather than recommending these items separately, making it extremely suitable for some scenarios such as product bundle recommendation and game bundle recommendation. Recent bundle recommendation approaches consider auxiliary data to mitigate sparse user-bundle interactions. However, these approaches obtain the node embeddings directly from the established user-bundle graph and do not explicitly exploit the relationships between users (bundles) when constructing recommendation models. Moreover, bundle recommendation approaches based on graph contrastive learning usually construct contrastive views by randomly discarding nodes (edges) in the graph, while discarding some essential nodes or edges will destroy the structure of the original graph, thereby deteriorating the quality of the learned node embeddings. Aiming at these limitations, we propose a bundle recommendation approach based on multi-view graph contrastive representation learning. First, we present a multi-view modeling method to model the relations between entities as several views from different perspectives. These views serve as inputs of graph neural networks for graph representation learning and provide contrastive views for the contrastive learning tasks. Second, we propose a novel framework for bundle recommendation. This framework obtains the user (bundle) embeddings from different views by performing multi-view graph representation learning and enhances the learned user and bundle embeddings through a two-level contrastive learning strategy. On this basis, the enhanced user (bundle) embeddings are fused for prediction. Finally, we design a joint optimization objective to optimize the model parameters, combining the prediction loss that supports multiple negative samples and the contrastive losses. Experiments on the Netease and Youshu datasets reveal that our approach outperforms the state-of-the-art (SOTA) baselines. Furthermore, the average improvements of Recall@K and NDCG@K of our approach over the SOTA baselines are approximately 3.38% and 2.80% on Netease and 3.94% and 4.84% on Youshu.

OPTIMIZING AUTHENTICATION SECURITY IN INTELLIGENT SYSTEMS THROUGH VISUAL BIOMETRICS FOR ENHANCED EFFICIENCY

Context. The primary objective of this article is to explore aspects related to ensuring security and enhancing the efficiency of authentication processes in intelligent systems through the application of visual biometrics. The focus is on advancing and refining authentication systems by employing sophisticated biometric identification methods. Objective. A specialized intelligent system has been developed, utilizing a Siamese neural network to establish secure user authentication within the existing system. Beyond incorporating fundamental security measures such as hashing and secure storage of user credentials, the contemporary significance of implementing two-factor authentication is underscored. This approach significantly fortifies user data protection, thwarting most contemporary hacking methods and safeguarding against data breaches. The study acknowledges certain limitations in its approach, possibly affecting the generalizability of the findings. These limitations provide avenues for future research and exploration, contributing to the ongoing evolution of authentication methodologies in intelligent systems. Method. The two-factor authentication system integrates facial recognition technology, employing visual biometrics for heightened security compared to alternative two-factor authentication methods. Various implementations of the Siamese neural network, utilizing Contrastive loss function and Triplet loss function, were evaluated. Subsequently, a neural network employing the Triplet loss function was implemented and trained. Results. The article emphasizes the practical implications of the developed intelligent system, showcasing its effectiveness in minimizing the risk of unauthorized access to user accounts. The integration of contemporary authentication methodologies ensures a secure and robust user authentication process. Conclusions. The implementation of facial recognition technology in authentication processes has broader social implications. It contributes to a more secure digital environment by preventing unauthorized access, ultimately safeguarding user privacy and data. The study’s originality lies in its innovative approach to authentication, utilizing visual biometrics within a Siamese neural network framework. The developed intelligent system represents a valuable contribution to the field, offering an effective and contemporary solution to user authentication challenges.

Consistency-driven feature scoring and regularization network for visible–infrared person re-identification

Recently, visible–infrared person re-identification (VI-ReID) has received considerable attention due to its practical importance. A number of methods extract multiple local features to enrich the diversity of feature representations. However, some local features often involve modality-relevant information, leading to deteriorated performance. Moreover, existing methods optimize the models by only considering the samples at each batch while ignoring the learned features at previous iterations. As a result, the features of the same person images drastically change at different training epochs, hindering the training stability. To alleviate the above issues, we propose a novel consistency-driven feature scoring and regularization network (CFSR-Net), which consists of a backbone network, a local feature learning block, a feature scoring block, and a global–local feature fusion block, for VI-ReID. On the one hand, we design a cross-modality consistency loss to highlight modality-irrelevant local features and suppress modality-relevant local features for each modality, facilitating the generation of a reliable compact local feature. On the other hand, we develop a feature consistency regularization strategy (including a momentum class contrastive loss and a momentum distillation loss) to impose consistency regularization on the learning of different levels of features by considering the learned features at historical epochs. This effectively enables smooth feature changes and thus improves the training stability. Extensive experiments on public VI-ReID datasets clearly show the effectiveness of our method against several state-of-the-art VI-ReID methods. Code will be released at https://github.com/cxtjl/CFSR-Net.

Contrastive Learning for Morphological Disambiguation Using Large Language Models in Low-Resource Settings

In this paper, a contrastive learning approach for morphological disambiguation (MD) using large language models (LLMs) is presented. A contrastive loss function is introduced for training the approach, which reduces the distance between the correct analysis and contextual embeddings while maintaining a margin between correct and incorrect embeddings. One of the aims of the paper is to analyze the effects of fine-tuning an LLM on MD in morphologically complex languages (MCLs) with special reference to low-resource languages such as Kazakh, as well as Turkish. Another goal of the paper is to consider various distance measures for this contrastive loss function, aiming to achieve better results when performing disambiguation by computing the distance between the context and the analysis embeddings. The existing approaches for morphological disambiguation, such as HMM-based and feature-engineering approaches, have limitations in modeling long-term dependencies and in the case of large, sparse tagsets. These challenges are mitigated in the proposed approach by leveraging LLMs, thus achieving better accuracy in handling the cases of ambiguity and OOV tokens without the need to rely on other features. Experiments were conducted on three datasets for two MCLs, Kazakh and Turkish—the former is a typical low-resource language. The results revealed that the proposed approach with contrastive loss improves MD performance when integrated with knowledge from large language models.

MVCLST: A spatial transcriptome data analysis pipeline for cell type classification based on multi-view comparative learning

Recent advancements in spatial transcriptomics sequencing technologies can not only provide gene expression within individual cells or cell clusters (spots) in a tissue but also pinpoint the exact location of this expression and generate detailed images of stained tissue sections, which offers invaluable insights into cell type identification and cell function exploration. However, effectively integratingthegene expression data, spatial location information, and tissue images from spatial transcriptomics data presents a significant challenge for computational methodsin cell classification. In this work, we propose MVCLST, a multi-view comparative learningmethod to analyze spatial transcriptomicsdata for accurate cell type classification. MVCLSTconstructs two views based on gene expression profiles, cell coordinates and image features. The multi-view method we proposed can significantly enhance the effectiveness of feature extraction while avoiding the impact of erroneous information in organizing image or gene expression data. The model employs four separate encoders to capture shared and unique features within each view. To ensure consistency and facilitate information exchange between the two views, MVCLST incorporates a contrastive learning loss function. The extracted shared and private features from both views are fused using corresponding decoders. Finally, the model utilizes the Leiden algorithm to clusterthe learned featuresfor cell type identification. Additionally, we establish a framework called MVCLST-CCFS for spatial transcriptomicsdata analysis based on MVCLST and consistent clustering. Our method achieves excellent results in clustering on human dorsolateral prefrontal cortex data and the mouse brain tissue data. Italso outperforms state-of-the-art techniques in the subsequent search for highly variable genes across cell types on the mouse olfactory bulbdata.

AFDFusion: An adaptive frequency decoupling fusion network for multi-modality image

The multi-modality image fusion goal is to create a single image that provides a comprehensive scene description and conforms to visual perception by integrating complementary information about the merits of the different modalities, e.g., salient intensities of infrared images and detail textures of visible images. Although some works explore decoupled representations of multi-modality images, they struggle with complex nonlinear relationships, fine modal decoupling, and noise handling. To cope with this issue, we propose an adaptive frequency decoupling module to perceive the associative invariant and inherent specific among cross-modality by dynamically adjusting the learnable low frequency weight of the kernel. Specifically, we utilize a contrastive learning loss for restricting the solution space of feature decoupling to learn representations of both the invariant and specific in the multi-modality images. The underlying idea is that: in decoupling, low frequency features, which are similar in the representation space, should be pulled closer to each other, signifying the associative invariant, while high frequencies are pushed farther away, also indicating the intrinsic specific. Additionally, a multi-stage training manner is introduced into our framework to achieve decoupling and fusion. Stage I, MixEncoder and MixDecoder with the same architecture but different parameters are trained to perform decoupling and reconstruction supervised by the contrastive self-supervised mechanism. Stage II, two feature fusion modules are added to integrate the invariant and specific features and output the fused image. Extensive experiments demonstrated the proposed method superiority over the state-of-the-art methods in both qualitative and quantitative evaluation on two multi-modal image fusion tasks.

Hierarchical Perception for Encrypted Traffic Classification via Class Incremental Learning

The rapid evolution of internet technology has resulted in an ongoing update of the types of encrypted network traffic. Therefore, efficient Encrypted Traffic Classification (ETC) is of significant importance for the security of user data and computer systems. Incremental Learning (IL) strategies for ETC methods allow them to evolve with the network environment, achieving remarkable results in real-world scenarios. However, existing IL frameworks for ETC tasks face issues of low computational efficiency and insufficient incremental capability, making it difficult to achieve satisfactory performance. In this work, we introduce an incremental ETC scheme, HCA-Net, which uses hierarchical perception to evolve with traffic flows. We design a feature-reweighted Depthwise separable convolution that ensures computational efficiency without compromising feature extraction capabilities. Additionally, our IL framework comprises a carefully constructed contrastive loss and a representative exemplar selection strategy, enabling the distillation of knowledge from learning old traffic categories to the parameters of learning new knowledge, mitigating the inevitable catastrophic forgetting problem in IL methods. Comprehensive experimental results on three public datasets show that our scheme outperforms the state-of-the-art methods, demonstrating exceptional performance in ETC tasks. By acquiring specific traffic samples at each training stage, our approach achieves incremental ETC, showcasing robust incremental capability and computational efficiency.

Heart disease detection using an acceleration-deceleration curve-based neural network with consumer-grade smartwatch data

Cardiovascular disease (CVD) is the most important cause of morbidity and mortality worldwide. Early detection, prevention or even prediction is of pivotal importance to reduce the burden of cardiovascular disease and its associated costs. Low cost, consumer-grade smartwatches have the potential to revolutionize cardiovascular medicine by enabling continuous monitoring of heart rate and activity. When combined with machine learning(ML), the resulting large amounts of time series data hold the potential of detection, or exclusion of CVD. However, analyzing such large datasets is challenging due to the sparse presence of informative segments. Efficient selection of these segments is essential for developing predictive models for clinical deployment. The objective of this paper was to investigate the potential of an acceleration-deceleration curvebased ML model as a novel clinical indicator for the detection of cardiovascular diseases. We used data from the ME-TIME study; 42 participants from which 21 have a cardiovascular disease and 21 are health controls. Data from each subject was normalized to decrease inter-subject variability. A neural network model aggregated predictions per week. We showed that per-subject normalization by the peak value of curves during inactivity, aggregation of model predictions over a week, and using a contrastive loss, resulted in a predictive model with 99 % ± 3 % specificity and 40 % ± 49 % sensitivity on the development set, and 100 % specificity with 67 % ± 47 % sensitivity on the test set. Acceleration-deceleration curves are effective patterns for ruling out the presence of cardiovascular disease, but caution must be taken to properly pre-process the curves and carefully choosing a model that reduces the variability in the extracted curves.

Classification of breast cancer histopathology images using a modified supervised contrastive learning method.

Deep neural networks have reached remarkable achievements in medical image processing tasks, specifically in classifying and detecting various diseases. However, when confronted with limited data, these networks face a critical vulnerability, often succumbing to overfitting by excessively memorizing the limited information available. This work addresses the challenge mentioned above by improving the supervised contrastive learning method leveraging both image-level labels and domain-specific augmentations to enhance model robustness. This approach integrates self-supervised pre-training with a two-stage supervised contrastive learning strategy. In the first stage, we employ a modified supervised contrastive loss that not only focuses on reducing false negatives but also introduces an elimination effect to address false positives. In the second stage, a relaxing mechanism is introduced that refines positive and negative pairs based on similarity, ensuring that only relevant image representations are aligned. We evaluate our method on the BreakHis dataset, which consists of breast cancer histopathology images, and demonstrate an increase in classification accuracy by 1.45% in the image level, compared to the state-of-the-art method. This improvement corresponds to 93.63% absolute accuracy, highlighting the effectiveness of our approach in leveraging properties of data to learn more appropriate representation space. The code implementation of this study is accessible on GitHub https://github.com/matinamehdizadeh/Breast-Cancer-Detection .

A MIL-based framework via contrastive instance learning and multimodal learning for long-term ECG classification

Recently, deep learning-based models are widely employed for electrocardiogram (ECG) classification. However, classifying long-term ECGs, which contain vast amounts of data, is challenging. Due to the limitation of memory with respect to the original data size, preprocessing techniques such as resizing or cropping are often applied, leading to information loss. Therefore, introducing multi-instance learning (MIL) to address long-term ECG classification problems is crucial. However, a major drawback of employing MIL is the destruction of sample integrity, which consequently hinders the interaction among instances. To tackle this challenge, we proposed a multimodal MIL neural network named CIMIL, which consists of three key components: an instance interactor, a feature fusion method based on attention mechanisms, and a multimodal contrastive instance loss. First, we designed an instance interactor to improve the interaction and keep continuity among instances. Second, we proposed a novel feature fusion method based on attention mechanisms to effectively aggregate multimodal instance features for final classification, which selects key instances within each class, not only enhances the performance of our model but also reduces the number of parameters. Third, a multimodal contrastive instance loss is proposed to enhance the model’s ability to distinguish positive and negative multimodal instances. Finally, we evaluated CIMIL on both intrapatient and interpatient patterns of two commonly used ECG datasets. The experimental results show that the proposed CIMIL outperforms existing state-of-the-art methods on long-term ECG tasks.

Multimodal Data Mining Based on Self Attention Feature Alignment

This study explores the application of multimodal data mining in text and image vector processing, aiming to improve the depth and breadth of data analysis by integrating information from different data types. We use the FairFace dataset combined with the CLIP model encoding layer to obtain text and image vectors, and use the K-Means clustering algorithm to achieve vector dimensionality reduction. Subsequently, we introduced the bipartite graph matching algorithm to achieve maximum matching between text vectors and image vectors, and calculated the contrastive learning loss and similarity loss. The entire process covers steps such as data preparation, feature extraction, vector dimensionality reduction, matching algorithms, and loss assessment, constructing a complete text and image matching task process. Our research contributions include using K-Means clustering algorithm to achieve vector dimensionality reduction, as well as introducing bipartite graph matching algorithm and calculating two types of losses in text and image vector matching, further improving matching quality.

Cryptocurrency Transaction Fraud Detection Based on Imbalanced Classification With Interpretable Analysis

This study introduces an interpretable imbalanced data classification method for detecting cryptocurrency transaction fraud. We address data imbalance using SMOTE oversampling and data augmentation through contrastive learning. Next, we introduce a Transformer-based deep learning model that learns sample relevance. The model undergoes pre-training with a contrastive loss and fine-tuning through Bayesian optimization to effectively extract high-dimensional, higher-order, and fraud-related features. We employ a SHAP-based interpreter along with attention scores to elucidate the role of various transaction features in fraud detection. Comparative results demonstrate the model's remarkable recall performance in identifying cryptocurrency transaction fraud. Furthermore, it achieves an excellent F1 value, striking a balance between accuracy and recall. This research not only enriches financial fraud detection but also enhances cryptocurrency transaction security, promotes market development, and contributes to economic stability and social security.