Cross-modal Learning Research Articles

DFCNet +: Cross-modal dynamic feature contrast net for continuous sign language recognition

In sign language communication, the combination of hand signs and facial expressions is used to convey messages in a fluid manner. Accurate interpretation relies heavily on understanding the context of these signs. Current methods, however, often focus on static images, missing the continuous flow and the story that unfolds through successive movements in sign language. To address this constraint, our research introduces the Dynamic Feature Contrast Net Plus (DFCNet+), a novel model that incorporates both dynamic feature extraction and cross-modal learning. The dynamic feature extraction module of DFCNet+ uses dynamic trajectory capture to monitor and record motion across frames and apply key features as an enhancement tool that highlights pixels that are critical for recognizing important sign language movements, allowing the model to follow the temporal variation of the signs. In the cross-modal learning module, we depart from the conventional approach of aligning video frames with textual descriptions. Instead, we adopt a gloss-level alignment, which provides a more detailed match between the visual signals and their corresponding text glosses, capturing the intricate relationship between what is seen and the associated text. The enhanced proficiency of DFCNet+ in discerning inter-frame details translates to heightened precision on benchmarks such as PHOENIX14, PHOENIX14-T and CSL-Daily. Such performance underscores its advantage in dynamic feature capture and inter-modal learning compared to conventional approaches to sign language interpretation. Our code is available at https://github.com/fyzjut/DFCNet_Plus.

CLIP-DFGS: A Hard Sample Mining Method for CLIP in Generalizable Person Re-Identification

Recent advancements in pre-trained vision-language models like CLIP have shown promise in person re-identification (ReID) applications. However, their performance in generalizable person re-identification tasks remains suboptimal. The large-scale and diverse image-text pairs used in CLIP's pre-training may lead to a lack or insufficiency of certain fine-grained features. In light of these challenges, we propose a hard sample mining method called DFGS (Depth-First Graph Sampler), based on depth-first search, designed to offer sufficiently challenging samples to enhance CLIP's ability to extract fine-grained features. DFGS can be applied to both the image encoder and the text encoder in CLIP. By leveraging the powerful cross-modal learning capabilities of CLIP, we aim to apply our DFGS method to extract challenging samples and form mini-batches with high discriminative difficulty, providing the image model with more efficient and challenging samples that are difficult to distinguish, thereby enhancing the model's ability to differentiate between individuals. Our results demonstrate significant improvements over other methods, confirming the effectiveness of DFGS in providing challenging samples that enhance CLIP's performance in generalizable person re-identification.

VITR: Augmenting Vision Transformers with Relation-Focused Learning for Cross-Modal Information Retrieval

The relations expressed in user queries are vital for cross-modal information retrieval. Relation-focused cross-modal retrieval aims to retrieve information that corresponds to these relations, enabling effective retrieval across different modalities. Pre-trained networks, such as Contrastive Language-Image Pre-training networks, have gained significant attention and acclaim for their exceptional performance in various cross-modal learning tasks. However, the Vision Transformer (ViT) used in these networks is limited in its ability to focus on image region relations. Specifically, ViT is trained to match images with relevant descriptions at the global level, without considering the alignment between image regions and descriptions. This article introduces VITR, a novel network that enhances ViT by extracting and reasoning about image region relations based on a local encoder. VITR is comprised of two key components. Firstly, it extends the capabilities of ViT-based cross-modal networks by enabling them to extract and reason with region relations present in images. Secondly, VITR incorporates a fusion module that combines the reasoned results with global knowledge to predict similarity scores between images and descriptions. The proposed VITR network was evaluated through experiments on the tasks of relation-focused cross-modal information retrieval. The results derived from the analysis of the Flickr30K, MS-COCO, RefCOCOg, and CLEVR datasets demonstrated that the proposed VITR network consistently outperforms state-of-the-art networks in image-to-text and text-to-image retrieval.

Dense 3D Point Cloud Environmental Mapping Using Millimeter-Wave Radar

To address the challenges of sparse point clouds in current MIMO millimeter-wave radar environmental mapping, this paper proposes a dense 3D millimeter-wave radar point cloud environmental mapping algorithm. In the preprocessing phase, a radar SLAM-based approach is introduced to construct local submaps, which replaces the direct use of radar point cloud frames. This not only reduces data dimensionality but also enables the proposed method to handle scenarios involving vehicle motion with varying speeds. Building on this, a 3D-RadarHR cross-modal learning network is proposed, which uses LiDAR as the target output to train the radar submaps, thereby generating a dense millimeter-wave radar point cloud map. Experimental results across multiple scenarios, including outdoor environments and underground tunnels, demonstrate that the proposed method can increase the point cloud density of millimeter-wave radar environmental maps by over 50 times, with a point cloud accuracy better than 0.1 m. Compared to existing algorithms, the proposed method achieves superior environmental map reconstruction performance while maintaining a real-time processing rate of 15 Hz.

Multimodal Social Media Fake News Detection Based on 1D-CCNet Attention Mechanism

Due to the explosive rise of multimodal content in online social communities, cross-modal learning is crucial for accurate fake news detection. However, current multimodal fake news detection techniques face challenges in extracting features from multiple modalities and fusing cross-modal information, failing to fully exploit the correlations and complementarities between different modalities. To address these issues, this paper proposes a fake news detection model based on a one-dimensional CCNet (1D-CCNet) attention mechanism, named BTCM. This method first utilizes BERT and BLIP-2 encoders to extract text and image features. Then, it employs the proposed 1D-CCNet attention mechanism module to process the input text and image sequences, enhancing the important aspects of the bimodal features. Meanwhile, this paper uses the pre-trained BLIP-2 model for object detection in images, generating image descriptions and augmenting text data to enhance the dataset. This operation aims to further strengthen the correlations between different modalities. Finally, this paper proposes a heterogeneous cross-feature fusion method (HCFFM) to integrate image and text features. Comparative experiments were conducted on three public datasets: Twitter, Weibo, and Gossipcop. The results show that the proposed model achieved excellent performance.

Exploring the synergy between textual identity and visual signals in human-object interaction

Human-Object Interaction (HOI) detection task aims to recognize and understand interactions between humans and objects depicted in images. Unlike instance recognition tasks, which focus on isolated objects, HOI detection requires considering various explanatory factors, such as instance identity, spatial relationships, and scene context. However, previous HOI detection methods have primarily relied on local visual cues, often overlooking the vital role of instance identity and thus limiting the performance of models. In this paper, we introduce textual features to expand the definition of HOI representations, incorporating instance identity into the HOI reasoning process. Drawing inspiration from the human activity perception process, we explore the synergy between textual identity and visual signals to leverage various explanatory factors more effectively and enhance HOI detection performance. Specifically, our method extracts HOI explanatory factors using both modal representations. Visual features capture interactive cues, while textual features explicitly denote instance identities within human-object pairs, delineating relevant interaction categories. Additionally, we utilize Contrastive Language-Image Pre-training (CLIP) to enhance the semantic alignment between visual and textual features and design a cross-modal learning module for integrating HOI multimodal information. Extensive experiments on several benchmarks demonstrate that our proposed framework surpasses most existing methods, achieving outstanding performance with a mean average precision (mAP) of 33.95 on the HICO-DET dataset and 63.2 mAP on the V-COCO dataset.

A domain adaptation framework for cross-modality SAR 3D reconstruction point clouds segmentation utilizing LiDAR data

Synthetic aperture radar (SAR) 3D point clouds reconstruction can eliminate the problems of layover in 2D SAR image projections, the recognition of the reconstructed point cloud can significantly enhance target identification and information extraction. Current SAR 3D point cloud segmentation methods, based on traditional machine learning clustering approaches, suffer from poor automation and accuracy. However, the absence of publicly labeled SAR datasets impedes the progress of deep learning-based SAR 3D point cloud segmentation methods. To tackle the aforementioned challenges, we introduce, for the first time, an alternative training approach for SAR 3D point cloud segmentation using LiDAR annotated data, which offers a more abundant sample pool, thus alleviating the lack of training sets. Nevertheless, a segmentation model trained on LiDAR point clouds exhibits a significant decline in performance when directly applied to SAR 3D reconstructed point clouds due to the cross-domain discrepancy. This research presents a pioneering domain adaptation 3D semantic segmentation framework to implement cross-modal learning for SAR point clouds. In our scheme, a simple yet effective technique is developed to achieve segmentation of SAR double-bounce scattering regions called SARDBS-Mix, which employs a mixing strategy, to capture the distinctive reflection characteristics of SAR data. Furthermore, we implement approaches including center alignment and normalization, local augmentation, and weighted cross entropy to mitigate LiDAR and SAR domain gap and class imbalances. The experimental results validate the feasibility and effectiveness of the proposed method for SAR 3D reconstructed point cloud segmentation.

Enhancing Vehicle Tracking and Recognition Across Multiple Cameras with Multimodal Contrastive Domain Sharing GAN and Topological Embeddings

Using Multimodal Contrastive Domain Sharing Generative Adversarial Networks (GAN) and topological embeddings, this study shows a new way to improve car tracking and classification across multiple camera feeds. Different camera angles and lighting conditions can make it hard for current car tracking systems to work correctly. This study tries to solve these problems. Common Objects in Context (COCO) and ImageNet are two datasets that are used in this method for training. Multimodal Contrastive Domain Sharing GAN is used for detection and tracking. It makes cross-modal learning easier by letting you see things from different camera angles. This framework lets the model learn shared representations, which makes it better at recognizing vehicles in a wider range of visual domains. The Topological Information Embedded Convolutional Neural Network (TIE-CNN) is used to re-identify the car after it has been found and tracked. This network embeds the paths of vehicles into a continuous latent space, keeping the important spatial connections needed for accurate tracking. Real-world multi-camera datasets used for experimental confirmation show that tracking accuracy and recognition performance are much better than with standard methods. The suggested framework works great in tough situations like blocked views and sudden changes in lighting, showing that it are reliable in complicated surveillance settings. This study adds to the progress in multi-camera car tracking and identification by combining geometric data analysis with deep learning methods. This method uses Multimodal Contrastive Domain Sharing GAN and topological embeddings to improve the timing and spatial coherence of tracking results. It also sets the stage for future improvements in monitoring and self-driving systems.

Cross-modal learning representation using new margin combination for speech recognition task

Cross-modal retrieval aims to elucidate the fusion of information, mimic human learning, and advance the field. The main challenge in cross-modal matching is to build a shared subspace reflecting semantic proximity. Previous works fail to capture asymmetric relevance by adopting symmetric similarity computations. To overcome these shortcomings, an efficient approach called quaternion representation learning (QRL) is introduced for better cross-modal matching. Thus, a better representation of the shared semantics is offered by virtue of its richer representation capacity of the quaternionic space and its strong expressive power. Transfer learning is a crucial aspect in this context. By leveraging pre-trained models, the knowledge gained from one task or domain can be effectively transferred to another, allowing for improved performance and generalization. In this study, transfer learning is employed to enhance the cross-modal retrieval system. Specifically, a pre-trained ResNet-512 model is utilized in conjunction with the proposed total margin (TM) loss function, which combines the QRL approach with the novel adaptive mean margin (AMM) methodology. The TM loss function, coupled with the pre-trained ResNet-512 model, is evaluated on the Audio-Visual Arabic Speech Database (AVAS) and the Arabic Visual Speech Database (AVSD), along with other audio-visual datasets. Experimental results demonstrate the effectiveness of the TM loss function in consistently improving performance on both databases. The recall scores (R@k) and mean average precision (mAP) values achieved on the AVAS Database are as follows: R@1: 42.1±0.7, R@2: 70.2±0.1, R@5: 78.5±1.0, and mAP: 53.0±1.1. Similarly, on the AVSD Database, the results are R@1: 41.7±0.3, R@2: 69.2±1.1, R@5: 78.0±0.3, and mAP: 52.7±0.5. By incorporating transfer learning and the TM loss function into the cross-modal retrieval framework, this study demonstrates the potential for improving clustering efficiency and enhancing visual and speech understanding. The combination of pre-trained models and the TM loss function offers a promising avenue for advancing crossmodal matching techniques and achieving state-of-the-art performance.

Cross-modal learning for optical flow estimation with events

Benefiting from the low latency and high dynamic range, event cameras have recently been adopted for Optical Flow (OF) prediction under harsh environments with high-speed motion or extreme lighting conditions. However, the emitted events only respond to brightness changes, mostly at edges with high-intensity contrast, leading to the ill-posedness of OF prediction in non-edge or low-contrast regions. To address this problem, we propose a network to learn OF by exploring the complements between the frame and the event domains. In particular, we first extract cross-modal features from frames and events via a fusion-based encoder and then apply a self-attention module of cross-dimension interaction to suppress disturbances in noisy events. Furthermore, a long-term memory recurrent module is built for local and global temporal information combination and cross-modal feature enhancement. Finally, a CNN decoder is designed to upscale cross-modal features and provide pyramidal results for multi-scale loss computation. Extensive experiments demonstrate that the proposed method can achieve leading performance compared with the state-of-the-art OF models.

Cross-modal learning using privileged information for long-tailed image classification

The prevalence of long-tailed distributions in real-world data often results in classification models favoring the dominant classes, neglecting the less frequent ones. Current approaches address the issues in long-tailed image classification by rebalancing data, optimizing weights, and augmenting information. However, these methods often struggle to balance the performance between dominant and minority classes because of inadequate representation learning of the latter. To address these problems, we introduce descriptional words into images as cross-modal privileged information and propose a cross-modal enhanced method for long-tailed image classification, referred to as CMLTNet. CMLTNet improves the learning of intraclass similarity of tail-class representations by cross-modal alignment and captures the difference between the head and tail classes in semantic space by cross-modal inference. After fusing the above information, CMLTNet achieved an overall performance that was better than those of benchmark long-tailed and cross-modal learning methods on the long-tailed cross-modal datasets, NUS-WIDE and VireoFood-172. The effectiveness of the proposed modules was further studied through ablation experiments. In a case study of feature distribution, the proposed model was better in learning representations of tail classes, and in the experiments on model attention, CMLTNet has the potential to help learn some rare concepts in the tail class through mapping to the semantic space.

AutoDLAR: A Semi-supervised Cross-modal Contact-free Human Activity Recognition System

WiFi-based human activity recognition (HAR) plays an essential role in various applications such as security surveillance, health monitoring, and smart home. Existing HAR methods, though yielding promising performance in indoor scenarios, highly depend on a massive labeled dataset for training which is extremely difficult to acquire in practical applications. In this paper, we present an automatic data labeling and HAR system, termed AutoDLAR. Taking a semi-supervised cross-modal learning framework with a hybrid loss function as the core, AutoDLAR transfers rich visual information to automatically label WiFi signals for WiFi-based HAR. Specifically, we devise a lightweight and multi-view WiFi sensing model with a parallel feature embedding method to accurately identify activities and accelerate recognition speed. Then, we exploit the video data to fine-tune a well-established visual HAR model, generating effective pseudo-labels for guiding the WiFi model’s training. We also build a synchronized Video-WiFi dataset with seven types of human activities under different scenarios to enable training and validating the semi-supervised HAR system. Extensive experiments on our collected activity dataset and the emotion recognition benchmark demonstrate that AutoDLAR attains an average accuracy of over 95.89% without manual labeling and only spends the inference time of 3.35 ms, outperforming the state-of-the-art (SOTA) methods.

A knowledge distillation based cross-modal learning framework for the lithium-ion battery state of health estimation

The accurate prediction of a lithium-ion battery’s State of Health is of critical importance for efficient battery health management. Existing data-driven estimation methodologies grapple with issues such as high model complexity and a dearth of guidance from prior knowledge, which impose constraints on their efficacy. This work introduces a novel cross-modal distillation network for battery State of Health estimation, structured around a TransformerEncoder as the teacher network and a Convolutional Neural Network as the student network. Initially, the teacher model is pre-trained offline using State of Health degradation data to learn the degradation patterns. The directly measurable feature data (such as voltage, temperature, and current) is subsequently fed into the student network for online training and computation of a hard loss. the student network’s output is then directed into the pre-trained the teacher network to compute a soft loss, thereby offering prior knowledge of degradation laws and steering the optimization process of the student network. Rigorous experiments are conducted utilizing various datasets, with the outcomes validating the superior estimation accuracy and degradation rule adherence of the model. Notably, among five different models, this model demonstrates the best performance on almost all datasets, achieving an RMSE of 0.0097 and an MAE of 0.0065 on Cell1 of the Oxford dataset. Moreover, the model also demonstrates robust performance across different usage scenarios, inclusive of multi-battery estimation. Furthermore, this paper also introduces a fine tuning method for State of Health predictions only using the first half of the data. Comparative analysis with other models underscores the competitiveness of the proposed model, showcasing its potential for broader application.

Enhancing Visual Grounding in Vision-Language Pre-Training With Position-Guided Text Prompts.

Vision-Language Pre-Training (VLP) has demonstrated remarkable potential in aligning image and text pairs, paving the way for a wide range of cross-modal learning tasks. Nevertheless, we have observed that VLP models often fall short in terms of visual grounding and localization capabilities, which are crucial for many downstream tasks, such as visual reasoning. In response, we introduce a novel Position-guided Text Prompt (PTP) paradigm to bolster the visual grounding abilities of cross-modal models trained with VLP. In the VLP phase, PTP divides an image into N x N blocks and employs a widely-used object detector to identify objects within each block. PTP then reframes the visual grounding task as a fill-in-the-blank problem, encouraging the model to predict objects in given blocks or regress the blocks of a given object, exemplified by filling "[P]" or "[O]" in a PTP sentence such as "The block [P] has a [O]." This strategy enhances the visual grounding capabilities of VLP models, enabling them to better tackle various downstream tasks. Additionally, we integrate the seconda-order relationships between objects to further enhance the visual grounding capabilities of our proposed PTP paradigm. Incorporating PTP into several state-of-the-art VLP frameworks leads to consistently significant improvements across representative cross-modal learning model architectures and multiple benchmarks, such as zero-shot Flickr30 k Retrieval (+5.6 in average recall@1) for ViLT baseline, and COCO Captioning (+5.5 in CIDEr) for the state-of-the-art BLIP baseline. Furthermore, PTP attains comparable results with object-detector-based methods and a faster inference speed, as it discards its object detector during inference, unlike other approaches.

Multimodal contrastive learning for brain–machine fusion: From brain-in-the-loop modeling to brain-out-of-the-loop application

Harnessing the complementarity between the brain and artificial neural networks has shown great promise for the development of novel brain–machine fusion systems that may rival the robustness and flexibility of the human visual system. Nonetheless, due to the requirement of human involvement, brain–machine fusion models are challenging to apply in a brain-in-the-loop manner to handle task demands that involve long duration, high intensity and complex operating environments. In this paper, we propose a brain–machine fusion approach to achieve the brain-in-the-loop modeling and brain-out-of-the-loop application. The similarities and differences between the image representations of brain responses and image features computed by deep convolutional neural network (DCNN) are firstly analyzed using a rhesus monkey dataset, and the results lay the foundation for the feasibility of brain–machine fusion. A brain–machine fusion model is then developed and a multimodal supervised contrastive learning method is proposed to jointly learn the image representations for brain responses and DCNNs. The fusion model can be applied in a brain-out-of-the-loop manner, effectively addressing the challenges encountered by human-involved approaches. Extensive experiments on a self-built human vehicle detection dataset demonstrate the effectiveness of the proposed method in improving the generalization ability of the downstream image classifiers, both in cross-modal learning or multimodal fusion settings.

Restricted Boltzmann Machines Implemented by Spin-Orbit Torque Magnetic Tunnel Junctions.

Artificial intelligence has surged forward with the advent of generative models, which rely heavily on stochastic computing architectures enhanced by true random number generators with adjustable sampling probabilities. In this study, we develop spin-orbit torque magnetic tunnel junctions (SOT-MTJs), investigating their sigmoid-style switching probability as a function of the driving voltage. This feature proves to be ideally suited for stochastic computing algorithms such as the restricted Boltzmann machines (RBM) prevalent in pretraining processes. We exploit SOT-MTJs as both stochastic samplers and network nodes for RBMs, enabling the implementation of RBM-based neural networks to achieve recognition tasks for both handwritten and spoken digits. Moreover, we further harness the weights derived from the preceding image and speech training processes to facilitate cross-modal learning from speech to image generation. Our results clearly demonstrate that these SOT-MTJs are promising candidates for the development of hardware accelerators tailored for Boltzmann neural networks and other stochastic computing architectures.

Experience transforms crossmodal object representations in the anterior temporal lobes.

Combining information from multiple senses is essential to object recognition, core to the ability to learn concepts, make new inferences, and generalize across distinct entities. Yet how the mind combines sensory input into coherent crossmodal representations - the crossmodal binding problem - remains poorly understood. Here, we applied multi-echo fMRI across a 4-day paradigm, in which participants learned three-dimensional crossmodal representations created from well-characterized unimodal visual shape and sound features. Our novel paradigm decoupled the learned crossmodal object representations from their baseline unimodal shapes and sounds, thus allowing us to track the emergence of crossmodal object representations as they were learned by healthy adults. Critically, we found that two anterior temporal lobe structures - temporal pole and perirhinal cortex - differentiated learned from non-learned crossmodal objects, even when controlling for the unimodal features that composed those objects. These results provide evidence for integrated crossmodal object representations in the anterior temporal lobes that were different from the representations for the unimodal features. Furthermore, we found that perirhinal cortex representations were by default biased toward visual shape, but this initial visual bias was attenuated by crossmodal learning. Thus, crossmodal learning transformed perirhinal representations such that they were no longer predominantly grounded in the visual modality, which may be a mechanism by which object concepts gain their abstraction.

Audiovisual Moments in Time: A large-scale annotated dataset of audiovisual actions.

We present Audiovisual Moments in Time (AVMIT), a large-scale dataset of audiovisual action events. In an extensive annotation task 11 participants labelled a subset of 3-second audiovisual videos from the Moments in Time dataset (MIT). For each trial, participants assessed whether the labelled audiovisual action event was present and whether it was the most prominent feature of the video. The dataset includes the annotation of 57,177 audiovisual videos, each independently evaluated by 3 of 11 trained participants. From this initial collection, we created a curated test set of 16 distinct action classes, with 60 videos each (960 videos). We also offer 2 sets of pre-computed audiovisual feature embeddings, using VGGish/YamNet for audio data and VGG16/EfficientNetB0 for visual data, thereby lowering the barrier to entry for audiovisual DNN research. We explored the advantages of AVMIT annotations and feature embeddings to improve performance on audiovisual event recognition. A series of 6 Recurrent Neural Networks (RNNs) were trained on either AVMIT-filtered audiovisual events or modality-agnostic events from MIT, and then tested on our audiovisual test set. In all RNNs, top 1 accuracy was increased by 2.71-5.94% by training exclusively on audiovisual events, even outweighing a three-fold increase in training data. Additionally, we introduce the Supervised Audiovisual Correspondence (SAVC) task whereby a classifier must discern whether audio and visual streams correspond to the same action label. We trained 6 RNNs on the SAVC task, with or without AVMIT-filtering, to explore whether AVMIT is helpful for cross-modal learning. In all RNNs, accuracy improved by 2.09-19.16% with AVMIT-filtered data. We anticipate that the newly annotated AVMIT dataset will serve as a valuable resource for research and comparative experiments involving computational models and human participants, specifically when addressing research questions where audiovisual correspondence is of critical importance.

Cross-modal learning with multi-modal model for video action recognition based on adaptive weight training

The canonical video action recognition methods usually label categories with numbers or one-hot vectors and train neural networks to classify a fixed set of predefined categories, thereby constraining their ability to recognise complex actions and transferable ability to unseen concepts. In contrast, cross-modal learning can improve the performance of individual modalities. Based on the facts that a better action recogniser can be built by reading the statements used to describe actions, we exploited the recent multimodal foundation model CLIP for action recognition. In this study, an effective Vision-Language action recognition adaptation was implemented based on few-shot examples spanning different modalities. We added semantic information to action categories by treating textual and visual label as training examples for action classifier construction rather than simply labelling them with numbers. Due to the different importance of words in text and video frames, simply averaging all sequential features may result in ignoring keywords or key video frames. To capture sequential and hierarchical representation, a weighted token-wise interaction mechanism was employed to exploit the pair-wise correlations adaptively. Extensive experiments with public datasets show that cross-modal action recognition learning helps for downstream action images classification, in other words, the proposed method can train better action classifiers by reading the sentences describing action itself. The method proposed in this study not only reaches good generalisation and zero-shot/few-shot transfer ability on Out of Distribution (OOD) test sets, but also performs lower computational complexity due to the lightweight interaction mechanism with 84.15% Top-1 accuracy on the Kinetics-400.

SoftCLIP: Softer Cross-Modal Alignment Makes CLIP Stronger

During the preceding biennium, vision-language pre-training has achieved noteworthy success on several downstream tasks. Nevertheless, acquiring high-quality image-text pairs, where the pairs are entirely exclusive of each other, remains a challenging task, and noise exists in the commonly used datasets. To address this issue, we propose SoftCLIP, a novel approach that relaxes the strict one-to-one constraint and achieves a soft cross-modal alignment by introducing a softened target, which is generated from the fine-grained intra-modal self-similarity. The intra-modal guidance is indicative to enable two pairs have some local similarities and model many-to-many relationships between the two modalities. Besides, since the positive still dominates in the softened target distribution, we disentangle the negatives in the distribution to further boost the relation alignment with the negatives in the cross-modal learning. Extensive experiments demonstrate the effectiveness of SoftCLIP. In particular, on ImageNet zero-shot classification task, using CC3M/CC12M as pre-training dataset, SoftCLIP brings a top-1 accuracy improvement of 6.8%/7.2% over the CLIP baseline.