Downstream Tasks Research Articles

Augmentation-aware self-supervised learning with conditioned projector

Self-supervised learning (SSL) is a powerful technique for learning from unlabeled data. By learning to remain invariant to applied data augmentations, methods such as SimCLR and MoCo can reach quality on par with supervised approaches. However, this invariance may be detrimental for solving downstream tasks that depend on traits affected by augmentations used during pretraining, such as color. In this paper, we propose to foster sensitivity to such characteristics in the representation space by modifying the projector network, a common component of self-supervised architectures. Specifically, we supplement the projector with information about augmentations applied to images. For the projector to take advantage of this auxiliary conditioning when solving the SSL task, the feature extractor learns to preserve the augmentation information in its representations. Our approach, coined Conditional Augmentation-aware Self-supervised Learning (CASSLE), is directly applicable to typical joint-embedding SSL methods regardless of their objective functions. Moreover, it does not require major changes in the network architecture or prior knowledge of downstream tasks. In addition to an analysis of sensitivity towards different data augmentations, we conduct a series of experiments, which show that CASSLE improves over various SSL methods, reaching state-of-the-art performance in multiple downstream tasks.11A short version of this paper appeared at the NeurIPS 2023 Workshop: Self-Supervised Learning - Theory and Practice (https://sslneurips23.github.io).

Robust Model Watermarking for Image Processing Networks via Structure Consistency.

The intellectual property of deep networks can be easily "stolen" by surrogate model attack. There has been significant progress in protecting the model IP in classification tasks. However, little attention has been devoted to the protection of image processing models. By utilizing consistent invisible spatial watermarks, the work (Zhang et al. 2020) first considered model watermarking for deep image processing networks and demonstrated its efficacy in many downstream tasks. Its success depends on the hypothesis that if a consistent watermark exists in all prediction outputs, that watermark will be learned into the attacker's surrogate model. However, when the attacker uses common data augmentation attacks (e.g., rotate, crop, and resize) during surrogate model training, it will fail because the underlying watermark consistency is destroyed. To mitigate this issue, we propose a new watermarking methodology, "structure consistency", based on which a new deep structure-aligned model watermarking algorithm is designed. Specifically, the embedded watermarks are designed to be aligned with physically consistent image structures, such as edges or semantic regions. Experiments demonstrate that our method is more robust than the baseline in resisting data augmentation attacks. Besides that, we test the generalization ability and robustness of our method to a broader range of adaptive attacks.

GMNI: Achieve good data augmentation in unsupervised graph contrastive learning

Graph contrastive learning (GCL) shows excellent potential in unsupervised graph representation learning. Data augmentation (DA), responsible for generating diverse views, plays a vital role in GCL, and its optimal choice heavily depends on the downstream task. However, it is impossible to measure task-relevant information under an unsupervised setting. Therefore, many GCL methods risk insufficient information by failing to preserve essential information necessary for the downstream task or risk encoding redundant information. In this paper, we propose a novel method called Minimal Noteworthy Information for unsupervised Graph contrastive learning (GMNI), featuring automated DA. It achieves good DA by balancing missing and excessive information, approximating the optimal views in contrastive learning. We employ an adversarial training strategy to generate views that share minimal noteworthy information (MNI), reducing nuisance information by minimization optimization and ensuring sufficient information by emphasizing noteworthy information. Besides, we introduce randomness based on MNI to augmentation, thereby enhancing view diversity and stabilizing the model against perturbations. Extensive experiments on unsupervised and semi-supervised learning over 14 datasets demonstrate the superiority of GMNI over GCL methods with automated and manual DA. GMNI achieves up to a 1.64% improvement over the state-of-the-art in unsupervised node classification, up to a 1.97% improvement in unsupervised graph classification, and up to a 3.57% improvement in semi-supervised graph classification.

Multi-level discriminator based contrastive learning for multiplex networks

Graph embedding is a technique for obtaining low-dimensional representations of nodes across diverse networks, which may then be used for various downstream tasks and applications. When it applies to heterogeneous networks, it is hard to handle heterogeneous networks because they usually contain different types of nodes and edges with more semantic and structural information. Recently, contrastive learning has developed as the preferred strategy for dealing with unsupervised heterogeneous graph embedding to reduce the cost of human label annotation. However, most multi-view contrastive learning approaches calculate the model’s loss only based on the mutual dependence between the node representation and graph representation. These approaches ignore that both node attributes and node clustering contain discriminative content. To solve this issue, we propose a model called Multi-Level Discriminator-based Contrastive Learning for Multiplex Networks (MLDCL). This model adopts a multi-level multi-discriminator-based approach that can simultaneously learn the global-level structural information, node-level attribute information, and local-level clustering information. Moreover, an augmentation strategy in the contrast learning process from the spectral domain is proposed to improve the representation and discriminative ability of MLDCL. Numerous tests with node clustering and classification tasks on widely used datasets demonstrate the efficacy of the proposed approach.

ClarityDiffuseNet: Enhancing fundus image quality under black shadows with diffusion model-based research

Deep learning models have achieved commendable success in the analysis of tasks related to fundus images. However, the performance of many models is affected by the quality of fundus images. A common quality issue observed in fundus images is the presence of severe black shadow artefact, primarily caused by opacities in the refractive media or due to insufficient or uneven illumination. Such low-quality images can compromise the model training and result in the models learning incorrect feature representations. The removal of black shadows can be regarded as a preprocessing problem in the enhancement of degraded images. Solutions typically involve either increasing the overall brightness of the image or restoring the dark shadowy areas specifically. Prior work on increasing image brightness has often utilized generative adversarial networks (GANs), while restoration has been approached with autoencoders and variational autoencoders (VAEs). However, approaches that focus on brightening often fall short in properly addressing local degradations, and restoration techniques can lead to loss of details or over-smoothing in the shadow areas.In this study, we introduce a method named ClarityDiffuseNet, a model for restoring low-quality fundus images based on diffusion generative models targeting severe black shadows. Our method restores areas with black shadows from regions of high quality, enhancing the image to be richer in detail and visually closer to artefact-free images. Compared to models based on GANs and inpainting methods, our approach demonstrates superior performance on four benchmark public datasets, with Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity index (SSIM) indices significantly surpassing the state-of-the-art models by 7% and 9%, respectively. Our method demonstrates notable improvements in downstream tasks, such as disease diagnosis—evidenced by 9% increase in the area under the curve (AUC) metric when tested on low-quality datasets—in vessel segmentation, which saw about 6% improvement in the Dice coefficient under similar conditions. These outcomes underscore the substantial promise of diffusion generative models within the realm of fundus image restoration, highlighting their effectiveness in enhancing image quality for further analysis.

Lifelong Generative Adversarial Autoencoder.

Lifelong learning describes an ability that enables humans to continually acquire and learn new information without forgetting. This capability, common to humans and animals, has lately been identified as an essential function for an artificial intelligence system aiming to learn continuously from a stream of data during a certain period of time. However, modern neural networks suffer from degenerated performance when learning multiple domains sequentially and fail to recognize past learned tasks after being retrained. This corresponds to catastrophic forgetting and is ultimately induced by replacing the parameters associated with previously learned tasks with new values. One approach in lifelong learning is the generative replay mechanism (GRM) that trains a powerful generator as the generative replay network, implemented by a variational autoencoder (VAE) or a generative adversarial network (GAN). In this article, we study the forgetting behavior of GRM-based learning systems by developing a new theoretical framework in which the forgetting process is expressed as an increase in the model's risk during the training. Although many recent attempts have provided high-quality generative replay samples by using GANs, they are limited to mainly downstream tasks due to the lack of inference. Inspired by the theoretical analysis while aiming to address the drawbacks of existing approaches, we propose the lifelong generative adversarial autoencoder (LGAA). LGAA consists of a generative replay network and three inference models, each addressing the inference of a different type of latent variable. The experimental results show that LGAA learns novel visual concepts without forgetting and can be applied to a wide range of downstream tasks.

A Task-Guided, Implicitly-Searched and Meta-Initialized Deep Model for Image Fusion.

Image fusion plays a key role in a variety of multi-sensor-based vision systems, especially for enhancing visual quality and/or extracting aggregated features for perception. However, most existing methods just consider image fusion as an individual task, thus ignoring its underlying relationship with these downstream vision problems. Furthermore, designing proper fusion architectures often requires huge engineering labor. It also lacks mechanisms to improve the flexibility and generalization ability of current fusion approaches. To mitigate these issues, we establish a Task-guided, Implicit-searched and Meta-initialized (TIM) deep model to address the image fusion problem in a challenging real-world scenario. Specifically, we first propose a constrained strategy to incorporate information from downstream tasks to guide the unsupervised learning process of image fusion. Within this framework, we then design an implicit search scheme to automatically discover compact architectures for our fusion model with high efficiency. In addition, a pretext meta initialization technique is introduced to leverage divergence fusion data to support fast adaptation for different kinds of image fusion tasks. Qualitative and quantitative experimental results on different categories of image fusion problems and related downstream tasks (e.g., visual enhancement and semantic understanding) substantiate the flexibility and effectiveness of our TIM.

Visualizing and Understanding Patch Interactions in Vision Transformer.

Vision transformer (ViT) has become a leading tool in various computer vision tasks, owing to its unique self-attention mechanism that learns visual representations explicitly through cross-patch information interactions. Despite having good success, the literature seldom explores the explainability of ViT, and there is no clear picture of how the attention mechanism with respect to the correlation across comprehensive patches will impact the performance and what is the further potential. In this work, we propose a novel explainable visualization approach to analyze and interpret the crucial attention interactions among patches for ViT. Specifically, we first introduce a quantification indicator to measure the impact of patch interaction and verify such quantification on attention window design and indiscriminative patches removal. Then, we exploit the effective responsive field of each patch in ViT and devise a window-free transformer (WinfT) architecture accordingly. Extensive experiments on ImageNet demonstrate that the exquisitely designed quantitative method is shown able to facilitate ViT model learning, leading the top-1 accuracy by 4.28% at most. More remarkably, the results on downstream fine-grained recognition tasks further validate the generalization of our proposal.

An Evidential Multi-Target Domain Adaptation Method Based on Weighted Fusion for Cross-Domain Pattern Classification.

For cross-domain pattern classification, the supervised information (i.e., labeled patterns) in the source domain is often employed to help classify the unlabeled target domain patterns. In practice, multiple target domains are usually available. The unlabeled patterns (in different target domains) which have high-confidence predictions, can also provide some pseudo-supervised information for the downstream classification task. The performance in each target domain would be further improved if the pseudo-supervised information in different target domains can be effectively used. To this end, we propose an evidential multi-target domain adaptation (EMDA) method to take full advantage of the useful information in the single-source and multiple target domains. In EMDA, we first align distributions of the source and target domains by reducing maximum mean discrepancy (MMD) and covariance difference across domains. After that, we use the classifier learned by the labeled source domain data to classify query patterns in the target domains. The query patterns with high-confidence predictions are then selected to train a new classifier for yielding an extra piece of soft classification results of query patterns. The two pieces of soft classification results are then combined by evidence theory. In practice, their reliabilities/weights are usually diverse, and an equal treatment of them often yields the unreliable combination result. Thus, we propose to use the distribution discrepancy across domains to estimate their weighting factors, and discount them before fusing. The evidential combination of the two pieces of discounted soft classification results is employed to make the final class decision. The effectiveness of EMDA was verified by comparing with many advanced domain adaptation methods on several cross-domain pattern classification benchmark datasets.

Building, benchmarking, and exploring perturbative maps of transcriptional and morphological data.

The continued scaling of genetic perturbation technologies combined with high-dimensional assays such as cellular microscopy and RNA-sequencing has enabled genome-scale reverse-genetics experiments that go beyond single-endpoint measurements of growth or lethality. Datasets emerging from these experiments can be combined to construct perturbative "maps of biology", in which readouts from various manipulations (e.g., CRISPR-Cas9 knockout, CRISPRi knockdown, compound treatment) are placed in unified, relatable embedding spaces allowing for the generation of genome-scale sets of pairwise comparisons. These maps of biology capture known biological relationships and uncover new associations which can be used for downstream discovery tasks. Construction of these maps involves many technical choices in both experimental and computational protocols, motivating the design of benchmark procedures to evaluate map quality in a systematic, unbiased manner. Here, we (1) establish a standardized terminology for the steps involved in perturbative map building, (2) introduce key classes of benchmarks to assess the quality of such maps, (3) construct 18 maps from four genome-scale datasets employing different cell types, perturbation technologies, and data readout modalities, (4) generate benchmark metrics for the constructed maps and investigate the reasons for performance variations, and (5) demonstrate utility of these maps to discover new biology by suggesting roles for two largely uncharacterized genes.

Label-guided contrastive learning with weighted pseudo-labeling: A novel mechanical fault diagnosis method with insufficient annotated data

Exploring fault diagnosis methods for mechanical equipment with weak dependency on annotated data is essential for industrial production. Contrastive learning (CL), capable of learning representations without labeling information, has achieved satisfactory performance in mechanical fault diagnosis. However, current CL-based approaches mainly encounter two limitations. First, the pre-training stage uses either unannotated or annotated samples exclusively while the fine-tuning stage solely relies on annotated ones, leading to inefficient sample utilization. Second, the representation learned by contrastive loss alone in the pretext task is sub-optimal for downstream diagnostic tasks. To address these issues, this paper proposed a novel diagnostic framework based on label-guided contrastive learning (LgCL) and weighted pseudo-labeling (WPL) strategy to improve fault diagnosis accuracy. In the pre-training stage, the proposed LgCL integrates two types of contrastive loss together with classification loss, enabling the encoder to learn discriminative representations that directly benefit the downstream diagnostic task. The devised hybrid fine-tuning strategy allows both labeled and unlabeled data to participate in fine-tuning via pseudo-labeling, enhancing model generalization. The pertinently designed WPL strategy mitigates the defect of noisy pseudo labels. Comparison and ablation experiments on two public datasets and one self-designed dataset validate the superiority of the proposed method for fault diagnosis with limited annotated data, with diagnostic accuracies improved by 25.30%, 5.47% and 10.02% over supervised, semi-supervised and contrastive learning methods, respectively.

Evaluation of a Task-Specific Self-Supervised Learning Framework in Digital Pathology Relative to Transfer Learning Approaches and Existing Foundation Models

An integral stage in typical digital pathology workflows involves deriving specific features from tiles extracted from a tessellated whole-slide image. Notably, various computer vision neural network architectures, particularly the ImageNet pretrained, have been extensively used in this domain. This study critically analyzes multiple strategies for encoding tiles to understand the extent of transfer learning and identify the most effective approach. The study categorizes neural network performance into 3 weight initialization methods: random, ImageNet-based, and self-supervised learning. Additionally, we propose a framework based on task-specific self-supervised learning, which introduces a shallow feature extraction method, employing a spatial-channel attention block to glean distinctive features optimized for histopathology intricacies. Across 2 different downstream classification tasks (patch classification and weakly supervised whole-slide image classification) with diverse classification data sets, including colorectal cancer histology, Patch Camelyon, prostate cancer detection, The Cancer Genome Atlas, and CIFAR-10, our task-specific self-supervised encoding approach consistently outperforms other convolutional neural network–based encoders. The better performances highlight the potential of task-specific attention-based self-supervised training in tailoring feature extraction for histopathology, indicating a shift from using pretrained models originating outside the histopathology domain. Our study supports the idea that task-specific self-supervised learning allows domain-specific feature extraction, encouraging a more focused analysis.

Learning to generate text with auxiliary tasks

Text generation is a vital natural language processing task that generates a sequence given an input text. Most recent text generation models have mainly utilized the power of pre-trained language models (PLMs) and achieved promising results. However, while working on this problem, we observed that important indicators like salient words and sentences from downstream generation tasks can still be useful and beneficial for enhancing the performance. This paper aims to fill this gap by exploring and investigating the contribution of auxiliary supervised learning tasks to improve the performance of PLMs for text generation. To this end, we observe the nature of text generation under three scenarios: incomplete utterance restoration, question generation, and summarization. The observation suggests that the indicators can be beneficial for text generation. We reflect this nature by defining a set of auxiliary tasks. Specifically, we design a model that considers both auxiliary tasks and the main text generation task under the joint training manner. The efficiency of the model is validated in the three generation scenarios. The results of the extensive experiments on 10 datasets for the three scenarios and languages show that the auxiliary tasks significantly contribute to improving the performance of PLMs for the main generation task. The ablation study facilitates the next studies of text generation by showing the behavior of the model in different settings. The model is simple, effective, and flexible for other text generation tasks.

PTE: Prompt tuning with ensemble verbalizers

Prompt tuning has achieved remarkable success in facilitating the performance of Pre-trained Language Models (PLMs) across various downstream NLP tasks, particularly in scenarios with limited downstream data. Reframing tasks as fill-in-the-blank questions represents an effective approach within prompt tuning. However, this approach necessitates the mapping of labels through a verbalizer consisting of one or more label tokens, constrained by manually crafted prompts. Furthermore, most existing automatic crafting methods either introduce external resources or rely solely on discrete or continuous optimization strategies. To address this issue, we have proposed a methodology for optimizing discrete verbalizers based on gradient descent, which we refer to this approach as PTE. This method integrates discrete tokens into verbalizers that can be continuously optimized, combining the distinct advantages of both discrete and continuous optimization strategies. In contrast to prior approaches, ours eschews reliance on prompts generated by other models or prior knowledge, merely augmenting a matrix. This approach boasts remarkable simplicity and flexibility, enabling prompt optimization while preserving the interpretability of output label tokens without constraints imposed by discrete vocabularies. Finally, employing this method in text classification tasks, we observe that PTE achieves results comparable to, if not surpassing, previous methods even under extreme conciseness. This furnishes a simple, intuitive, and efficient solution for automatically constructing verbalizers. Moreover, through quantitative analysis of optimized verbalizers, we uncover that language models likely rely not only on semantic information but also on other features for text classification. This revelation unveils new avenues for future research and model enhancements.

HoopTransformer: Advancing NBA Offensive Play Recognition with Self-Supervised Learning from Player Trajectories.

Understanding and recognizing basketball offensive set plays, which involve intricate interactions between players, have always been regarded as challenging tasks for untrained humans, not to mention machines. In this study, our objective is to propose an artificial intelligence model that can automatically recognize offensive plays using a novel self-supervised learning approach. The dataset was collected by SportVU from 632 games during the 2015-2016 season of the National Basketball Association (NBA), with a total of 90,524 possessions. A multi-agent motion prediction pretraining model was built on the basis of axial-attention transformer and trained with different masking strategies: motion prediction (MP), motion reconstruction (MR), and MP+MR joint strategy. A downstream play-level classification task and similarity search were used to evaluate the models' performance. The results showed that the MP+MR joint masking strategy maximized the ability of the model compared with individual masking strategies. For the classification task, the joint strategy achieved a top-1 accuracy of 81.5% and top-3 accuracy of 97.5%. In the similarity search evaluation, the joint strategy attained a top-5 accuracy of 76% and top-10 accuracy of 59%. Additionally, with the same MP+MR joint masking strategy, our HoopTransformer model outperformed the two baseline models in the classification task and similarity search. This study presents a self-supervised learning model and demonstrates the effectiveness and potential of the model in accurately comprehending and capturing player movements and complex interactions during offensive plays.

D-MHGCN: An End-to-End Individual Behavioral Prediction Model Using Dual Multi-Hop Graph Convolutional Network.

Predicting individual behavior is a crucial area of research in neuroscience. Graph Neural Networks (GNNs), as powerful tools for extracting graph-structured features, are increasingly being utilized in various functional connectivity (FC) based behavioral prediction tasks. However, current predictive models primarily focus on enhancing GNNs' ability to extract features from FC networks while neglecting the importance of upstream individual network construction quality. This oversight results in constructed functional networks that fail to adequately represent individual behavioral capacity, thereby affecting the subsequent prediction accuracy. To address this issue, we proposed a new GNN-based behavioral prediction framework, named Dual Multi-Hop Graph Convolutional Network (D-MHGCN). Through the joint training of two GCNs, this framework integrates individual functional network construction and behavioral prediction into a unified optimization model. It allows the model to dynamically adjust the individual functional cortical parcellation according to the downstream tasks, thus creating task-aware, individual-specific FCNs that largely enhance its ability to predict behavior scores. Additionally, we employed multi-hop graph convolution layers instead of traditional single-hop methods in GCN to capture complex hierarchical connectivity patterns in brain networks. Our experimental evaluations, conducted on the large, public Human Connectome Project dataset, demonstrate that our proposed method outperforms existing methods in various behavioral prediction tasks. Moreover, it produces more functionally homogeneous cortical parcellation, showcasing its practical utility and effectiveness. Our work not only enhances the accuracy of individual behavioral prediction but also provides deeper insights into the neural mechanisms underlying individual differences in behavior.

Norm constraints pyramid for image dehazing

Dark channel prior-based methods have achieved remarkable performance for image dehazing. However, previous studies are mostly focused on the accuracy of the assumptions used in the target scenes, which incurs color distortion and brightness reduction when the models are used for real-world hazy images.We propose a norm constraints pyramid framework to improve the generalization performance of dehazing. First, a local color adaptive correction approach is devised to ascertain whether there is any color bias and thereafter rectify it automatically. Furthermore, multiple norm constraint methods are developed to improve the transmission and accomplish the first image removal. Finally, a non-linear enhancement method is created via this restriction that precisely modifies the brightness of an image.Through extensive experiments, we demonstrate that our framework establishes the new state-of- the-art performance for real-world dehazing, in terms of visual quality assessed by no-reference quality metrics as well as subjective evaluation and downstream task performance indicator.

Exploiting Neighbor Effect: Conv-Agnostic GNN Framework for Graphs With Heterophily.

Due to the homophily assumption in graph convolution networks (GCNs), a common consensus in the graph node classification task is that graph neural networks (GNNs) perform well on homophilic graphs but may fail on heterophilic graphs with many interclass edges. However, the previous interclass edges' perspective and related homo-ratio metrics cannot well explain the GNNs' performance under some heterophilic datasets, which implies that not all the interclass edges are harmful to GNNs. In this work, we propose a new metric based on the von Neumann entropy to reexamine the heterophily problem of GNNs and investigate the feature aggregation of interclass edges from an entire neighbor identifiable perspective. Moreover, we propose a simple yet effective Conv-Agnostic GNN framework (CAGNNs) to enhance the performance of most GNNs on the heterophily datasets by learning the neighbor effect for each node. Specifically, we first decouple the feature of each node into the discriminative feature for downstream tasks and the aggregation feature for graph convolution (GC). Then, we propose a shared mixer module to adaptively evaluate the neighbor effect of each node to incorporate the neighbor information. The proposed framework can be regarded as a plug-in component and is compatible with most GNNs. The experimental results over nine well-known benchmark datasets indicate that our framework can significantly improve performance, especially for the heterophily graphs. The average performance gain is 9.81%, 25.81%, and 20.61% compared with graph isomorphism network (GIN), graph attention network (GAT), and GCN, respectively. Extensive ablation studies and robustness analysis further verify the effectiveness, robustness, and interpretability of our framework. Code is available at https://github.com/JC-202/CAGNN.

A heterogeneous graph transformer framework for accurate cancer driver gene prediction and downstream analysis

Accurately predicting cancer driver genes remains a formidable challenge amidst the burgeoning volume and intricacy of cancer genomic data. In this investigation, we propose HGTDG, an innovative heterogeneous graph transformer framework tailored for precisely predicting cancer driver genes and exploring downstream tasks. A heterogeneous graph construction module is central to the framework, which assembles a gene-protein heterogeneous network leveraging the Kyoto Encyclopedia of Genes and Genomes (KEGG) pathways and protein-protein interactions sourced from the STRING (search tool for recurring instances of neighboring genes) database. Moreover, our framework introduces a pioneering heterogeneous graph transformer module, harnessing multi-head attention mechanisms for nuanced node embedding. This transformative module proficiently captures distinct representations for both nodes and edges, thereby enriching the model's predictive capacity. Subsequently, the generated node embeddings are seamlessly integrated into a classification module, facilitating the discrimination between driver and non-driver genes. Our experimental findings evince the superiority of HGTDG over existing methodologies, as evidenced by the enhanced performance metrics, including the area under the receiver operating characteristic curves (AUROC) and the area under the precision-recall curves (AUPRC). Furthermore, the downstream analysis utilizing the newly identified cancer driver genes underscores the efficacy and versatility of our proposed framework.

Preserving text space integrity for robust compositional zero-shot learning via mixture of pretrained experts

In the current landscape of Compositional Zero-Shot Learning (CZSL) methods that leverage CLIP, the predominant approach is based on prompt learning paradigms. These methods encounter significant computational complexity when dealing with a large number of categories. Additionally, when confronted with new classification tasks, there is a necessity to learn the prompts again, which can be both time-consuming and resource-intensive. To address these challenges, We present a new methodology, named the Mixture of Pretrained Expert (MoPE), for enhancing Compositional Zero-shot Learning through Logit-Level Fusion with Mutile Expert Fusion Module. The MoPE skillfully blends the benefits of extensive pre-trained models like CLIP, Bert, GPT-3 and Word2Vec for effectively tackling Compositional Zero-shot Learning. Firstly, we extract the text label space for each language model individually, then map the visual feature vectors to their respective text spaces. This maintains the integrity and structure of the original text space. During this process, the pre-trained expert parameters are kept frozen. The mapping of visual features to the corresponding text spaces is subject to learning and could be considered as multiple learnable visual experts. In the model fusion phase, we propose a new fusion strategy that features a gating mechanism that adjusts the contributions of various models dynamically. This enables our approach to adapt more effectively to a range of tasks and data sets. The method’s robustness is demonstrated by the fact that the language model is not tailored to specific downstream task datasets or losses. This preserves the larger model’s topology and expands the potential for application. Preliminary experiments conducted on the UT-Zappos, AO-Clever, and C-GQA datasets indicate that MoPE performs competitively when compared to existing techniques.