Mutual Guidance Research Articles

PMGNet: Disentanglement and entanglement benefit mutually for compositional zero-shot learning

Compositional zero-shot learning (CZSL) aims to model compositions of two primitives (i.e., attributes and objects) to classify unseen attribute-object pairs. Most studies are devoted to integrating disentanglement and entanglement strategies to circumvent the trade-off between contextuality and generalizability. Indeed, the two strategies can mutually benefit when used together. Nevertheless, they neglect the significance of developing mutual guidance between the two strategies. In this work, we take full advantage of guidance from disentanglement to entanglement and vice versa. Additionally, we propose exploring multi-scale feature learning to achieve fine-grained mutual guidance in a progressive framework. Our approach, termed Progressive Mutual Guidance Network (PMGNet), unifies disentanglement–entanglement representation learning, allowing them to learn from and teach each other progressively in one unified model. Furthermore, to alleviate overfitting recognition on seen pairs, we adopt a relaxed cross-entropy loss to train PMGNet, without an increase of time and memory cost. Extensive experiments on three benchmarks demonstrate that our method achieves distinct improvements, reaching state-of-the-art performance. Moreover, PMGNet exhibits promising performance under the most challenging open-world CZSL setting, especially for unseen pairs.

MTLSC-Diff: Multitask learning with diffusion models for hyperspectral image super-resolution and classification

Hyperspectral image (HSIs) super-resolution (SR) can improve the spatial resolution of images for subsequent application tasks. In recent years, SR methods based on deep learning have gained widespread attention. However, most of the existing SR methods do not take into account the needs of specific application tasks when designing the network structure. These methods may not be able to efficiently generate high-quality images that satisfy the specific application tasks, leading to degradation of the performance of subsequent application tasks. To solve this problem, we propose a multi-task learning architecture based on the diffusion model, namely MTLSC-Diff. MTLSC-Diff combines the SR network and the classification network in a multi-task learning manner on the basis of the diffusion model. MTLSC-Diff achieves mutual guidance of the two tasks by iterating the image super-resolution and classification tasks, thus gradually reconstructing high-quality images and improving classification accuracy. The guided operations for each time step are performed by the specially designed Mutual-Guidance SR-Classification Synergy Module (M-GSCS). M-GSCS refines the multi-scale image obtained at the previous time step and uses the predicted high spatial resolution image for classification. Meanwhile, a class-guided SR dynamic refinement strategy (C-GSR) is proposed in M-GSCS, which uses multi-scale classification results to guide target scale images to learn new knowledge to further reconstruct high-quality images. Experimental results on relevant datasets show that our method significantly improves the super-resolution performance as well as the classification performance.

TriCAFFNet: A Tri-Cross-Attention Transformer with a Multi-Feature Fusion Network for Facial Expression Recognition.

In recent years, significant progress has been made in facial expression recognition methods. However, tasks related to facial expression recognition in real environments still require further research. This paper proposes a tri-cross-attention transformer with a multi-feature fusion network (TriCAFFNet) to improve facial expression recognition performance under challenging conditions. By combining LBP (Local Binary Pattern) features, HOG (Histogram of Oriented Gradients) features, landmark features, and CNN (convolutional neural network) features from facial images, the model is provided with a rich input to improve its ability to discern subtle differences between images. Additionally, tri-cross-attention blocks are designed to facilitate information exchange between different features, enabling mutual guidance among different features to capture salient attention. Extensive experiments on several widely used datasets show that our TriCAFFNet achieves the SOTA performance on RAF-DB with 92.17%, AffectNet (7 cls) with 67.40%, and AffectNet (8 cls) with 63.49%, respectively.

HQ-Net: A heatmap-based query backbone for point cloud understanding

With the advancement of sensor technology, including LiDAR and RGB-D cameras, tasks related to point cloud understanding have gained widespread application. The sampling and learning of feature points play a crucial role in point cloud understanding, representing a fundamental and essential aspect of 3D computer vision. In the realm of point cloud processing, traditional methods such as Farthest Point Sampling (FPS) are frequently employed to select feature points. These points are then used in conjunction with effective feature aggregation operators to facilitate reasoning. However, these traditional sampling methods primarily focus on the Euclidean distance between points, lacking a comprehensive consideration of point cloud features. To address this limitation, we introduce HQ-Net, a novel point cloud understanding model designed to optimize feature point sampling using a Heatmap-based Query Backbone. The Heatmap-based Query Backbone utilizes the Heatmap Module to select category-specific point features, effectively generating a heatmap tailored to the category features and automatically extracting features from the heatmap. Subsequently, the proposed Query Encoder is employed to facilitate the interaction between point features and query features, providing mutual guidance and enhancing the model’s ability to capture spatial structure and semantic information within point clouds. Additionally, we propose task-specific heads, such as classification and segmentation heads for different tasks, which can be seamlessly integrated with the backbone to achieve excellent performance. In various point cloud understanding tasks, including shape classification, part segmentation, and semantic segmentation, HQ-Net demonstrates excellent performance.

Position-aware Interactive Attention Network for multi-intent spoken language understanding

Spoken Language Understanding (SLU) is a crucial component of task-oriented dialog systems. In recent years, there has been increasing attention on multi-intent SLU due to its relevance to complex real-world applications. Most existing joint models only utilize multi-intent information to guide the slot filling, with only a small number of models achieving bidirectional interaction. Additionally, unlike traditional single-intent SLU, multi-intent SLU is scope-dependent, where each intent in a sentence has its specific dependency range. However, current bidirectional joint models often employ a pipeline approach to implement interaction between the two sub-tasks, which fails to fully leverage the semantic information between them and can lead to error propagation. Moreover, attention-based multi-intent joint models do not adequately model the positional relationships between words, resulting in suboptimal overall performance. In this paper, we propose a novel multi-intent SLU model called Position-aware Interactive Attention Network (PIAN), which consists of interactive attention and rotary position embedding. The aim of PIAN is to fully exploit the correlation between the two sub-tasks and capturing the positional dependencies between words. This facilitates mutual guidance and enhancement between the two sub-tasks. Additionally, to address the issue of uncoordinated slot problem generated by traditional slot filling decoders, we employ an MCRF slot filling decoder to constrain the slot labels. We evaluate our model on two public multi-intent SLU datasets, and the experimental results demonstrate that our model achieves state-of-the-art performance on key metrics. Code for this paper is publicly available at https://github.com/puhahahahahaha/SLU_with_Co_PRoE.

A joint model for lesion segmentation and classification of MS and NMOSD.

Multiple sclerosis (MS) and neuromyelitis optic spectrum disorder (NMOSD) are mimic autoimmune diseases of the central nervous system with a very high disability rate. Their clinical symptoms and imaging findings are similar, making it difficult to diagnose and differentiate. Existing research typically employs the T2-weighted fluid-attenuated inversion recovery (T2-FLAIR) MRI imaging technique to focus on a single task in MS and NMOSD lesion segmentation or disease classification, while ignoring the collaboration between the tasks. To make full use of the correlation between lesion segmentation and disease classification tasks of MS and NMOSD, so as to improve the accuracy and speed of the recognition and diagnosis of MS and NMOSD, a joint model is proposed in this study. The joint model primarily comprises three components: an information-sharing subnetwork, a lesion segmentation subnetwork, and a disease classification subnetwork. Among them, the information-sharing subnetwork adopts a dualbranch structure composed of a convolution module and a Swin Transformer module to extract local and global features, respectively. These features are then input into the lesion segmentation subnetwork and disease classification subnetwork to obtain results for both tasks simultaneously. In addition, to further enhance the mutual guidance between the tasks, this study proposes two information interaction methods: a lesion guidance module and a crosstask loss function. Furthermore, the lesion location maps provide interpretability for the diagnosis process of the deep learning model. The joint model achieved a Dice similarity coefficient (DSC) of 74.87% on the lesion segmentation task and accuracy (ACC) of 92.36% on the disease classification task, demonstrating its superior performance. By setting up ablation experiments, the effectiveness of information sharing and interaction between tasks is verified. The results show that the joint model can effectively improve the performance of the two tasks.

MFMENet: multi-scale features mutual enhancement network for change detection in remote sensing images

ABSTRACT Change detection, an important task in remote sensing image analysis, has been extensively studied in recent years. However, change detection still faces problems such as difficulty in detecting small targets and incomplete edge detection. Furthermore, pseudo changes such as seasonal changes can also lead to many false detections. In response to these challenges, we propose the Multi-scale Features Mutual Enhancement Network (MFMENet), a simple yet efficient network. MFMENet maximizes feature utilization through mutual guidance and supplementation, enhancing the detection capabilities for small targets and edges. First, we use a lightweight feature extraction network to extract features, which mitigates the information loss caused by continuous downsampling of an overly deep network structure. Then, we design a Context Adaptive Interaction Module (CAIM) to realize the complementarity of feature information at different levels. This facilitates shallow features in gaining more semantic information and deep features in acquiring more texture information, thereby enhancing the model’s capability to capture more comprehensive edge features while effectively mitigating interference from pseudo changes. Finally, we introduce a Feature Aggregation Comparison Module (FACM), which uses a combination of aggregation and comparison methods to refine and enhance features. FACM can not only highlight the changed features but also retain more details, improving the model’s detection ability of small targets and edge details. The full utilization of features and effective mutual enhancement of information ensure the improvement of MFMENet’s performance in small target and edge detection. Extensive experiments on three publicly available datasets (LEVIR, DSIFN, and CDD) demonstrate that our approach achieves superior performance with fewer parameters compared to state-of-the-art methods in recent years. In comparison to these baseline methods, our proposed approach achieves improvements of 0.98%, 12.24%, and 2.03% in the IOU metric on the LEVIR, DSIFN, and CDD datasets, respectively, while utilizing only 1.1 M parameters.

Exploring modality enhancement and compensation spaces for visible-infrared person re-identification

Visible-infrared person re-identification (VI-ReID) is a challenging task in computer vision due to the substantial modality gaps between visible and infrared images. The currently existing approaches can improve performance by addressing cross-modality discrepancies, but they often fail to generate compensation features that fully utilize the unique information present in each modality. Additionally, these methods mainly focus on pixel-level fusion of images, disregarding the challenge of modality misalignment. To address these issues, we propose a novel visible-infrared person re-identification method that explores modality enhancement and compensation spaces to extract more discriminative modality information. Furthermore, we introduce a modality mutual guidance strategy incorporating identity information mutual learning loss and modality-guided alignment loss, which can effectively leverage learned identity-related feature to guide alignment between visible and infrared modalities. Extensive experiments on public datasets demonstrate the significant superiority of our proposed method over existing state-of-the-art approaches.

Knowledge-Embedded Mutual Guidance for Visual Reasoning.

Visual reasoning between visual images and natural language is a long-standing challenge in computer vision. Most of the methods aim to look for answers to questions only on the basis of the analysis of the offered questions and images. Other approaches treat knowledge graphs as flattened tables to search for the answer. However, there are two major problems with these works: 1) the model disregards the fact that the world we surrounding us interlinks our hearing and speaking of natural language and 2) the model largely ignores the structure of the acrlong KG. To overcome these challenging deficiencies, a model should jointly consider two modalities of vision and language, as well as the rich structural and logical information embedded in knowledge graphs. To this end, we propose a general joint representation learning framework for visual reasoning, namely, knowledge-embedded mutual guidance. It realizes mutual guidance not only between visual data and natural language descriptions but also between knowledge graphs and reasoning models. In addition, it exploits the knowledge derived from the reasoning model to boost knowledge graphs when applying the visual relation detection task. The experimental results demonstrate that the proposed approach performs dramatically better than state-of-the-art methods on two benchmarks for visual reasoning.

Towards Multi-Intent Spoken Language Understanding via Hierarchical Attention and Optimal Transport

Multi-Intent spoken language understanding (SLU) can handle complicated utterances expressing multiple intents, which has attracted increasing attention from researchers. Although existing models have achieved promising performance, most of them still suffer from two leading problems: (1) each intent has its specific scope and the semantic information outside the scope might potentially hinder accurate predictions, i.e. scope barrier; (2) only the guidance from intent to slot is modeled but the guidance from slot to intent is often neglected, i.e. unidirectional guidance. In this paper, we propose a novel Multi-Intent SLU framework termed HAOT, which utilizes hierarchical attention to divide the scopes of each intent and applies optimal transport to achieve the mutual guidance between slot and intent. Experiments demonstrate that our model achieves state-of-the-art performance on two public Multi-Intent SLU datasets, obtaining the 3.4 improvement on MixATIS dataset compared to the previous best models in overall accuracy.

Unsupervised Pan-Sharpening via Mutually Guided Detail Restoration

Pan-sharpening is a task that aims to super-resolve the low-resolution multispectral (LRMS) image with the guidance of a corresponding high-resolution panchromatic (PAN) image. The key challenge in pan-sharpening is to accurately modeling the relationship between the MS and PAN images. While supervised deep learning methods are commonly employed to address this task, the unavailability of ground-truth severely limits their effectiveness. In this paper, we propose a mutually guided detail restoration method for unsupervised pan-sharpening. Specifically, we treat pan-sharpening as a blind image deblurring task, in which the blur kernel can be estimated by a CNN. Constrained by the blur kernel, the pan-sharpened image retains spectral information consistent with the LRMS image. Once the pan-sharpened image is obtained, the PAN image is blurred using a pre-defined blur operator. The pan-sharpened image, in turn, is used to guide the detail restoration of the blurred PAN image. By leveraging the mutual guidance between MS and PAN images, the pan-sharpening network can implicitly learn the spatial relationship between the two modalities. Extensive experiments show that the proposed method significantly outperforms existing unsupervised pan-sharpening methods.

MFMG-Net: Multispectral Feature Mutual Guidance Network for Visible–Infrared Object Detection

Drones equipped with visible and infrared sensors play a vital role in urban road supervision. However, conventional methods using RGB-IR image pairs often struggle to extract effective features. These methods treat these spectra independently, missing the potential benefits of their interaction and complementary information. To address these challenges, we designed the Multispectral Feature Mutual Guidance Network (MFMG-Net). To prevent learning bias between spectra, we have developed a Data Augmentation (DA) technique based on the mask strategy. The MFMG module is embedded between two backbone networks, promoting the exchange of feature information between spectra to enhance extraction. We also designed a Dual-Branch Feature Fusion (DBFF) module based on attention mechanisms, enabling deep feature fusion by emphasizing correlations between the two spectra in both the feature channel and space dimensions. Finally, the fused features feed into the neck network and detection head, yielding ultimate inference results. Our experiments, conducted on the Aerial Imagery (VEDAI) dataset and two other public datasets (M3FD and LLVIP), showcase the superior performance of our method and the effectiveness of MFMG in enhancing multispectral feature extraction for drone ground detection.

Integrating image and gene-data with a semi-supervised attention model for prediction of KRAS gene mutation status in non-small cell lung cancer.

KRAS is a pathogenic gene frequently implicated in non-small cell lung cancer (NSCLC). However, biopsy as a diagnostic method has practical limitations. Therefore, it is important to accurately determine the mutation status of the KRAS gene non-invasively by combining NSCLC CT images and genetic data for early diagnosis and subsequent targeted therapy of patients. This paper proposes a Semi-supervised Multimodal Multiscale Attention Model (S2MMAM). S2MMAM comprises a Supervised Multilevel Fusion Segmentation Network (SMF-SN) and a Semi-supervised Multimodal Fusion Classification Network (S2MF-CN). S2MMAM facilitates the execution of the classification task by transferring the useful information captured in SMF-SN to the S2MF-CN to improve the model prediction accuracy. In SMF-SN, we propose a Triple Attention-guided Feature Aggregation module for obtaining segmentation features that incorporate high-level semantic abstract features and low-level semantic detail features. Segmentation features provide pre-guidance and key information expansion for S2MF-CN. S2MF-CN shares the encoder and decoder parameters of SMF-SN, which enables S2MF-CN to obtain rich classification features. S2MF-CN uses the proposed Intra and Inter Mutual Guidance Attention Fusion (I2MGAF) module to first guide segmentation and classification feature fusion to extract hidden multi-scale contextual information. I2MGAF then guides the multidimensional fusion of genetic data and CT image data to compensate for the lack of information in single modality data. S2MMAM achieved 83.27% AUC and 81.67% accuracy in predicting KRAS gene mutation status in NSCLC. This method uses medical image CT and genetic data to effectively improve the accuracy of predicting KRAS gene mutation status in NSCLC.

ScalableTrack: Scalable One-Stream Tracking via Alternating Learning.

Transformer-based one-stream trackers are widely used to extract features and interact information for visual object tracking. However, the current one-stream tracker has fixed computational dimensions between different stages, which limits the network's ability to learn context clues and global representations, resulting in a decrease in the ability to distinguish between targets and backgrounds. To address this issue, a new scalable one-stream tracking framework, ScalableTrack, is proposed. It unifies feature extraction and information integration by intrastage mutual guidance, leveraging the scalability of target-oriented features to enhance object sensitivity and obtain discriminative global representations. In addition, we bridge interstage contextual cues by introducing an alternating learning strategy and solve the arrangement problem of the two modules. The alternating learning strategy uses alternate stacks of feature extraction and information interaction to focus on tracked objects and prevent catastrophic forgetting of target information between different stages. Experiments on eight challenging benchmarks (TrackingNet, GOT-10k, VOT2020, UAV123, LaSOT, LaSOT [Formula: see text] , OTB100, and TC128) show that ScalableTrack outperforms state-of-the-art (SOTA) methods with better generalization and global representation ability.

Research on the Path Construction and Practice of Bidirectional Cultural Mutual Guidance in Higher Vocational English Course Civics Teaching under the Background of Big Data

Abstract This study delves into the effective integration of bi-directional cultural intercultural instruction with Civic and Political teaching in a higher vocational English course and analyzes it with an association rule mining algorithm. By screening the weighted candidate itemsets through the weighted support degree model, the study identifies the itemsets’ positive and negative correlation attributes and mines the association rules between English teaching and students’ bidirectional cultural interduction ability by combining with the decision support model. The article also analyzes the content of Civics and Chinese and foreign cultural elements in the English curriculum. It compares and examines the changes in students’ two-way cultural core literacy and its introduction effect. The results of the study show that after the bi-directional cultural Civics teaching in the English curriculum, students experienced significant changes in worldview, values and Chinese and foreign cultural cognition, with t-values of −4.312, −5.034, and −4.275, respectively, and p-values of less than 0.001. In addition, the students’ bi-directional cultural communication competence improved significantly, with scores ranging from 3.8 to 4.3. The study provides valuable insights into the promotion of Chinese-foreign cultural communication and the reform of higher vocational English teaching.

Explainable Pedestrian Crossing Intention Prediction Based On Multi-Task Mutual Guidance Network

Explainable Pedestrian Crossing Intention Prediction Based On Multi-Task Mutual Guidance Network

Multiple Guidance Network for Industrial Product Surface Inspection With One Labeled Target Sample.

Most automatic product surface inspection methods in industry are data-hungry and task-specific. It is difficult to collect adequate labeled samples in practice due to factors including expensive data annotation cost, inadequate samples for some categories, and limitations on the initial production stage. In this article, a multiple guidance network (MGNet) is proposed to address these issues. In the network, the feature extraction machine (FEM) produces four feature maps of different functions to enhance the inspection ability of the algorithm. Also, the probability map generation (PMG) module is designed for coarse positioning of objects. Moreover, the structures of the mutual guidance and historical guidance (HG) guarantee that the network can fully utilize the information of the auxiliary dataset. Only one support sample containing the labeled objects is required for reference, and the network can determine whether the same labeled objects exist in the query images and locate them. For a comprehensive evaluation of MGNet, three experiments are carried out using three real-world datasets. Experiment results verify that the proposed method is promising for industrial product surface inspection with one labeled target sample.

CDS-Net: Cooperative dual-stream network for image manipulation detection

To accurately locate manipulated regions, many existing approaches employ a dual-stream framework to extract a wide range of manipulation clues, including local noise, edge artifacts, and global inconsistency. However, these approaches treat each stream in isolation and fail to consider the complementary and mutual guidance ability between the streams. Moreover, we notice the use of vanilla vision transformers in previous approaches can result in disruptions of object semantics, causing incomplete predictions. To address these challenges, we introduce the cooperative dual-stream network (CDS-Net) comprising an RGB Stream and a Noise Stream. In the Noise Stream, we propose a K-means Transformer (KT) that encourages both inter-patch and intra-patch information transmission to mitigate the semantic fragmentation phenomenon caused by patch partitioning. Additionally, we introduce a novel Feature Interaction Block (FIB) that explicitly encourages cross-stream collaboration at each encoding stage. Comprehensive experiments on publicly available datasets demonstrate the effectiveness and robustness of CDS-Net.

MTS-DVGAN: Anomaly detection in cyber-physical systems using a dual variational generative adversarial network

Deep generative models are promising in detecting novel cyber-physical attacks, mitigating the vulnerability of Cyber-physical systems (CPSs) without relying on labeled information. Nonetheless, these generative models face challenges in identifying attack behaviors that closely resemble normal data, or deviate from the normal data distribution but are in close proximity to the manifold of the normal cluster in latent space. To tackle this problem, this article proposes a novel unsupervised dual variational generative adversarial model named MST-DVGAN, to perform anomaly detection in multivariate time series data for CPS security. The central concept is to enhance the model's discriminative capability by widening the distinction between reconstructed abnormal samples and their normal counterparts. Specifically, we propose an augmented module by imposing contrastive constraints on the reconstruction process to obtain a more compact embedding. Then, by exploiting the distribution property and modeling the normal patterns of multivariate time series, a variational autoencoder is introduced to force the generative adversarial network (GAN) to generate diverse samples. Furthermore, two augmented loss functions are designed to extract essential characteristics in a self-supervised manner through mutual guidance between the augmented samples and original samples. Finally, a specific feature center loss is introduced for the generator network to enhance its stability. Empirical experiments are conducted on three public datasets, namely SWAT, WADI and NSL_KDD. Comparing with the state-of-the-art methods, the evaluation results show that the proposed MTS-DVGAN is more stable and can achieve consistent performance improvement.

Two-Way Complementary Tracking Guidance

Recently, most impressive Siamese network-based trackers are equipped with two independent branches: tracked object classification and bounding box regression. However, there is no tracking information exchange between them during the tracking optimization process. This may lead to the task-mismatch and accuracy inconsistency between both classification and regression branches during inference. To tackle the problems, we propose a novel Mutual Guidance (MG) strategy for visual object tracking, which constructs the bidirectional and complementary tracking information interaction to maintain the tracked object is well-classified to also be well-localized, between classification and regression branches. Specifically, the classification branch can guide the regression one to pay more attention to the sample with high classified scores, by re-weighting the regression loss with the classification confidence. Similarity, the regression branch also guides the classifier optimization process to focus on samples with larger IoU values. And then, the proposed Mutual Guidance is completed by a series of regularization designs on classification score and regression IoU, which dynamically re-assign the adaptive weights to the losses for each sample during the joint tracking optimization. The developed MG is generic and easy to be plugged into various tracking frameworks such as anchor-based, anchor-free based and transformer based, and boost their performance to some extent with negligible additional cost. In addition, we also develop an adaptive localization(L) branch selection scheme to further assist trackers, which determines proper localization branch for different trackers according to the difference in the way of discriminating positive and negative samples. Extensive experiments verify the effectiveness of MGL and its superiority against the state-of-the-art tracking modules on OTB100, GOT-10K, LaSOT, TrackingNet, UAV123, VOT2018 and VOT2019.