Feature Weights Research Articles

Multiparametric MRI Radiomics With Machine Learning for Differentiating HER2-Zero, -Low, and -Positive Breast Cancer: Model Development, Testing, and Interpretability Analysis.

BACKGROUND: MRI radiomics has been explored for three-tiered classification of breast cancer HER2 expression (i.e., HER2-zero, HER2-low, or HER2-positive), although understanding of how such models reach their predictions is lacking. OBJECTIVE: To develop and test multiparametric MRI radiomics machine-learning models for differentiating three-tiered HER2 expression levels in patients with breast cancer, and to explain the contributions of model features through local and global interpretations using SHapley Additive exPlanation (SHAP) analysis. METHODS: This retrospective study included 737 patients (mean age, 54.1±10.6 years) with breast cancer from two centers (center 1: n=578; center 2: n=159), who underwent breast MRI and had HER2 expression determined after excisional biopsy. Analysis entailed two tasks: differentiating HER2-negative (i.e., HER2-zero or HER2-low) from HER2-positive tumors (task 1), and differentiating HER2-zero from HER2-low tumors (task 2). For each task, patients from center 1 were randomly assigned in 7:3 ratio to training (task 1: n=405; task 2: n=284) or internal test (task 1: n=173; task 2: n=122) sets; those from center 2 formed an external test set (task 1: n=159; task 2: n=105). Radiomics features were extracted from early-phase dynamic contrast-enhanced images (DCE), T2-weighted images (T2WI), and DWI. For each task, a support vector machine (SVM) was used for feature selection; a multiparametric radiomics score (radscore) was computed using feature weights from SVM correlation coefficients; conventional MRI and combined models were constructed; and model performances were evaluated. SHAP analysis was used to provide local and global interpretations for model outputs. RESULTS: In the external test set, for task 1, AUCs for the conventional MRI model, radscore, and combined model were 0.624, 0.757, and 0.762, respectively; for task 2, AUC for radscore was 0.754, and no conventional MRI model or combined model could be constructed. SHAP analysis identified early-phase DCE features as having the strongest influence for both tasks; T2WI features also had a prominent role for task 2. CONCLUSION: The findings indicate suboptimal performance of MRI radiomics models for noninvasive characterization of HER2 expression. CLINICAL IMPACT: The study provides an example of the use of SHAP interpretation analysis to better understand predictions of imaging-based machine learning models.

A Feature-Weighted Support Vector Regression Machine Based on Hilbert–Schmidt Independence Criterion Least Absolute Shrinkage and Selection Operator

Support vector regression (SVR) is a powerful kernel-based regression prediction algorithm that performs excellently in various application scenarios. However, for real-world data, the general SVR often fails to achieve good predictive performance due to its inability to assess feature contribution accurately. Feature weighting is a suitable solution to address this issue, applying correlation measurement methods to obtain reasonable weights for features based on their contributions to the output. In this paper, based on the idea of a Hilbert–Schmidt independence criterion least absolute shrinkage and selection operator (HSIC LASSO) for selecting features with minimal redundancy and maximum relevance, we propose a novel feature-weighted SVR that considers the importance of features to the output and the redundancy between features. In this approach, the HSIC is utilized to effectively measure the correlation between features as well as that between features and the output. The feature weights are obtained by solving a LASSO regression problem. Compared to other feature weighting methods, our method takes much more comprehensive consideration of weight calculation, and the obtained weighted kernel function can lead to more precise predictions for unknown data. Comprehensive experiments on real datasets from the University of California Irvine (UCI) machine learning repository demonstrate the effectiveness of the proposed method.

Estimation of Lower Limb Joint Angles Using sEMG Signals and RGB-D Camera.

Estimating human joint angles is a crucial task in motion analysis, gesture recognition, and motion intention prediction. This paper presents a novel model-based approach for generating reliable and accurate human joint angle estimation using a dual-branch network. The proposed network leverages combined features derived from encoded sEMG signals and RGB-D image data. To ensure the accuracy and reliability of the estimation algorithm, the proposed network employs a convolutional autoencoder to generate a high-level compression of sEMG features aimed at motion prediction. Considering the variability in the distribution of sEMG signals, the proposed network introduces a vision-based joint regression network to maintain the stability of combined features. Taking into account latency, occlusion, and shading issues with vision data acquisition, the feature fusion network utilizes high-frequency sEMG features as weights for specific features extracted from image data. The proposed method achieves effective human body joint angle estimation for motion analysis and motion intention prediction by mitigating the effects of non-stationary sEMG signals.

Physical information-guided multidirectional gated recurrent unit network fusing attention to solve the Black-Scholes equation

Reasonable option pricing is crucial in the financial derivatives market. Finding analytical solutions for the Black-Scholes (BS) equation, particularly for American options or with fluctuating volatility and interest rates, is challenging. BS equations exhibit strong time-series characteristics, with asset prices typically adhering to geometric Brownian motion. To address the BS equations, we propose a sequence-to-sequence model guided by physical information (PI), called PiMGA. The PiMGA fuses a multidirectional gated recurrent unit (GRU) network with an attention module, where multidirectional GRU enhances the coding performance of the input sequences and the attention module balances the feature weights of the hidden variables. Prior physical knowledge in BS equations is jointly used as a constraint, forming the penalty function for objective optimization. This allows PiMGA to serve as an efficient approximation function in the learning paradigm of physically informed machine learning to solve BS equations. BS equations with various complexities illustrate the accuracy and feasibility of PiMGA for numerical solutions. Furthermore, the out-of-distribution generalization ability of PiMGA is verified by predicting the Nasdaq 100 index.

Research on Multi-Sensor Fusion Fault Diagnosis Method Based on Spatiotemporal Attention Mechanism

As industrial systems become increasingly complex, real-time monitoring and intelligent management of industrial equipment have become imperative. However, the limitations in coverage and accuracy of single sensors make it challenging to comprehensively characterize the operational state of equipment, leading to reduced system reliability and increased pressures on data transmission and storage. To address these challenges, this study presents a novel fault diagnosis method based on multi-sensor fusion using a spatio-temporal attention mechanism. Initially, one-dimensional convolutional neural networks (1D-CNN) are employed to extract features from raw signals, effectively capturing local characteristics and ensuring the integrity and validity of fault signals. Subsequently, the spatiotemporal attention mechanism adjusts the feature weights based on the temporal and spatial correlations of different sensors, as well as their respective importance, thereby capturing the spatio-temporal dependencies across multiple sensors and enhancing the efficacy of information fusion. Finally, the proposed method is validated through experiments on a nickel flash smelting furnace system. The results demonstrate that the method achieves a fault diagnosis accuracy exceeding 97.78%, significantly enhancing fault detection and decision-making performance.

Risk Prediction Model for Tailings Ponds Based on EEMD-DA-LSTM Model

With the passage of time, the constant changes in relevant factors, and the daily maintenance of tailings ponds, the difficulty of tailings pond safety management is increasing day by day. In order to systematically improve the early warning ability for tailings pond dam break risk, the relationship between and influence of various related dam break risk factors of tailings ponds are utilised and the combination with dual attention is innovatively proposed. The risk prediction model for tailings ponds, EEMD-DA-LSTM, is improved. First, Pearson correlation coefficients are used to analyse the correlation between risk factors of tailings ponds. Then, the EEMD method is used to decompose the nonlinear displacement sequence, and the weights of input features are dynamically adjusted by double attention (DA). Finally, the LSTM network model is constructed to predict the displacement change. Taking valley-type tailings pond WKB-1 and mountainside tailings pond WKB-2 as examples, the dam break risk prediction models for tailings ponds are constructed based on three different models, the prediction results of different models are compared and analysed, and the prediction accuracy of the models is evaluated by three different evaluation criteria. The research results show that the integration of the EEMD-LSTM model with the DA model, that is, the EEMD-DA-LSTM model, has a better prediction effect for the dam break risk of tailings ponds WKB-1 and WKB-2 than other models through experimental verification. Therefore, the EEMD-DA-LSTM model is of great significance for preventing and resolving the safety risks of tailings ponds. It is valuable for practitioners in the mining industry and environmentally sustainable development.

Polycrystalline silicon photovoltaic cell defects detection based on global context information and multi-scale feature fusion in electroluminescence images

In photovoltaic (PV) cell inspection, electroluminescence (EL) imaging provides high spatial resolution for detecting various types of defects. The recent integration of EL imaging with deep learning models has enhanced the recognition of defects in PV cells. However, the high surface impurity content in polycrystalline silicon PV cells presents a challenge. Defect features can be similar to complex background textures, making highlighting features for small target defects difficult and leading to potential misclassification as background or other classes. To address these challenges, we propose a novel deep convolutional neural network (CNN) model for effectively identifying small target defects in polycrystalline PV cells. We first utilize a global context information (GCI) block to improve CNN’s modeling of global information, aiding in distinguishing PV cell defects with similar local details. Moreover, we design a channel weight feature pyramid (CWFP) that dynamically adjusts the channel weights of extracted features. We further achieve multi-scale feature fusion through the pyramid structure to enhance the resolution capability for small targets. The proposed model was evaluated on a publicly available dataset of 8 defect classes in polycrystalline PV cells, achieving an accuracy of 96.36 %. The experimental results show that the proposed model outperforms existing deep learning models in detecting small target defects with higher precision.

SFDA-T: A novel source-free domain adaptation method with strong generalization ability for fault diagnosis

Currently, source free domain adaptation (SFDA) methods are employed to address the issue of inaccessible source domain data (SDD) in transfer learning. However, existing SFDA methods often suffer from overfitting to specific domains, leading to poor generalization ability in the target domain. To address these challenges, this paper proposes a novel SFDA method named SFDA-T for fault diagnosis. Specifically, a Transformer-CNN-based feature extractor is constructed, to mine the transferable feature knowledge of faults in the SDD. The approach reduces the overfitting of the model to domain-specific information and improves model’s generalization ability. In addition, the feature attention loss is designed to calculate attention weights of the sample features to increase the model’s attention to the crucial feature regions in the target domain. A source similarity guided exponential loss is developed to guide target samples based on the decision boundaries of the source domain, facilitating cluster alignment of target sample categories and expanding distances between different categories. Furthermore, a self-training pseudo-labeling constraint is employed to reduce the effect of incorrect label matching and further constrain the model. The results of the experiments on gearboxes and bearings indicate that the proposed method achieves high fault diagnosis accuracy while effectively decoupling from SDD.

Support vector machine classification of irritable bowel syndrome patients based on whole-brain resting-state functional connectivity features.

Irritable bowel syndrome (IBS) is a disorder characterized by signaling dysregulation between the brain and gut, leading to gastrointestinal dysfunction. Symptoms such as abdominal pain and constipation can manifest periodically or persistently, and negative emotions may exacerbate the symptoms. Previous studies have shown that the pathogenesis of IBS is closely related to the brain-gut axis and brain function, but there are still difficulties in disease diagnosis. Therefore, this study applied a machine-learning approach based on resting-state functional magnetic resonance imaging (rs-fMRI) whole-brain functional connectivity (FC) to distinguish IBS patients from healthy controls (HCs). A total of 176 subjects, comprising 88 consecutive patients with IBS and 88 age-, sex- and education-matched HCs, were enrolled in this study between January 2020 and January 2024 at the First Affiliated Hospital of Shantou University Medical College. All the subjects underwent rs-fMRI and high-resolution anatomical T1-weighted imaging (T1WI) examinations. Following the preprocessing of the rs-fMRI image data, FC matrices between all regions of interest (ROIs) were extracted using automated anatomical labeling (AAL). Subsequently, supervised machine learning was performed using whole-brain FC for classification features to identify the best-performing model. Finally, weights of the optimal model's features were exported to confirm the neuroanatomical regions significantly influencing model establishment. Compared with other supervised learning models, the support vector machine (SVM) model had significantly higher classification accuracy and performed significantly better than the other models (P<0.05) with a classification accuracy of 75% and an area under the curve (AUC) of 0.7788 (95% confidence interval [CI]: 0.6861-0.8715) (P<0.01). In addition, the FC features from the Rolandic operculum (ROL) to the anterior cingulate gyrus (ACG), the calcarine sulcus (CAL) to the triangular part of the inferior frontal gyrus (IFG), the gyrus rectus (REC) to the inferior occipital gyrus (IOG), the lingual gyrus (LING) to the putamen (PUT), and the IOG to the angular gyrus (ANG) were relatively important in the construction of the machine-learning models. The SVM was the optimal machine-learning model for effectively classifying IBS patients and HCs based on whole-brain resting-state FC matrices. The FC features between the emotion-related brain regions significantly affected the construction of the machine-learning models. As a classification feature in machine learning, whole-brain resting-state FC holds the potential to achieve precision medicine in IBS and enhance disease diagnostic efficacy.

PS-Net: A Learning Strategy for Accurately Exposing the Professional Photoshop Inpainting.

Restoring missing areas without leaving visible traces has become a trivial task with Photoshop inpainting tools. However, such tools have potentially illegal or unethical uses, such as removing specific objects in images to deceive the public. Despite the emergence of many forensics methods of image inpainting, their detection ability is still insufficient when attending to professional Photoshop inpainting. Motivated by this, we propose a novel method termed primary-secondary network (PS-Net) to localize the Photoshop inpainted regions in images. To the best of our knowledge, this is the first forensic method devoted specifically to Photoshop inpainting. The PS-Net is designed to deal with the problems of delicate and professional inpainted images. It consists of two subnetworks: the primary network (P-Net) and the secondary network (S-Net). The P-Net aims at mining the frequency clues of subtle inpainting features through the convolutional network and further identifying the tampered region. The S-Net enables the model to mitigate compression and noise attacks to some extent by increasing the co-occurring feature weights and providing features that are not captured by the P-Net. Furthermore, the dense connection, Ghost modules, and channel attention blocks (C-A blocks) are adopted to further strengthen the localization ability of PS-Net. Extensive experimental results illustrate that PS-Net can successfully distinguish forged regions in elaborate inpainted images, outperforming several state-of-the-art solutions. The proposed PS-Net is also robust against some postprocessing operations commonly used in Photoshop.

Multimodal and multiscale feature fusion for weakly supervised video anomaly detection

Weakly supervised video anomaly detection aims to detect anomalous events with only video-level labels. In the absence of boundary information for anomaly segments, most existing methods rely on multiple instance learning. In these approaches, the predictions for unlabeled video snippets are guided by the classification of labeled untrimmed videos. However, these methods do not account for issues such as video blur and visual occlusion, which can hinder accurate anomaly detection. To address these issues, we propose a novel weakly supervised video anomaly detection method that fuses multimodal and multiscale features. Firstly, RGB and optical flow snippets are input into pre-trained I3D to extract appearance and motion features. Then, we introduce an Attention De-redundancy (AD) module, which employs an attention mechanism to filter out task-irrelevant redundancy in these appearance and motion features. Next, to mitigate the effects of video blurring and visual occlusion, we propose a Multi-scale Feature Learning module. This module captures long-term and short-term temporal dependencies among video snippets to provide global and local guidance for blurred or occluded video snippets. Finally, to effectively utilize the discriminative features of different modalities, we propose an Adaptive Feature Fusion module. This module adaptively fuses appearance and motion features based on their respective feature weights. Extensive experimental results demonstrate that our proposed method outperforms mainstream unsupervised and weakly supervised methods in terms of AUC. Specifically, our proposed method achieves 97.00% AUC and 85.31% AUC on two benchmark datasets, i.e., ShanghaiTech and UCF-Crime, respectively.

Assessing the limitations of relief-based algorithms in detecting higher-order interactions

BackgroundEpistasis, the interaction between genetic loci where the effect of one locus is influenced by one or more other loci, plays a crucial role in the genetic architecture of complex traits. However, as the number of loci considered increases, the investigation of epistasis becomes exponentially more complex, making the selection of key features vital for effective downstream analyses. Relief-Based Algorithms (RBAs) are often employed for this purpose due to their reputation as “interaction-sensitive” algorithms and uniquely non-exhaustive approach. However, the limitations of RBAs in detecting interactions, particularly those involving multiple loci, have not been thoroughly defined. This study seeks to address this gap by evaluating the efficiency of RBAs in detecting higher-order epistatic interactions. Motivated by previous findings that suggest some RBAs may rank predictive features involved in higher-order epistasis negatively, we explore the potential of absolute value ranking of RBA feature weights as an alternative approach for capturing complex interactions. In this study, we assess the performance of ReliefF, MultiSURF, and MultiSURFstar on simulated genetic datasets that model various patterns of genotype-phenotype associations, including 2-way to 5-way genetic interactions, and compare their performance to two control methods: a random shuffle and mutual information.ResultsOur findings indicate that while RBAs effectively identify lower-order (2 to 3-way) interactions, their capability to detect higher-order interactions is significantly limited, primarily by large feature count but also by signal noise. Specifically, we observe that RBAs are successful in detecting fully penetrant 4-way XOR interactions using an absolute value ranking approach, but this is restricted to datasets with only 20 total features.ConclusionsThese results highlight the inherent limitations of current RBAs and underscore the need for the development of Relief-based approaches with enhanced detection capabilities for the investigation of epistasis, particularly in datasets with large feature counts and complex higher-order interactions.

Clarifying the impact of engine operating parameters of heavy-duty diesel vehicles on NOx and CO2 emissions using multimodal fusion methods

The issue of air pollution from transportation sources remains a major concern, particularly the emissions from heavy-duty diesel vehicles, which pose serious threats to ecosystems and human health. China VI emission standards mandate On-Board Diagnostics (OBD) systems in heavy-duty diesel vehicles for real-time data transmission, yet the current data quality, especially concerning crucial parameters like NOx output, remains inadequate for effective regulation. To address this, a novel approach integrating Multimodal Feature Fusion with Particle Swarm Optimization (OBD-PSOMFF) is proposed. This network employs Long Short-Term Memory (LSTM) networks to extract features from OBD indicators, capturing temporal dependencies. PSO optimizes feature weights, enhancing prediction accuracy. Testing on 23 heavy-duty vehicles demonstrates significant improvements in predicting NOx and CO2 mass emission rates, with mean squared errors reduced by 65.205 % and 70.936 % respectively compared to basic LSTM models. This innovative multimodal fusion method offers a robust framework for emission prediction, crucial for effective vehicle emission regulation and environmental preservation.

A multi-task spatio-temporal fully convolutional model incorporating interaction patterns for traffic flow prediction

Previous traffic flow prediction studies have utilized spatio-temporal neural networks combined with the multi-task learning framework to seek complementary information for enhancing prediction performance. However, the existing methods still face two challenges: they fail to capture global interaction patterns between regions and lack consideration for inter-correlations within interaction patterns. To solve these issues, we propose a novel multi-task spatio-temporal fully convolutional model named MSTFCM. First, the model includes the interaction tensor and raster tensor as task inputs, where the interaction tensor extends the raster tensor by incorporating global interaction patterns between regions. Second, a multi-task framework combined spatio-temporal convolutional block was used to learn generalized features and interaction features. A channel spatio-temporal attention is added to adaptively adjust feature weights and capture inter-correlations. To train the MSTFCM, the uncertainty loss was designed as the learnable loss functions, which capture various flow fluctuations, to facilitate multi-task optimization. The proposed model was validated on two real-world traffic datasets collected in Xiamen, China. Experimental results showed that MSTFCM outperformed nine baselines in one-step and multi-step prediction, with slower performance degradation as predicted time intervals and steps increased. We further validated the model’s effectiveness through designed variants and visualization results.

Anomaly Detection Method for Industrial Control System Operation Data Based on Time-Frequency Fusion Feature Attention Encoding.

Anomaly detection in industrial control system (ICS) data is one of the key technologies for ensuring the security monitoring of ICSs. ICS data are characterized as complex, multi-dimensional, and long-sequence time-series data that embody ICS business logic. Due to its complex and varying periodic characteristics, as well as the presence of long-distance and misaligned temporal associations among features, current anomaly detection methods in ICS are insufficient for feature extraction. This paper proposes an anomaly detection method named TFANet, based on time-frequency fusion feature attention encoding. Considering that periodic variations are more concentrated in the frequency domain, this method first transforms the time-domain data into the frequency domain, obtaining both amplitude and phase data. Then, these data, together with the original time-series data, are used to extract features from two perspectives: long-term temporal changes and long-distance associations. Finally, the six features learned from both the time and frequency domains are fused, and the feature weights are calculated using an attention mechanism to complete the anomaly classification. In multi-classification tasks on three ICS datasets, the proposed method outperforms three popular time-series models-iTransformer, Crossformer, and TimesNet-across five metrics: accuracy, precision, recall, F1 score, and AUC-ROC, with average improvements of approximately 19%, 37%, 31%, 35%, and 22%, respectively.

Time-Series Feature Selection for Solar Flare Forecasting

Solar flares are significant occurrences in solar physics, impacting space weather and terrestrial technologies. Accurate classification of solar flares is essential for predicting space weather and minimizing potential disruptions to communication, navigation, and power systems. This study addresses the challenge of selecting the most relevant features from multivariate time-series data, specifically focusing on solar flares. We employ methods such as Mutual Information (MI), Minimum Redundancy Maximum Relevance (mRMR), and Euclidean Distance to identify key features for classification. Recognizing the performance variability of different feature selection techniques, we introduce an ensemble approach to compute feature weights. By combining outputs from multiple methods, our ensemble method provides a more comprehensive understanding of the importance of features. Our results show that the ensemble approach significantly improves classification performance, achieving values 0.15 higher in True Skill Statistic (TSS) values compared to individual feature selection methods. Additionally, our method offers valuable insights into the underlying physical processes of solar flares, leading to more effective space weather forecasting and enhanced mitigation strategies for communication, navigation, and power system disruptions.

MIDF-DMAP: Multimodal information dynamic fusion for drug molecule activity prediction

BackgroundComputer-aided drug development can alleviate limitations in drug research and development processes, such as long cycles, high costs, and limited targeting. Deep learning models can be used to predict the activity of drug compound molecules. However, existing models still need to adequately address semantic redundancy, conflicts, and noise between different data modalities. It results in low accuracy in predicting molecular activity and many false positive results that need further resolution. ObjectiveThis study aims to design a parallel deep learning model for processing multimodal molecular information fusion, dynamically adjusting the weights of different modal features during the fusion process. This approach ensures that high-quality modal information is more significant in molecular activity prediction, improving the model’s predictive accuracy. MethodsFirst, three modal features of drug molecules are extracted: one-dimensional fingerprints, two-dimensional topological structures, and three-dimensional geometric structures. Then, a dynamic weighting mechanism is proposed to fuse these three modal features organically. Finally, the fused features are input into the model classifier to obtain molecular activity prediction results. ResultsExperimental results show that the proposed model improved the Receiver Operating Characteristic-Area Under the Curve (ROC-AUC) value by 29.96% compared to the Weave model and 26.38% compared to the GC model on the BBBP dataset. Compared to the GraphMVP model and the multi-channel substructure graph model MSGG, the ROC-AUC value increased by 20.44% and 15.80%, respectively. This method also achieved good results on imbalanced samples and small molecule datasets. ConclusionThis study provides an effective multimodal dynamic fusion method for virtual drug screening, enriching computer-aided drug development theory and improving drug research and development efficiency.

An Improved Instance Segmentation Method for Complex Elements of Farm UAV Aerial Survey Images.

Farm aerial survey layers can assist in unmanned farm operations, such as planning paths and early warnings. To address the inefficiencies and high costs associated with traditional layer construction, this study proposes a high-precision instance segmentation algorithm based on SparseInst. Considering the structural characteristics of farm elements, this study introduces a multi-scale attention module (MSA) that leverages the properties of atrous convolution to expand the sensory field. It enhances spatial and channel feature weights, effectively improving segmentation accuracy for large-scale and complex targets in the farm through three parallel dense connections. A bottom-up aggregation path is added to the feature pyramid fusion network, enhancing the model's ability to perceive complex targets such as mechanized trails in farms. Coordinate attention blocks (CAs) are incorporated into the neck to capture richer contextual semantic information, enhancing farm aerial imagery scene recognition accuracy. To assess the proposed method, we compare it against existing mainstream object segmentation models, including the Mask R-CNN, Cascade-Mask, SOLOv2, and Condinst algorithms. The experimental results show that the improved model proposed in this study can be adapted to segment various complex targets in farms. The accuracy of the improved SparseInst model greatly exceeds that of Mask R-CNN and Cascade-Mask and is 10.8 and 12.8 percentage points better than the average accuracy of SOLOv2 and Condinst, respectively, with the smallest number of model parameters. The results show that the model can be used for real-time segmentation of targets under complex farm conditions.

Triple-Stream Commonsense Circulation Transformer Network for Image Captioning

Traditional image captioning methods only have a local perspective at the dataset level, allowing them to explore dispersed information within individual images. However, the lack of a global perspective prevents them from capturing common characteristics among similar images. To address the limitation, this paper introduces a novel Triple-stream Commonsense Circulating Transformer Network (TCCTN). It incorporates contextual stream into the encoder, combining enhanced channel stream and spatial stream for comprehensive feature learning. The proposed commonsense-aware contextual attention (CCA) module queries commonsense contextual features from the dataset, obtaining global contextual association information by projecting grid features into the contextual space. The pure semantic channel attention (PSCA) module leverages compressed spatial domain for channel pooling, focusing on attention weights of pure channel features to capture inherent semantic features. The region spatial attention (RSA) module enhances spatial concepts in semantic learning by incorporating region position information. Furthermore, leveraging the complementary differences among the three features, TCCTN introduces the mixture of experts strategy to enhance the unique discriminative ability of features and promote their integration in textual feature learning. Extensive experiments on the MS-COCO dataset demonstrate the effectiveness of contextual commonsense stream and the superior performance of TCCTN.

Prior-guided attention fusion transformer for multi-lesion segmentation of diabetic retinopathy

To solve the issue of diagnosis accuracy of diabetic retinopathy (DR) and reduce the workload of ophthalmologists, in this paper we propose a prior-guided attention fusion Transformer for multi-lesion segmentation of DR. An attention fusion module is proposed to improve the key generator to integrate self-attention and cross-attention and reduce the introduction of noise. The self-attention focuses on lesions themselves, capturing the correlation of lesions at a global scale, while the cross-attention, using pre-trained vessel masks as prior knowledge, utilizes the correlation between lesions and vessels to reduce the ambiguity of lesion detection caused by complex fundus structures. A shift block is introduced to expand association areas between lesions and vessels further and to enhance the sensitivity of the model to small-scale structures. To dynamically adjust the model’s perception of features at different scales, we propose the scale-adaptive attention to adaptively learn fusion weights of feature maps at different scales in the decoder, capturing features and details more effectively. The experimental results on two public datasets (DDR and IDRiD) demonstrate that our model outperforms other state-of-the-art models for multi-lesion segmentation.