Network Transformation Research Articles

ResNet-Vision Transformer based MRI-endoscopy fusion model for predicting treatment response to neoadjuvant chemoradiotherapy in locally advanced rectal cancer: A multicenter study.

Neoadjuvant chemoradiotherapy followed by radical surgery has been a common practice for patients with locally advanced rectal cancer, but the response rate varies among patients. This study aimed to develop a ResNet-Vision Transformer based magnetic resonance imaging (MRI)-endoscopy fusion model to precisely predict treatment response and provide personalized treatment. In this multicenter study, 366 eligible patients who had undergone neoadjuvant chemoradiotherapy followed by radical surgery at eight Chinese tertiary hospitals between January 2017 and June 2024 were recruited, with 2928 pretreatment colonic endoscopic images and 366 pelvic MRI images. An MRI-endoscopy fusion model was constructed based on the ResNet backbone and Transformer network using pretreatment MRI and endoscopic images. Treatment response was defined as good response or non-good response based on the tumor regression grade. The Delong test and the Hanley-McNeil test were utilized to compare prediction performance among different models and different subgroups, respectively. The predictive performance of the MRI-endoscopy fusion model was comprehensively validated in the test sets and was further compared to that of the single-modal MRI model and single-modal endoscopy model. The MRI-endoscopy fusion model demonstrated favorable prediction performance. In the internal validation set, the area under the curve (AUC) and accuracy were 0.852 (95% confidence interval [CI]: 0.744-0.940) and 0.737 (95% CI: 0.712-0.844), respectively. Moreover, the AUC and accuracy reached 0.769 (95% CI: 0.678-0.861) and 0.729 (95% CI: 0.628-0.821), respectively, in the external test set. In addition, the MRI-endoscopy fusion model outperformed the single-modal MRI model (AUC: 0.692 [95% CI: 0.609-0.783], accuracy: 0.659 [95% CI: 0.565-0.775]) and the single-modal endoscopy model (AUC: 0.720 [95% CI: 0.617-0.823], accuracy: 0.713 [95% CI: 0.612-0.809]) in the external test set. The MRI-endoscopy fusion model based on ResNet-Vision Transformer achieved favorable performance in predicting treatment response to neoadjuvant chemoradiotherapy and holds tremendous potential for enabling personalized treatment regimens for locally advanced rectal cancer patients.

Advanced multiple document summarization via iterative recursive transformer networks and multimodal transformer

The proliferation of digital information necessitates advanced techniques for multiple document summarization, capable of distilling vast textual data efficiently. Traditional approaches often struggle with coherence, integration of multimodal data, and suboptimal learning strategies. To address these challenges, this work introduces novel neural architectures and methodologies. At its core is recursive transformer networks (ReTran), merging recursive neural networks with transformer architectures for superior comprehension of textual dependencies, projecting a 5–10% improvement in ROUGE scores. Cross-modal summarization employs a multimodal transformer with cross-modal attention, amalgamating text, images, and metadata for more holistic summaries, expecting an 8 to 12% enhancement in quality metrics. Actor-critic reinforcement learning refines training by optimizing summary quality, surpassing Q-learning-based strategies by 5–8%. Meta-learning for zero-shot summarization addresses summarizing unseen domains, projecting a 6–10% uptick in performance. Knowledge-enhanced transformer integrates external knowledge for improved semantic coherence, potentially boosting ROUGE scores by 7 to 12%. These advancements not only improve numerical performance but also produce more informative and coherent summaries across diverse domains and modalities. This work represents a significant stride in multiple document summarization, setting a new benchmark for future research and applications.

ASFormer: Attentive Semantic Feature Fusion Transformer for Pixel -level Defect Detection

Abstract Surface defect detection is pivotal for ensuring product quality in various industries. These defects typically manifest as low background contrast, substantial variations in shape, and a scarcity of balanced training samples. Traditional defect detection methods often encounter limitations in terms of low detection accuracy and insufficient detection granularity. This study introduces an innovative attentive semantic feature fusion transformer network (ASFormer) tailored for pixel-level defect detection tasks to overcome these challenges. Initially, the transformer architecture is employed to extract the multiscale features of defects. Subsequently, a multiscale semantic fusion module is integrated, complemented by a dynamic upsampling mechanism, to mitigate the loss of detailed information during feature fusion. Moreover, a scale-aware dual-attention module is developed, which effectively captures the intricacies of the fused features across both channel and spatial dimensions, ensuring pixel-level detection precision. Additionally, a contextual boundary loss function is proposed to augment the ability of the network to discern defect boundaries, different categories and defect scales. Experimental validation on the NEU-Seg and Crack500 surface defect datasets demonstrated that the proposed ASFormer achieved state-of-the-art performance, with mIoU scores of 85.24% and 78.20%, respectively.

Local Pyramid Vision Transformer: Millimeter-Wave Radar Gesture Recognition Based on Transformer with Integrated Local and Global Awareness

A millimeter-wave radar is widely accepted by the public due to its low susceptibility to interference, such as changes in light, and the protection of personal privacy. With the development of the deep learning theory, the deep learning method has been dominant in the millimeter-wave radar field, which usually uses convolutional neural networks for feature extraction. In recent years, transformer networks have also been highly valued by researchers due to their parallel processing capabilities and long-distance dependency modeling capabilities. However, traditional convolutional neural networks (CNNs) and vision transformers each have their limitations: CNNs usually overlook the global features of images and vision transformers may neglect local image continuity, and both of them may impede gesture recognition performance. In addition, whether CNN or transformer, their implementation is hindered by the scarcity of public radar gesture datasets. To address these limitations, this paper proposes a new recognition method using a local pyramid visual transformer (LPVT) based on millimeter-wave radar. LPVT can capture global and local features in dynamic gesture spectrograms, ultimately improving the recognition ability of gestures. In this paper, we mainly carried out the following two tasks: building the corresponding datasets and executing gesture recognition. First, we constructed a gesture dataset for training. In this stage, we use a 77 GHz radar to collect the echo signals of gestures and preprocess them to build a dataset. Second, we propose the LPVT network specifically designed for gesture recognition tasks. By integrating local sensing into the globally focused transformer, we improve its capacity to capture both global and local features in dynamic gesture spectrograms. The experimental results using the dataset we constructed show that the proposed LPVT network achieved a gesture recognition accuracy of 92.2%, which exceeds the performance of other networks.

HybridGT: An Integration of Graph Transformer and LSTM for Effective Hyperspectral Band Selection

ABSTRACT Hyperspectral imagery has a high-dimensional curse due to numerous spectral bands. Band selection (BS) is crucial for efficiently reducing dimensionality, retaining only essential bands containing valuable information. However, deep learning-based techniques have gained more attention through trained networks for band selection. Recently, graph-based learning has been extensively used in hyperspectral imagery, revealing intrinsic data relationships. This article presents a novel hybrid approach for hyperspectral band selection, addressing the curse of dimensionality in hyperspectral imagery (HSI). Integrating Long Short Term Memory (LSTM) and Graph Transformer (GT), the method employs Bi-dimensional Empirical Mode Decomposition (BEMD) for spatial data enhancement. Using transfer learning, we explore a ResNet-50 deep network to identify optimal intrinsic mode functions (IMFs). The final band subset will be obtained by concatenating the features extracted from the graph transformer and LSTM networks from selected IMFs and residual IMF, respectively. The proposed HybridGT-BS technique surpasses state-of-the-art methods in classification accuracy across three well-known HSI datasets – IP-Indian Pines, SA-Salinas, and PU-PaviaU. With the support of experimental results, the proposed technique significantly outperforms the classification accuracy with the best bands of the HSIs.

Dual scale light weight cross attention transformer for skin lesion classification.

Skin cancer is rapidly growing globally. In the past decade, an automated diagnosis system has been developed using image processing and machine learning. The machine learning methods require hand-crafted features, which may affect performance. Recently, a convolution neural network (CNN) was applied to dermoscopic images to diagnose skin cancer. The CNN improved its performance through its high-dimension feature extraction capability. However, these methods lack global co-relation of the spatial features. In this study, we design a dual-scale lightweight cross-attention vision transformer network (DSCATNet) that provides global attention to high-dimensional spatial features. In the DSCATNet, we extracted features from different patch sizes and performed cross-attention. The attention from different scales improved the spatial features by focusing on the different parts of the skin lesion. Furthermore, we applied a fusion strategy for the different scale spatial features. After that, enhanced features are fed to the lightweight transformer encoder for global attention. We validated the model superiority on the HAM 10000 and PAD datasets. Furthermore, the model's performance is compared with CNN and ViT-based methods. Our DSCATNet achieved an average kappa and accuracy of 95.84% and 97.80% on the HAM 10000 dataset, respectively. Moreover,the model obtained 94.56% and 95.81% kappa and precision values on the PAD dataset.

Multi-channel fused vision transformer network for bearing fault diagnosis under different working conditions

Abstract Many of the current fault diagnosis methods rely on time-domain signals. While the richest information are contained in these signals, their complexity poses challenges to network learning and limits the ability to fully characterize them. To address these issues, a novel multi-channel fused vision transformer network (MFVTN) is proposed in this paper. Firstly, the overlapping patch embedding module is introduced to overlap the time-domain map with edge information, preserving the global continuous features of the time-domain map and adding positional encoding for sorting. This integration helps the vision transformer merge detailed features and construct the global mapping. Secondly, multiple dimensional time domain signal features are extracted and fused in parallel, enabling multi-domain fault diagnosis of bearings. In order to enhance the network ability to extract domain-invariant features, an adversarial training strategy combined with Wasserstein distance is utilized. The results demonstrate that the diagnostic accuracy of the proposed MFVTN can reach 98.2%.

An Improved Generative Adversarial Network-Based and U-Shaped Transformer Method for Glass Curtain Crack Deblurring Using UAVs

Drones have emerged as a critical tool for the detection of high-altitude glass curtain cracks. However, their utility is often compromised by vibrations and other environmental factors that can induce motion blur, compromising image quality and the accuracy of crack detection. This paper presents a novel GAN-based and enhanced U-shaped Transformer network, named GlassCurtainCrackDeblurNet, designed specifically for the deblurring of drone-captured images of glass curtain cracks. To optimize the performance of our proposed method for this application, we have meticulously created the GlassCurtainCrackDeblur Dataset. Our method demonstrates superior qualitative and quantitative outcomes when compared to other established deblurring techniques on both the GoPro Dataset and the GlassCurtainCrackDeblur Dataset.

GACT-PPIS: Prediction of protein-protein interaction sites based on graph structure and transformer network

The prediction of protein-protein interaction sites (PPIS) is currently crucial for regulating many biological activities in cells and developing drugs for various diseases. Deep learning-based methods have been proposed for predicting PPIS, significantly reducing the manpower and time costs associated with traditional experimental methods such as yeast two-hybrid, mass spectrometry, and affinity purification. However, the predictive accuracy of these deep learning methods has not yet reached the expected level. Therefore, we introduce a model called GACT-PPIS.The design of the GACT-PPIS algorithm aims to utilize combined information from protein sequences and structures as input to predict protein-protein interaction sites. The core of GACT-PPIS utilizes an Enhanced Graph Attention Network (EGAT) with initial residual and identity mappings, along with a deep Transformer network as the basic units, supplemented by Graph Convolutional Networks (GCN), effectively aggregating information from neighboring nodes for each node. After multiple network layers, the information of the entire protein is also fused into the nodes, and the Transformer network further enhances the model's performance.Experimental results show that GACT-PPIS outperforms the most representative models in terms of Recall, F1-measure, MCC, AUROC, and AUPRC on the benchmark test set (Test-60). Additionally, on other independent test sets (UBTest-31-6), GACT-PPIS leads in terms of Accuracy, Precision, Recall, F1-measure, MCC, AUROC, and AUPRC compared to the most representative models. It is worth noting that GACT-PPIS demonstrates excellent generalization and versatility across different test sets, showcasing good performance on multiple test sets for the same trained GACT-PPIS model.

Time-segment-wise feature fusion transformer for multi-modal fault diagnosis

Mechanical fault diagnosis is crucial to ensure the safe operations of equipment in intelligent manufacturing systems. Recently, deep learning based fault diagnosis methods have achieved remarkable advancements with monitored data from a single sensor. However, obtaining satisfactory diagnostic results based on a single sensor is often difficult because the complementary information between different sensors is ignored. Extracting comprehensive fault features from multi-modal data is a problem that remains to be solved. To address these challenges, a time-segment-wise feature fusion Transformer (FFTR) is proposed in this paper. First, the signals from various modalities as multiple channels are normalized channel-by-channel and form a multi-modal sample. Second, a time-segment-wise feature learning network is designed to transform a multi-modal sample into several fusion features through the sequential processes of sample segmentation, segment-level feature extraction and time-aligned feature fusion. Finally, a Transformer network is employed for comprehensive multi-modal feature analysis and fault classification. In addition, a joint loss function is designed to comprehensively train the end-to-end FFTR. The comparison experiment with other baseline methods is conducted on two multi-modal datasets. The experimental results show that FFTR achieves 3.69% and 3.93% higher diagnostic accuracy than baseline on two datasets respectively and can address real-world problems effectively.

Neural Networks for Conversion of Simulated NMR Spectra from Low-Field to High-Field for Quantitative Metabolomics

Background: The introduction of benchtop NMR instruments has made NMR spectroscopy a more accessible, affordable option for research and industry, but the lower spectral resolution and SNR of a signal acquired on low magnetic field spectrometers may complicate the quantitative analysis of spectra. Methods: In this work, we compare the performance of multiple neural network architectures in the task of converting simulated 100 MHz NMR spectra to 400 MHz with the goal of improving the quality of the low-field spectra for analyte quantification. Multi-layered perceptron networks are also used to directly quantify metabolites in simulated 100 and 400 MHz spectra for comparison. Results: The transformer network was the only architecture in this study capable of reliably converting the low-field NMR spectra to high-field spectra in mixtures of 21 and 87 metabolites. Multi-layered perceptron-based metabolite quantification was slightly more accurate when directly processing the low-field spectra compared to high-field converted spectra, which, at least for the current study, precludes the need for low-to-high-field spectral conversion; however, this comparison of low and high-field quantification necessitates further research, comparison, and experimental validation. Conclusions: The transformer method of NMR data processing was effective in converting low-field simulated spectra to high-field for metabolomic applications and could be further explored to automate processing in other areas of NMR spectroscopy.

Classification of Motor Imagery Tasks Using EEG Based on Wavelet Scattering Transform and Convolutional Neural Network

Classification of Motor Imagery Tasks Using EEG Based on Wavelet Scattering Transform and Convolutional Neural Network

DTTCNet: Time-to-Collision Estimation With Autonomous Emergency Braking Using Multi-Scale Transformer Network

DTTCNet: Time-to-Collision Estimation With Autonomous Emergency Braking Using Multi-Scale Transformer Network

Driver Multi-task Emotion Recognition Network Based on Multi-modal Facial Video Analysis

Driver emotion recognition is crucial for enhancing the safety and user experience in driving scenarios. However, current emotion recognition methods often rely solely on a single modality and a single-task setup, leading to suboptimal performance in driving scenarios. To address this, this paper proposes a driver multitask emotion recognition method based on multimodal facial video analysis (MER-MFVA). This method extracts facial expression features and remote photoplethysmography (rPPG) signals from driver facial videos. Facial expression features include facial action units and eye movement information, representing the driver's external characteristics. rPPG information, representing the driver's internal characteristics, is enhanced through a designed dual-path Transformer network and an introduced focus module. We also propose a cross-modal mutual attention computation mechanism to effectively fuse multimodal features by calculating mutual attention between facial expression features and rPPG information. In the final task output, we employ a multitask learning mechanism, setting discrete emotion recognition as the primary task and emotion valence recognition, emotion arousal recognition, and the previous rPPG information extraction as auxiliary tasks to facilitate effective information sharing across different tasks. Experimental results on the established driver emotion dataset demonstrate that our proposed method significantly improves driver emotion recognition performance, achieving an accuracy of 86.98% and an F1 score of 85.83% in the primary task. This validates the effectiveness of the proposed approach.

Selective Auditory Attention Detection Using Combined Transformer and Convolutional Graph Neural Networks

Attention is one of many human cognitive functions that are essential in everyday life. Given our limited processing capacity, attention helps us focus only on what matters. Focusing attention on one speaker in an environment with many speakers is a critical ability of the human auditory system. This paper proposes a new end-to-end method based on the combined transformer and graph convolutional neural network (TraGCNN) that can effectively detect auditory attention from electroencephalograms (EEGs). This approach eliminates the need for manual feature extraction, which is often time-consuming and subjective. Here, the first EEG signals are converted to graphs. We then extract attention information from these graphs using spatial and temporal approaches. Finally, our models are trained with these data. Our model can detect auditory attention in both the spatial and temporal domains. Here, the EEG input is first processed by transformer layers to obtain a sequential representation of EEG based on attention onsets. Then, a family of graph convolutional layers is used to find the most active electrodes using the spatial position of electrodes. Finally, the corresponding EEG features of active electrodes are fed into the graph attention layers to detect auditory attention. The Fuglsang 2020 dataset is used in the experiments to train and test the proposed and baseline systems. The new TraGCNN approach, as compared with state-of-the-art attention classification methods from the literature, yields the highest performance in terms of accuracy (80.12%) as a classification metric. Additionally, the proposed model results in higher performance than our previously graph-based model for different lengths of EEG segments. The new TraGCNN approach is advantageous because attenuation detection is achieved from EEG signals of subjects without requiring speech stimuli, as is the case with conventional auditory attention detection methods. Furthermore, examining the proposed model for different lengths of EEG segments shows that the model is faster than our previous graph-based detection method in terms of computational complexity. The findings of this study have important implications for the understanding and assessment of auditory attention, which is crucial for many applications, such as brain–computer interface (BCI) systems, speech separation, and neuro-steered hearing aid development.

An Intelligent Maneuver Decision-Making Approach for Air Combat Based on Deep Reinforcement Learning and Transformer Networks

The traditional maneuver decision-making approaches are highly dependent on accurate and complete situation information, and their decision-making quality becomes poor when opponent information is occasionally missing in complex electromagnetic environments. In order to solve this problem, an autonomous maneuver decision-making approach is developed based on deep reinforcement learning (DRL) architecture. Meanwhile, a Transformer network is integrated into the actor and critic networks, which can find the potential dependency relationships among the time series trajectory data. By using these relationships, the information loss is partially compensated, which leads to maneuvering decisions being more accurate. The issues of limited experience samples, low sampling efficiency, and poor stability in the agent training state appear when the Transformer network is introduced into DRL. To address these issues, the measures of designing an effective decision-making reward, a prioritized sampling method, and a dynamic learning rate adjustment mechanism are proposed. Numerous simulation results show that the proposed approach outperforms the traditional DRL algorithms, with a higher win rate in the case of opponent information loss.

Revolutionizing Wildfire Detection Through UAV-Driven Fire Monitoring with a Transformer-Based Approach

The rapid detection and accurate localization of wildfires are critical for effective disaster management and response. This study proposes an innovative Unmanned aerial vehicles (UAVs)-based fire detection system leveraging a modified Miti-DETR model tailored to meet the computational constraints of drones. The enhanced architecture incorporates a redesigned AlexNet backbone with residual depthwise separable convolution blocks, significantly reducing computational load while improving feature extraction and accuracy. Furthermore, a novel residual self-attention mechanism addresses convergence issues in transformer networks, ensuring robust feature representation for complex aerial imagery. The model, which was trained on the FLAME dataset encompassing diverse fire scenarios, demonstrates superior performance in terms of Mean Average Precision (mAP) and Intersection over Union (IoU) metrics compared to existing systems. Its capability to detect and localize fires across varied backgrounds highlights its practical application in real-world scenarios. This advancement represents a pivotal step forward in applying deep learning for real-time wildfire detection, with implications for broader emergency management applications.

Solitonic solutions for the reduced Maxwell-Bloch equations via the Darboux transformation and artificial neural network in nonlinear wave dynamics

Abstract In this article, we explore nonlinear wave dynamics by presenting solitonic solutions for the reduced Maxwell-Bloch equations, a model relevant to both nonlinear optics and Bose-Einstein condensates. First, we derive the Lax pair for the system and apply a Darboux transformation to obtain and analyze the soliton solutions. Then, we introduce a novel approach by integrating the Darboux transformation, a powerful analytical tool, with the Levenberg-Marquardt artificial neural network, a reliable numerical method. This combination enables the identification and validation of soliton solutions, supported by detailed graphical and tabular analyses. The Levenberg-Marquardt artificial neural network effectively demonstrates the uniqueness and convergence of the solutions, with the accuracy of the results validated through comprehensive graphical representations.

Lane extraction from trajectories at road intersections based on Graph Transformer Network

Lane-level road networks are crucial components of high-precision maps and play a significant role in intelligent transportation systems. Extracting lane-level road networks at intersections presents considerable challenges due to the complex structures of intersections and diverse driving behaviors. A graph-learning based method is proposed for extracting lanes from high-precision trajectories at road intersections. A trajectory relation graph is designed to encode the directional, shape, and distance features of trajectories, capturing both the intrinsic and extrinsic relationships between trajectories. Subsequently, a Graph Transformer Network is developed to extract a representative subset of trajectories as lanes. To alleviate the problem of generating missing and extraneous lanes, a set-based lane extraction loss is introduced to achieve implicit pruning of redundancy through the attention mechanism. Comprehensive experimental results demonstrate that the proposed method outperforms state-of-the-art methods in three positional and topological accuracy metrics. The method achieves lane extraction with minimal omissions and redundancies and exhibits strong performance in complex scenarios such as U-turns, lane merging, and lane diverging regions.

Second-order transformer network for video recognition

The video recognition community is undergoing a significant change in backbone shifting from CNNs to transformers. However, due to the temporal information existing in the video, vision transformers, which have been shown to be effective in image tasks, cannot model spatio-temporal structure in video recognition. In addition, current transformer-based video models exclusively utilize either class tokens or visual tokens for classification, failing to combine the merits of these two types of tokens. To solve the above problems, we propose a novel video second-order transformer network (ViSoT). As there are complex motion and appearance information in video, visual tokens of the last ViSoT layer are aggregated by cross-covariance pooling to model spatio-temporal information, which is combined with class tokens for classification. Meanwhile, a fast singular value power normalization is further introduced to achieve effective aggregation of visual tokens. For temporal modeling, before ViSoT block temporal convolutions are performed continuously in the early stage after the convolution stem via a short-cut connection. In ViSoT, token shift module and space–time attention are proposed for modeling temporal relations and spatio-temporal interaction across adjacent frames respectively. Validating ViSoT on four video benchmarks demonstrates its effectiveness and efficiency, comparable to or better than state-of-the-art methods with fewer parameters and GFLOPs.