Convolution Transformation Research Articles

U-shaped fusion convolutional transformer based workflow for fast optical coherence tomography angiography generation in lips.

Oral disorders, including oral cancer, pose substantial diagnostic challenges due to late-stage diagnosis, invasive biopsy procedures, and the limitations of existing non-invasive imaging techniques. Optical coherence tomography angiography (OCTA) shows potential in delivering non-invasive, real-time, high-resolution vasculature images. However, the quality of OCTA images are often compromised due to motion artifacts and noise, necessitating more robust and reliable image reconstruction approaches. To address these issues, we propose a novel model, a U-shaped fusion convolutional transformer (UFCT), for the reconstruction of high-quality, low-noise OCTA images from two-repeated OCT scans. UFCT integrates the strengths of convolutional neural networks (CNNs) and transformers, proficiently capturing both local and global image features. According to the qualitative and quantitative analysis in normal and pathological conditions, the performance of the proposed pipeline outperforms that of the traditional OCTA generation methods when only two repeated B-scans are performed. We further provide a comparative study with various CNN and transformer models and conduct ablation studies to validate the effectiveness of our proposed strategies. Based on the results, the UFCT model holds the potential to significantly enhance clinical workflow in oral medicine by facilitating early detection, reducing the need for invasive procedures, and improving overall patient outcomes.

Two Birds With One Stone: Knowledge-Embedded Temporal Convolutional Transformer for Depression Detection and Emotion Recognition

Depression is a critical problem in modern society that affects an estimated 350 million people worldwide, causing feelings of sadness and a lack of interest and pleasure. Emotional disorders are gaining interest and are closely entwined with depression, because one contributes to an understanding of the other. Despite the achievements in the two separate tasks of emotion recognition and depression detection, there has not been much prior effort to build a unified model that can connect these two tasks with different modalities, including multimedia (text, audio, and video) and unobtrusive physiological signals (e.g., electroencephalography). We propose a novel <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">temporal convolutional transformer with knowledge embedding</b> to address the joint task of depression detection and emotion recognition. This approach not only learns multimodal embeddings across domains via the temporal convolutional transformer but also exploits special-domain knowledge from medical knowledge graphs to improve the performance of detection and recognition. It is essential that the features learned by our method can be perceived as a priori and are suitable for increasing the performance of other related tasks. Our method illustrates the case of “two birds with one stone” in the sense that two or more tasks can be efficiently handled with our unique model, which captures effective features. Experimental results on ten real-world datasets show that the proposed approach significantly outperforms other state-of-the-art approaches. On the other hand, experiments in which our methodology is applied to other reasoning tasks show that our approach effectively supports model reasoning related to emotion and improves its performance.

Monte Carlo DropBlock for modeling uncertainty in object detection

With the advancements made in deep learning, computer vision problems have seen a great improvement in performance. However, in many real-world applications such as autonomous driving vehicles, the risk associated with incorrect predictions of objects or segmentation of images is very high. Standard deep learning models for object detection and segmentation such as YOLO models are often overconfident in their predictions and do not take into account the uncertainty in predictions on out-of-distribution data. In this work, we propose an efficient and effective approach, Monte-Carlo DropBlock (MC-DropBlock), to model uncertainty in YOLO and convolutional vision Transformers for object detection. The proposed approach applies drop-block during training time and testing time on the convolutional layer of the deep learning models such as YOLO and convolutional transformer. We theoretically show that this leads to a Bayesian convolutional neural network capable of capturing the epistemic uncertainty in the model. Additionally, we capture the aleatoric uncertainty in the data using a Gaussian likelihood. We demonstrate the effectiveness of the proposed approach on modeling uncertainty in object detection and segmentation tasks using out-of-distribution experiments. Experimental results show that MC-DropBlock improves the generalization, calibration, and uncertainty modeling capabilities of YOLO models and convolutional Transformer models for object detection and segmentation.

Transformer-based network with temporal depthwise convolutions for sEMG recognition

Considerable progress has been made in pattern recognition of surface electromyography (sEMG) with deep learning, bringing improvements to sEMG-based gesture classification. Current deep learning techniques are mainly based on convolutional neural networks (CNNs), recurrent neural networks (RNNs), and their hybrids. However, CNNs focus on spatial and local information, while RNNs are unparallelizable, and they suffer from gradient vanishing/exploding. Their hybrids often face problems of model complexity and high computational cost. Because sEMG signals have a sequential nature, motivated by the sequence modeling network Transformer and its self-attention mechanism, we propose a Transformer-based network, temporal depthwise convolutional Transformer (TDCT), for sparse sEMG recognition. With this network, higher recognition accuracy is achieved with fewer convolution parameters and a lower computational cost. Specifically, this network has parallel capability and can capture long-range features inside sEMG signals. We improve the locality and channel correlation capture of multi-head self-attention (MSA) for sEMG modeling by replacing the linear transformation with the proposed temporal depthwise convolution (TDC), which can reduce the convolution parameters and computations for feature learning performance. Four sEMG datasets, Ninapro DB1, DB2, DB5, and OYDB, are used for evaluations and comparisons. In the results, our model outperforms other methods, including Transformer-based networks, in most windows at recognizing the raw signals of sparse sEMG, thus achieving state-of-the-art classification accuracy.

A token selection-based multi-scale dual-branch CNN-transformer network for 12-lead ECG signal classification

The timely identification of cardiovascular diseases is critical for effective intervention, with the electrocardiogram (ECG) serving as a pivotal diagnostic tool. Recent advancements in deep learning-based methods have significantly enhanced the accuracy of ECG signal classification. In clinical settings, cardiologists rely on diagnoses derived from standardized 12-lead ECG recordings. It must be acknowledged that there is considerable redundancy in the 12-lead ECG recordings used for ECG signal classification, thereby hindering their generalization capabilities. Meanwhile, considering multi-scale features in 12-lead ECG recordings is a crucial aspect that is often overlooked by existing methods. Based on the above observations, we develop a multi-scale Convolutional Transformer network for ECG signal classification. By utilizing learnable Convolutional neural network (CNN) blocks and novel dual-branch Transformer encoders, the proposed network automatically extracts features at different scales, resulting in superior feature representation. Additionally, by discarding low-importance patches and focusing on high-importance patches, we effectively alleviate information redundancy in the 12-lead ECG recordings. We conduct comprehensive experiments on three commonly used ECG datasets. The Research results show that our proposed network outperforms existing state-of-the-art networks in multiple tasks.

Capsule Neural Network Guided by Compact Convolutional Transformer for Discriminating Earthquakes from Quarry Blasts

Abstract Misclassified nonearthquake seismic events like quarry blasts can contaminate the earthquake catalog. The local earthquakes sometimes have similar features as the quarry blasts, which makes manual discrimination difficult and unreliable. Thus, we propose to use the compact convolutional transformer (CCT) and capsule neural network to discriminate between earthquakes and quarry blasts. First, we extract 60 s three-channel seismograms, that is, 10 and 50 s before and after the P-wave arrival time. Then, we transform the time-series data into a time–frequency domain (scalogram) using the continuous wavelet transform. Afterward, we utilize the CCT network to extract the most significant features from the input scalograms. The capsule neural network is utilized to extract the spatial relation between the extracted features using the routing-by-agreement approach (dynamic routing). The capsule neural network extracts different digit vectors for the earthquake and the quarry blast classes, allowing a robust classification accuracy. The proposed algorithm is evaluated using the seismic dataset recorded by the Egyptian Seismic Network. The dataset is divided into 80% for training and 20% for testing. Although the dataset is unbalanced, the proposed algorithm shows promising results. The testing accuracy of the proposed algorithm is 97.31%. The precision, recall, and F1-score are 97.23%, 98.83%, and 98.02%, respectively. In addition, the proposed algorithm outperforms the traditional deep learning models, for example, convolutional neural network, ResNet, Visual Geometry Group (VGG), and AlexNet networks. Finally, the proposed method is demonstrated to enjoy a high-generalization ability through a real-time monitoring experiment.

CTS-Unet : Urban change detection by convolutional Siamese concatenate network with Swin transformer

Due to the fast progress of Deep Learning (DL) methods in remote sensing applications, many change detection (CD) algorithms have been recently proposed based on CNN networks and the mechanism of self-attention. These algorithms extract features without focusing on the temporal dependency between features. This shortcoming led to the introduction of the Transformer mechanism. In this paper, we design a convolutional transformer network with a Siamese U-shaped structure and name it CTS-Unet to solve the CD problem. We exploit the ability of the CNN to extract effective semantic features and that of the transformer to extract global information effectively. The Siamese architecture allows using CNN to simultaneously extract effective semantic features from low-resolution bi-temporal images. The transformer part contains an encoder and a decoder, all of which use the Swin transformer module as their basic unit. The encoder processes the features extracted from the CNN using patch merging and the Swin transformer module to produce semantic features. The encoder extracts the detailed information from the features using patch expansion, the Swin transformer module, and convolutional upsampling to create a CD map. The experiments were performed on the widely used LEVIR-CD and DSIFN-CD datasets. Compared with other state-of-the-art CD methods, CTS-Unet provides higher performance with F1-scores of 91.87% and 69.60% for LEVIR-CD and DSIFN-CD datasets, respectively.

A Transformer-Based Contrastive Semi-Supervised Learning Framework for Automatic Modulation Recognition

The application of deep learning improves the processing speed and the accuracy of automatic modulation recognition (AMR). As a result, it realizes intelligent spectrum management and electronic reconnaissance. However, deep learning-aided AMR usually requires a large number of labeled samples to obtain a reliable neural network model. In practical applications, due to economic costs and privacy constraints, there is a small number of labeled samples but a large number of unlabeled samples. This paper proposes a Transformer-based contrastive semi-supervised learning framework for AMR. First, self-supervised contrastive pre-training of the Transformer-based encoder is completed using unlabeled samples, and data augmentation is realized through time warping. Then, the pre-trained encoder and a randomly initialized classifier are fine-tuned using labeled samples, and hierarchical learning rates are employed to ensure classification accuracy. Considering the problems of applying Transformer to AMR, a convolutional transformer deep neural network is proposed, which involves convolutional embedding, attention bias, and attention pooling. In experiments, the feasibility of the framework is analyzed through linear evaluation of the framework components on the RML2016.10a dataset. Also, the proposed framework is compared with existing semi-supervised methods on RML2016.10a and RML2016.10b datasets to verify its superiority and stability.

A systematic literature review of emotion recognition using EEG signals

In this study, we conducted a systematic literature review of 107 primary studies conducted between 2017 and 2023 to discern trends in datasets, classifiers, and contributions to human emotion recognition using EEG signals. We identified DEAP (43%), SEED (29%), DREAMER (8%), and SEED-IV (5%) as the most commonly used EEG signal datasets. Deep learning techniques, especially transformer neural architecture search (TNAS), global-to-local feature aggregation network (GLFANet), attention-based convolutional transformer neural network (ACTNN) and efficient CNN-contrastive learning (ECNN-C), demonstrate superior performance across different datasets. Our comparative analysis of the validation scenarios revealed that subject-dependent validations outperformed subject-independent validations, highlighting the challenge of individual differences. This finding underscores the need for future research to address these individual differences in emotion recognition and enhance the generalizability of the models. The research contributions spanned seven categories: data augmentation, domain adaptation, channel selection, preprocessing, feature selection, classifiers, and feature extraction, with a major emphasis on feature extraction and classification (34%). This trend suggests a growing recognition of the importance of these contributions in the development of effective emotion-recognition systems. Our study contributes to the field by providing a comprehensive overview of recent trends, illuminating the performance of various classifiers, and highlighting key areas for future research on EEG-based emotion recognition. This work has significant implications for various applications, including mental health, human–computer interaction, and affective brain–computer interfaces.

Parkinsonian Tremor Detection with Compact Convolutional Transformer from Bispectrum Representation of tri-Axial Accelerometer Signals.

After the breakthroughs of Transformer networks in Natural Language Processing (NLP) tasks, they have led to exciting progress in visual tasks as well. Nonetheless, there has been a parallel growth in the number of parameters and the amount of training data, which led to the conclusion that Transformers are not suited for small datasets. This paper is the first to convey the feasibility of Compact Convolutional Transformers (CCT) for the prediction of Parkinsonian postural tremor based on the Bispectrum (BS) representation of IMU accelerometer time series. The dataset includes tri-axial accelerometer signals collected unobtrusively in-the-wild while subjects are on a phone call, and labelled by neurologists and signal processing experts. The BS is a noise-immune, higher-order representation that reflects a signal's deviation from Gaussianity and measures quadratic phase coupling. We performed comparative classification experiments using the CCT, pre-trained CNNs such as VGG-16 and ResNet-50, and the conventional Vision Transformer (ViT). Our model achieves competitive prediction accuracy and F1 score of 96% with only 1.016 M trainable parameters, compared to the ViT with 21.659 M trainable parameters, in a five-fold cross-validation scheme. Our model also outperforms pre-trained CNNs such as VGG-16 and ResNet-50. Furthermore, we show that the performance gains are maintained when training on a larger dataset of BS images. Our effort here is motivated by the hypothesis that data-efficient transformers outperform transfer learning using pre-trained CNNs, paving the way for promising deep learning architecture for small-scale, novel and noisy medical imaging datasets.Clinical relevance- Novel deep learning model for unobtrusive prediction of Parkinsonian Postural Tremor from Bispectrum image representation of tri-axial accelerometer signals collected in-the-wild.

DWT-CNNTRN: a Convolutional Transformer for ECG Classification with Discrete Wavelet Transform.

Cardiovascular diseases are the leading cause of death worldwide. The diagnoses of cardiovascular diseases are usually carried out by cardiologists utilizing Electrocardiograms (ECGs). To assist these physicians in making an accurate diagnosis, there is a growing need for reliable and automatic ECG classifiers.In this study, a new method is proposed to classify 12-lead ECG recordings. The proposed model is composed of four components: the CNN(Convolutional Neural Network) module, the transformer module, the global hybrid pooling layer, and a classification layer. To improve the classification performance, the model takes the discrete wavelet transform of ECG signals as the model inputs and utilizes a hybrid pooling layer to condense the most important features over each period.The proposed model is evaluated using the test set of the China Physiological Signal Challenge 2018 dataset with 12-lead ECGs. It performs with an average accuracy of 0.86 and an average F1-scores of 0.83. The scores are particularly good for the block conditions (LBBB, RBBB, I-AVB). The main advantage of the proposed model is that, it obtains good results with a significantly smaller number of parameters compared to other individual and ensemble models.Clinical relevance- This work establishes a new ECG classifier model with high performance and low model size. It can make automatic ECG analysis more accessible, efficient, and accurate, especially in remote or underserved areas.

Progressive convolutional transformer for image restoration

In the past few years, convolutional neural networks (CNNs) have become the primary workhorse for image restoration tasks. However, the deficiency of modeling long-range dependencies due to the local computational property of convolution greatly limits the restoration performance. To overcome this limitation, we propose a novel multi-stage progressive convolutional Transformer to recursively restore the degraded images, termed PCformer, which enjoys a high capability for capturing local context and global dependencies with friendly computational cost. Specifically, each stage of PCformer is an asymmetric encoder–decoder network whose bottleneck is built upon a tailored Transformer block with convolution operation deployed to avoid any loss of local context. Both encoder and decoder are convolution-based modules, thus allowing to explore rich contextualized information for image recovery. Taking the low-resolution features encoded by the encoder as tokens input into the Transformer bottleneck guarantees that long-range pixel interactions are captured while reducing the computational burden. Meanwhile, we apply a gated module for filtering redundant information propagation between every two phases. In addition, long-range enhanced inpainting is further introduced to mining the ability of PCformer to exploit distant complementary features. Extensive experiments yield superior results and in particular establishing new state-of-the-art results on several image restoration tasks, including deraining (+0.37 dB on Rain13K), denoising (+0.11 dB on DND), dehazing (+0.56 dB on I-HAZE), enhancement (+0.72 dB on SICE), and shadow removal (+0.65 RMSE on ISTD). The implementation code is available at https://github.com/Jeasco/PCformer.

ConTriFormer: triggers-guided contextual informer for remaining useful life prediction of rolling bearings

Rolling bearings are critical components in many industrial fields, and their stability directly affects the performance and safety of the industrial equipment. Accurate prediction of remaining useful life (RUL) of rolling bearings is a heated topic in modern research. Traditional strategies are unable to efficiently exploit the significant features of the data, resulting in the inability to determine the starting time of prediction along with a reduced prediction accuracy. Accordingly, this paper proposes a novel data-driven prediction model named ConTriFormer, which incorporates multi-feature triggers focusing on various scales of input signals, and the ConvNeXt V2 sparse convolution strategy within the contextual Informer architecture for estimating RUL. Firstly, significant feature indicators of the original data are calculated to construct feature triggers, resulting in a multi-feature fusion. Secondly, the starting time for prediction is obtained through quantified results from fault-sensitive triggers. Thirdly, the original signal with triggers embedded is encoded and organized into sparse matrices to facilitate the simplification of subsequent computations. Sparse features and dynamic context information reflecting bearing state changes are obtained through ConvNeXt V2 sparse convolution, which is input into the Informer structure with contextual attentive structures inside for better adaptability to long time-span dynamic data and lower spatiotemporal complexity for feature mining and prediction. Finally, the prediction results are obtained by mapping output values to the remaining life through a fully connected layer. The proposed algorithm is compared with mainstream deep learning algorithms such as Bi-LSTM and Convolutional Transformer using the XJTU-SY dataset and PHM 2012 dataset, and the effectiveness of model is verified with ablation study. Results show that, the proposed method can more accurately predict RUL, providing a high-precision and intelligent method for prognostics health management of rolling bearings.

Convolution-Enhanced Evolving Attention Networks.

Attention-based neural networks, such as Transformers, have become ubiquitous in numerous applications, including computer vision, natural language processing, and time-series analysis. In all kinds of attention networks, the attention maps are crucial as they encode semantic dependencies between input tokens. However, most existing attention networks perform modeling or reasoning based on representations, wherein the attention maps of different layers are learned separately without explicit interactions. In this paper, we propose a novel and generic evolving attention mechanism, which directly models the evolution of inter-token relationships through a chain of residual convolutional modules. The major motivations are twofold. On the one hand, the attention maps in different layers share transferable knowledge, thus adding a residual connection can facilitate the information flow of inter-token relationships across layers. On the other hand, there is naturally an evolutionary trend among attention maps at different abstraction levels, so it is beneficial to exploit a dedicated convolution-based module to capture this process. Equipped with the proposed mechanism, the convolution-enhanced evolving attention networks achieve superior performance in various applications, including time-series representation, natural language understanding, machine translation, and image classification. Especially on time-series representation tasks, Evolving Attention-enhanced Dilated Convolutional (EA-DC-) Transformer outperforms state-of-the-art models significantly, achieving an average of 17% improvement compared to the best SOTA. To the best of our knowledge, this is the first work that explicitly models the layer-wise evolution of attention maps. Our implementation is available at https://github.com/pkuyym/EvolvingAttention.

An Aggregated Convolutional Transformer Based on Slices and Channels for Multivariate Time Series Classification

Convolutional neural network has achieved remarkable success, and has excellent local feature extraction ability. Similarly, Transformer has been developed markedly in recent years, achieving excellent representation capabilities in terms of global features, which has aroused heated discussions. In terms of multivariate time series classification, most previous networks had convolution and long and short-term memory structures. This paper innovatively proposes a combination of Transformer-encoder and convolutional structures, which we refer to as the Multivariate time series classification Convolutional Transformer Network (MCTNet). The different advantages of convolutional neural network and self-attention are used to capture potential deep information in multivariate time series more accurately. The Transformer is considered to be data-hungry, and combined with the induction bias of the convolutional neural network to solve this problem, early features are extracted through the convolutional layers, and the both squeeze and excitation convolution encoder (BC-Encoder) structure is proposed. Attentional prototype learning is also used to mitigate the limited label problem. Moreover, a new network design that focuses on slices and channels is proposed, moving beyond the concept that the use of Transformer will require many parameters. Experimental results from 26 datasets of the well-known multivariate time series archive UEA show that the performance of our model is better than that of most state-of-the-art models.

Deep Convolutional Pooling Transformer for Deepfake Detection

Recently, Deepfake has drawn considerable public attention due to security and privacy concerns in social media digital forensics. As the wildly spreading Deepfake videos on the Internet become more realistic, traditional detection techniques have failed in distinguishing between real and fake. Most existing deep learning methods mainly focus on local features and relations within the face image using convolutional neural networks as a backbone. However, local features and relations are insufficient for model training to learn enough general information for Deepfake detection. Therefore, the existing Deepfake detection methods have reached a bottleneck to further improve the detection performance. To address this issue, we propose a deep convolutional Transformer to incorporate the decisive image features both locally and globally. Specifically, we apply convolutional pooling and re-attention to enrich the extracted features and enhance efficacy. Moreover, we employ the barely discussed image keyframes in model training for performance improvement and visualize the feature quantity gap between the key and normal image frames caused by video compression. We finally illustrate the transferability with extensive experiments on several Deepfake benchmark datasets. The proposed solution consistently outperforms several state-of-the-art baselines on both within- and cross-dataset experiments.

A Computer-Aided Diagnostic System to Identify Diabetic Retinopathy, Utilizing a Modified Compact Convolutional Transformer and Low-Resolution Images to Reduce Computation Time

Diabetic retinopathy (DR) is the foremost cause of blindness in people with diabetes worldwide, and early diagnosis is essential for effective treatment. Unfortunately, the present DR screening method requires the skill of ophthalmologists and is time-consuming. In this study, we present an automated system for DR severity classification employing the fine-tuned Compact Convolutional Transformer (CCT) model to overcome these issues. We assembled five datasets to generate a more extensive dataset containing 53,185 raw images. Various image pre-processing techniques and 12 types of augmentation procedures were applied to improve image quality and create a massive dataset. A new DR-CCTNet model is proposed. It is a modification of the original CCT model to address training time concerns and work with a large amount of data. Our proposed model delivers excellent accuracy even with low-pixel images and still has strong performance with fewer images, indicating that the model is robust. We compare our model’s performance with transfer learning models such as VGG19, VGG16, MobileNetV2, and ResNet50. The test accuracy of the VGG19, ResNet50, VGG16, and MobileNetV2 were, respectively, 72.88%, 76.67%, 73.22%, and 71.98%. Our proposed DR-CCTNet model to classify DR outperformed all of these with a 90.17% test accuracy. This approach provides a novel and efficient method for the detection of DR, which may lower the burden on ophthalmologists and expedite treatment for patients.

Automated Skeletal Bone Age Assessment with Two-Stage Convolutional Transformer Network Based on X-ray Images.

Human skeletal development is continuous and staged, and different stages have various morphological characteristics. Therefore, bone age assessment (BAA) can accurately reflect the individual's growth and development level and maturity. Clinical BAA is time consuming, highly subjective, and lacks consistency. Deep learning has made considerable progress in BAA in recent years by effectively extracting deep features. Most studies use neural networks to extract global information from input images. However, clinical radiologists are highly concerned about the ossification degree in some specific regions of the hand bones. This paper proposes a two-stage convolutional transformer network to improve the accuracy of BAA. Combined with object detection and transformer, the first stage mimics the bone age reading process of the pediatrician, extracts the hand bone region of interest (ROI) in real time using YOLOv5, and proposes hand bone posture alignment. In addition, the previous information encoding of biological sex is integrated into the feature map to replace the position token in the transformer. The second stage extracts features within the ROI by window attention, interacts between different ROIs by shifting the window attention to extract hidden feature information, and penalizes the evaluation results using a hybrid loss function to ensure its stability and accuracy. The proposed method is evaluated on the data from the Pediatric Bone Age Challenge organized by the Radiological Society of North America (RSNA). The experimental results show that the proposed method achieves a mean absolute error (MAE) of 6.22 and 4.585 months on the validation and testing sets, respectively, and the cumulative accuracy within 6 and 12 months reach 71% and 96%, respectively, which is comparable to the state of the art, markedly reducing the clinical workload and realizing rapid, automatic, and high-precision assessment.

Spectral‐spatial sequence characteristics‐based convolutional transformer for hyperspectral change detection

AbstractRecently, ground coverings change detection (CD) driven by bitemporal hyperspectral images (HSIs) has become a hot topic in the remote sensing community. There are two challenges in the HSI‐CD task: (1) attribute feature representation of pixel pairs and (2) feature extraction of attribute patterns of pixel pairs. To solve the above problems, a novel spectral‐spatial sequence characteristics‐based convolutional transformer (S3C‐CT) method is proposed for the HSI‐CD task. In the designed method, firstly, an eigenvalue extrema‐based band selection strategy is introduced to pick up spectral information with salient attribute patterns. Then, a 3D tensor with spectral‐spatial sequence characteristics is proposed to represent the attribute features of pixel pairs in the bitemporal HSIs. Next, a fusion framework of the convolutional neural network (CNN) and Transformer encoder (TE) is designed to extract high‐order sequence semantic features, taking into account both local context information and global sequence dependencies. Specifically, a spatial‐spectral attention mechanism is employed to prevent information reduction and enhance dimensional interactivity between the CNN and TE. Finally, the binary change map is determined according to the fully‐connected layer. Experimental results on real HSI datasets indicated that the proposed S3C‐CT method outperforms other well‐known and state‐of‐the‐art detection approaches in terms of detection performance.

Unmixing-Guided Convolutional Transformer for Spectral Reconstruction

Deep learning networks based on CNNs or transformers have made progress in spectral reconstruction (SR). However, many methods focus solely on feature extraction, overlooking the interpretability of network design. Additionally, models exclusively based on CNNs or transformers may lose other prior information, sacrificing reconstruction accuracy and robustness. In this paper, we propose a novel Unmixing-Guided Convolutional Transformer Network (UGCT) for interpretable SR. Specifically, transformer and ResBlock components are embedded in Paralleled-Residual Multi-Head Self-Attention (PMSA) to facilitate fine feature extraction guided by the excellent priors of local and non-local information from CNNs and transformers. Furthermore, the Spectral–Spatial Aggregation Module (S2AM) combines the advantages of geometric invariance and global receptive fields to enhance the reconstruction performance. Finally, we exploit a hyperspectral unmixing (HU) mechanism-driven framework at the end of the model, incorporating detailed features from the spectral library using LMM and employing precise endmember features to achieve a more refined interpretation of mixed pixels in HSI at sub-pixel scales. Experimental results demonstrate the superiority of our proposed UGCT, especially in the grss_d f c_2018 dataset, in which UGCT attains an RMSE of 0.0866, outperforming other comparative methods.