Convolution Transformation Research Articles

CTBANet: Convolution transformers and bidirectional attention for medical image segmentation

In the last few years, Transformer has revolutionized the area of medical image segmentation. Several similar studies have used the UNet architecture to combine convolutional neural networks with transformers. However, these approaches fail to account for the speed at which segmentation occurs and the ability to extract features within the Transformer. They fail to consider the fact that changing the shape of the feature maps in a subtle way can be used for rapid extraction of local and global information. To solve the above problems, CTBANet (Convolutional Transformer and Bidirectional Attention Based for Medical Image Segmentation) is proposed, which has two prominent components, CTblock (Convolutional Combined Transformer module) and BAblock (Bidirectional Attentionblock). CTblock integrates the strengths of CNNs and Transformers, enabling it to extract spatial details and global data. In order to improve the speed and accuracy of the model, multi-scale pyramid pooling is embedded into PAM, named APAM (Asymmetric PAM), and strip convolution is embedded into CAM, named ACAM (Asymmetric CAM). Medical image segmentation is a critical issue in the medical field, and the experimental results of the benchmarks show that our model is obviously more accurate and faster than the other methods in segmenting medical images.

Physics-informed convolutional transformer for predicting volatility surface

Predicting volatility is important for asset predicting, option pricing and hedging strategies because it cannot be directly observed in the financial market. The dynamics of the volatility surface is difficult to estimate. In this paper, we establish a novel architecture based on physics-informed neural networks and convolutional transformers. The performance of the new architecture is directly compared to other well-known deep-learning architectures, such as standard physics-informed neural networks, convolutional long-short term memory (ConvLSTM), and self-attention ConvLSTM. Numerical evidence indicates that the proposed physics-informed convolutional transformer network achieves a superior performance than other methods.

Real‐time prediction of horizontal drilling pressure based on convolutional Transformer

SummaryDuring horizontal drilling operations, real‐time prediction of drilling pressure during the drilling process can help the drilling team cope with the complex and changing working environment downhole, adjust the parameters of the drilling rig promptly, make correct decisions, reduce the probability of drilling accidents, and avoid affecting the duration and cost of the project. This study provides a method for real‐time prediction of the drilling pressure of horizontal drilling rigs. A deep learning model based on a convolutional Transformer is trained for accurate real‐time prediction by extracting real‐time operating data of the horizontal drilling rig from the data acquisition system. The method proposed in this study can be a useful tool to improve the performance of horizontal drilling rigs and can assist the drilling team in operating horizontal drilling rigs. The results of the case study show that: (1) the proposed convolutional Transformer model provides reliable real‐time prediction with an MAE of 0.304 MPa and an RMSE of 0.508 MPa; (2) the proposed method can quickly and accurately predict the trend of drilling pressure change in the next period based on the current change of drilling pressure, and grasp the dynamics of drilling pressure of horizontal drilling rigs in advance. Further research could focus on assisted decision‐making and intelligent optimization to provide solutions for preventing drilling accidents and improving horizontal rig performance based on the prediction.

Residual Mask in Cascaded Convolutional Transformer for Spectral Reconstruction

Residual Mask in Cascaded Convolutional Transformer for Spectral Reconstruction

A Novel Method for Code Clone Detection Based on Minimally Random Kernel Convolutional Transform

A Novel Method for Code Clone Detection Based on Minimally Random Kernel Convolutional Transform

U-shaped convolutional transformer GAN with multi-resolution consistency loss for restoring brain functional time-series and dementia diagnosis.

The blood oxygen level-dependent (BOLD) signal derived from functional neuroimaging is commonly used in brain network analysis and dementia diagnosis. Missing the BOLD signal may lead to bad performance and misinterpretation of findings when analyzing neurological disease. Few studies have focused on the restoration of brain functional time-series data. In this paper, a novel U-shaped convolutional transformer GAN (UCT-GAN) model is proposed to restore the missing brain functional time-series data. The proposed model leverages the power of generative adversarial networks (GANs) while incorporating a U-shaped architecture to effectively capture hierarchical features in the restoration process. Besides, the multi-level temporal-correlated attention and the convolutional sampling in the transformer-based generator are devised to capture the global and local temporal features for the missing time series and associate their long-range relationship with the other brain regions. Furthermore, by introducing multi-resolution consistency loss, the proposed model can promote the learning of diverse temporal patterns and maintain consistency across different temporal resolutions, thus effectively restoring complex brain functional dynamics. We theoretically tested our model on the public Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset, and our experiments demonstrate that the proposed model outperforms existing methods in terms of both quantitative metrics and qualitative assessments. The model's ability to preserve the underlying topological structure of the brain functional networks during restoration is a particularly notable achievement. Overall, the proposed model offers a promising solution for restoring brain functional time-series and contributes to the advancement of neuroscience research by providing enhanced tools for disease analysis and interpretation.

DCTN: Dual-Branch Convolutional Transformer Network With Efficient Interactive Self-Attention for Hyperspectral Image Classification

DCTN: Dual-Branch Convolutional Transformer Network With Efficient Interactive Self-Attention for Hyperspectral Image Classification

Self-Supervised Masked Convolutional Transformer Block for Anomaly Detection.

Anomaly detection has recently gained increasing attention in the field of computer vision, likely due to its broad set of applications ranging from product fault detection on industrial production lines and impending event detection in video surveillance to finding lesions in medical scans. Regardless of the domain, anomaly detection is typically framed as a one-class classification task, where the learning is conducted on normal examples only. An entire family of successful anomaly detection methods is based on learning to reconstruct masked normal inputs (e.g. patches, future frames, etc.) and exerting the magnitude of the reconstruction error as an indicator for the abnormality level. Unlike other reconstruction-based methods, we present a novel self-supervised masked convolutional transformer block (SSMCTB) that comprises the reconstruction-based functionality at a core architectural level. The proposed self-supervised block is extremely flexible, enabling information masking at any layer of a neural network and being compatible with a wide range of neural architectures. In this work, we extend our previous self-supervised predictive convolutional attentive block (SSPCAB) with a 3D masked convolutional layer, a transformer for channel-wise attention, as well as a novel self-supervised objective based on Huber loss. Furthermore, we show that our block is applicable to a wider variety of tasks, adding anomaly detection in medical images and thermal videos to the previously considered tasks based on RGB images and surveillance videos. We exhibit the generality and flexibility of SSMCTB by integrating it into multiple state-of-the-art neural models for anomaly detection, bringing forth empirical results that confirm considerable performance improvements on five benchmarks: MVTec AD, BRATS, Avenue, ShanghaiTech, and Thermal Rare Event.

SWFormer: Stochastic Windows Convolutional Transformer for Hybrid Modality Hyperspectral Classification.

Joint classification of hyperspectral images with hybrid modality can significantly enhance interpretation potentials, particularly when elevation information from the LiDAR sensor is integrated for outstanding performance. Recently, the transformer architecture was introduced to the HSI and LiDAR classification task, which has been verified as highly efficient. However, the existing naive transformer architectures suffer from two main drawbacks: 1) Inadequacy extraction for local spatial information and multi-scale information from HSI simultaneously. 2) The matrix calculation in the transformer consumes vast amounts of computing power. In this paper, we propose a novel Stochastic Window Transformer (SWFormer) framework to resolve these issues. First, the effective spatial and spectral feature projection networks are built independently based on hybrid-modal heterogeneous data composition using parallel feature extraction, which is conducive to excavating the perceptual features more representative along different dimensions. Furthermore, to construct local-global nonlinear feature maps more flexibly, we implement multi-scale strip convolution coupled with a transformer strategy. Moreover, in an innovative random window transformer structure, features are randomly masked to achieve sparse window pruning, alleviating the problem of information density redundancy, and reducing the parameters required for intensive attention. Finally, we designed a plug-and-play feature aggregation module that adapts domain offset between modal features adaptively to minimize semantic gaps between them and enhance the representational ability of the fusion feature. Three fiducial datasets demonstrate the effectiveness of the SWFormer in determining classification results.

Tomato maturity recognition with convolutional transformers

Tomatoes are a major crop worldwide, and accurately classifying their maturity is important for many agricultural applications, such as harvesting, grading, and quality control. In this paper, the authors propose a novel method for tomato maturity classification using a convolutional transformer. The convolutional transformer is a hybrid architecture that combines the strengths of convolutional neural networks (CNNs) and transformers. Additionally, this study introduces a new tomato dataset named KUTomaData, explicitly designed to train deep-learning models for tomato segmentation and classification. KUTomaData is a compilation of images sourced from a greenhouse in the UAE, with approximately 700 images available for training and testing. The dataset is prepared under various lighting conditions and viewing perspectives and employs different mobile camera sensors, distinguishing it from existing datasets. The contributions of this paper are threefold: firstly, the authors propose a novel method for tomato maturity classification using a modular convolutional transformer. Secondly, the authors introduce a new tomato image dataset that contains images of tomatoes at different maturity levels. Lastly, the authors show that the convolutional transformer outperforms state-of-the-art methods for tomato maturity classification. The effectiveness of the proposed framework in handling cluttered and occluded tomato instances was evaluated using two additional public datasets, Laboro Tomato and Rob2Pheno Annotated Tomato, as benchmarks. The evaluation results across these three datasets demonstrate the exceptional performance of our proposed framework, surpassing the state-of-the-art by 58.14%, 65.42%, and 66.39% in terms of mean average precision scores for KUTomaData, Laboro Tomato, and Rob2Pheno Annotated Tomato, respectively. This work can potentially improve the efficiency and accuracy of tomato harvesting, grading, and quality control processes.

CTNet: convolutional transformer network for diabetic retinopathy classification

CTNet: convolutional transformer network for diabetic retinopathy classification

Extracting long‐term spatiotemporal characteristics of traffic flow using attention‐based convolutional transformer

AbstractPredicting traffic flow is vital for optimizing transportation efficiency, reducing fuel consumption, and minimizing commute times. While artificial intelligence tools have been effective in addressing this, there have been some difficulties in processing spatial and temporal data. Current transformer‐based methods, although cutting‐edge for traffic prediction, encounter challenges with handling long sequences and capturing temporal relations effectively. Addressing these, the research introduces a model combining multi‐scale attention modules within transformer layers. This model employs spatio‐temporal transformer blocks, enriched with multi‐scale convolutional attention mechanisms, allowing for a deeper understanding of temporal and spatial traffic patterns. This unique attention mechanism enhances data feature interpretation, leading to heightened prediction precision. The tests on extensive traffic datasets showcase the model's prowess in capturing both local and global traffic features, resulting in superior traffic status predictions. In summary, the innovative model offers an efficacious approach to long‐sequence traffic data learning and temporal relationship extraction, setting a new benchmark in traffic flow prediction accuracy.

Convolutional transformer network for fine-grained action recognition

Fine-grained action recognition is one of the critical problems in video processing, which aims to recognize similar actions of subtle interactions between humans and objects. Inspired by the remarkable performance of the Transformer in natural language processing, Transformer has been applied to the fine-grained action recognition task. However, Transformer needs abundant training data and extra supervision to achieve comparable results with convolutional neural networks (CNNs). To address these issues, we propose a Convolutional Transformer Network (CTN), which integrates the merits of CNN (e.g., sharing weights, capturing low-level features in videos and locality) and the benefits of Transformer (e.g., dynamic attention and learning long-range dependencies). In this paper, we propose two modifications to the original Transformer: (i) We propose a video-to-tokens module that can extract tokens from extracted spatial-temporal features in videos by 3D convolutions instead of the direct token embedding from raw input video clips; (ii) We completely replace the linear mapping in multi-head self-attention layer with depth-wise convolutional mapping, which applies a depth-wise separable convolution operation on embedded token maps. With these two modifications, our approach can extract effective spatial-temporal features from videos and process the long sequences of tokens encountered in videos. Experimental results demonstrate that our proposed CTN can achieve state-of-the-art accuracy on two fine-grained action recognition datasets (i.e., Epic-Kitchens and Diving 48) with a small computational increase.

A lightweight transformer and depthwise separable convolution model for remaining useful life prediction of turbofan engines

The degradation of turbofan engines under complex operating conditions makes it difficult to predict their remaining useful life (RUL), which affects aircraft maintenance efficiency and reliability. To maintain prediction accuracy while improving prediction speed under the limited computing power and memory resources of edge devices, a lightweight Transformer and depthwise separable convolutional neural network (DSCformer) prediction model has been proposed. In the proposed DSCformer method, a probsparse self-attention mechanism with convolutional transformation of the Value branch is developed to improve the efficiency of dot-product, and depthwise separable convolution is employed to extract local spatiotemporal features replace the decoder in Transformer. Additionally, the model’s ability to capture overall trends is improved by incorporating a scaling factor in the Bayesian optimization algorithm, which also accelerates the search for the smoothing coefficient. The evaluation on the C-MAPSS dataset shows that the proposed method achieves a root mean square error of 11.33 and 12.44, as well as scores of 634.22 and 947.35 for predicting FD002 and FD004, respectively, within a shorter training time. These results indicate that the proposed method outperforms state-of-the-art prediction methods under multiple operating conditions for aero engine RUL prediction.

SEMG-Based End-to-End Continues Prediction of Human Knee Joint Angles Using the Tightly Coupled Convolutional Transformer Model.

Wearable exoskeleton robots can promote the rehabilitation of patients with physical dysfunction. And improving human-computer interaction performance is a significant challenge for exoskeleton robots. The traditional feature extraction process based on surface Electromyography(sEMG) is complex and requires manual intervention, making real-time performance difficult to guarantee. In this study, we propose an end-to-end method to predict human knee joint angles based on sEMG signals using a tightly coupled convolutional transformer (TCCT) model. We first collected sEMG signals from 5 healthy subjects. Then, the envelope was extracted from the noise-removed sEMG signal and used as the input to the model. Finally, we developed the TCCT model to predict the knee joint angle after 100 ms. For the prediction performance, we used the Root Mean Square Error(RMSE), Pearson Correlation Coefficient(CC), and Adjustment R2 as metrics to evaluate the error between the actual knee angle and the predicted knee angle. The results show that the model can predict the human knee angle quickly and accurately. The mean RMSE, Adjustment R2, and (CC) values of the model are 3.79°, 0.96, and 0.98, respectively, which are better than traditional deep learning models such as Informer (4.14, 0.95, 0.98), CNN (5.56, 0.89, 0.96) and CNN-BiLSTM (3.97, 0.95, 0.98). In addition, the prediction time of our proposed model is only 11.67±0.67 ms, which is less than 100 ms. Therefore, the real-time and accuracy of the model can meet the continuous prediction of human knee joint angle in practice.

Semi-supervised diagnosis method of refrigeration compressor hidden defect based on convolutional transformer autoencoder model

Improving product quality and service life by diagnosing defects in compressors through vibration signal analysis of the compressor's shell is challenging due to strong noise interference and limited labeled samples. The paper proposed a semi-supervised diagnosis method based on the convolution Transformer autoencoder (CTAE) for diagnosing hidden defects with small samples and strong noise. Firstly, an optimized variational modal decomposition is utilized to decouple and reduce noise from the raw vibrations, and the decomposition results are concatenated with the vibration signals as input to the model; Secondly, the CTAE is employed to learn the feature distribution of unlabeled samples and to extract and fuse local and global features from the input data; Finally, a labeled samples are used to fine-tune the model and to fuse features from multi-sensor information. The results of using a compressor dataset for validation show that the proposed method has high diagnosis accuracy and robustness with limited labeled samples and different signal-to-noise ratios.

Deep convolutional transformer network for hyperspectral unmixing

ABSTRACT Hyperspectral unmixing (HU) is considered one of the most important ways to improve hyperspectral image analysis. HU aims to break down the mixed pixel into a set of spectral signatures, often commonly referred to as endmembers, and determine the fractional abundance of those endmembers. Deep learning (DL) approaches have recently received great attention regarding HU. In particular, convolutional neural networks (CNNs)-based methods have performed exceptionally well in such tasks. However, the ability of CNNs to learn deep semantic features is limited, and computing cost increase dramatically with the number of layers. The appearance of the transformer addresses these issues by effectively representing high-level semantic features well. In this article, we present a novel approach for HU that utilizes a deep convolutional transformer network. Firstly, the CNN-based autoencoder (AE) is used to extract low-level features from the input image. Secondly, the concept of tokenizer is applied for feature transformation. Thirdly, the transformer module is used to capture the deep semantic features derived from the tokenizer. Finally, a convolutional decoder is utilized to reconstruct the input image. The experimental results on synthetic and real datasets demonstrate the effectiveness and superiority of the proposed method compared with other unmixing methods.

Knowledge enhanced ensemble method for remaining useful life prediction under variable working conditions

Remaining useful life (RUL) prediction is essential in enhancing the safety and reliability of rotating machinery. Deep learning techniques have been extensively researched and demonstrated promising results in RUL prediction tasks. But most existing models are designed for machinery equipment in a specific condition. In this case, a novel prediction method, knowledge-enhanced convolutional Transformer ensemble model (KE-CTEM), is proposed in this study. First, a feature extraction neural network (FENN) is introduced to extract features and transfer the working conditions information of existing datasets as knowledge to downstream RUL prediction tasks. Then, a convolutional Transformer model is leveraged to capture the input data degradation patterns and predict RUL values. Finally, knowledge-enhanced strategy and ensemble strategy are proposed to enhance the robustness of the model and improve the prediction accuracy.To verify the practicality and effectiveness of the proposed method, run-to-failure data of bearings from PRONOSTIA platform are utilized for RUL prognostics. Compared with several representative and state-of-the-art methods, the experimental results demonstrate the superiority and feasibility of the proposed method. And ablation study indicates the high efficiency and robustness of each module within the proposed model. Compared with representative RUL prediction methods, the proposed KE-CTEM demonstrates superior performance in terms of RMSE and MAPE with a reduction of 32.0% and 16.2%, respectively.

U-shaped fusion convolutional transformer based workflow for fast optical coherence tomography angiography generation in lips.

Oral disorders, including oral cancer, pose substantial diagnostic challenges due to late-stage diagnosis, invasive biopsy procedures, and the limitations of existing non-invasive imaging techniques. Optical coherence tomography angiography (OCTA) shows potential in delivering non-invasive, real-time, high-resolution vasculature images. However, the quality of OCTA images are often compromised due to motion artifacts and noise, necessitating more robust and reliable image reconstruction approaches. To address these issues, we propose a novel model, a U-shaped fusion convolutional transformer (UFCT), for the reconstruction of high-quality, low-noise OCTA images from two-repeated OCT scans. UFCT integrates the strengths of convolutional neural networks (CNNs) and transformers, proficiently capturing both local and global image features. According to the qualitative and quantitative analysis in normal and pathological conditions, the performance of the proposed pipeline outperforms that of the traditional OCTA generation methods when only two repeated B-scans are performed. We further provide a comparative study with various CNN and transformer models and conduct ablation studies to validate the effectiveness of our proposed strategies. Based on the results, the UFCT model holds the potential to significantly enhance clinical workflow in oral medicine by facilitating early detection, reducing the need for invasive procedures, and improving overall patient outcomes.

Two Birds With One Stone: Knowledge-Embedded Temporal Convolutional Transformer for Depression Detection and Emotion Recognition

Depression is a critical problem in modern society that affects an estimated 350 million people worldwide, causing feelings of sadness and a lack of interest and pleasure. Emotional disorders are gaining interest and are closely entwined with depression, because one contributes to an understanding of the other. Despite the achievements in the two separate tasks of emotion recognition and depression detection, there has not been much prior effort to build a unified model that can connect these two tasks with different modalities, including multimedia (text, audio, and video) and unobtrusive physiological signals (e.g., electroencephalography). We propose a novel <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">temporal convolutional transformer with knowledge embedding</b> to address the joint task of depression detection and emotion recognition. This approach not only learns multimodal embeddings across domains via the temporal convolutional transformer but also exploits special-domain knowledge from medical knowledge graphs to improve the performance of detection and recognition. It is essential that the features learned by our method can be perceived as a priori and are suitable for increasing the performance of other related tasks. Our method illustrates the case of “two birds with one stone” in the sense that two or more tasks can be efficiently handled with our unique model, which captures effective features. Experimental results on ten real-world datasets show that the proposed approach significantly outperforms other state-of-the-art approaches. On the other hand, experiments in which our methodology is applied to other reasoning tasks show that our approach effectively supports model reasoning related to emotion and improves its performance.