Multi-stage Transformer Research Articles

MDH-Net: advancing 3D brain MRI registration with multi-stage transformer and dual-stream feature refinement hybrid network

MDH-Net: advancing 3D brain MRI registration with multi-stage transformer and dual-stream feature refinement hybrid network

DDformer: Dimension decomposition transformer with semi-supervised learning for underwater image enhancement

Vision-guided Autonomous Underwater Vehicles (AUVs) have gradually become significant tools for human exploration of the ocean. However, distorted images severely limit the visual ability, making it difficult to meet the needs of complex underwater environment perception. Fortunately, recent advancements in deep learning have led to rapid developments in underwater image enhancement. The emergence of the Transformer architecture has further enhanced the capabilities of deep learning. However, the direct application of Transformer to underwater image enhancement presents challenges in computing pixel-level global information and extracting local features. In this paper, we present a novel approach that merges dimension decomposition Transformer with semi-supervised learning for underwater image enhancement. To begin, dimension decomposition attention is proposed, which enables Transformer to compute global dependencies directly at the original scale and correct color distortions effectively. Concurrently, we employ convolutional neural networks to compensate for Transformer's limitations in extracting local features, thereby enriching details and textures. Subsequently, a multi-stage Transformer strategy is introduced to divide the network into high- and low-resolution stages for multi-scale global information extraction. It helps correct color distortions while enhancing the network's focus on regions with severe degradation. Moreover, we design a semi-supervised learning framework to reduce the reliance on paired datasets and construct a corresponding multi-scale fusion discriminator to enhance the sensitivity to input data. Experimental results demonstrate that our method outperforms state-of-the-art approaches, showcasing excellent learning and generalization capabilities on subjective perception and overall evaluation metrics. Furthermore, outstanding results highlight the significant improvements it brings to downstream visual engineering applications. The code of the proposed DDformer is available at https://github.com/ZhiGao-hfuu/DDformer.

SAMT-generator: A second-attention for image captioning based on multi-stage transformer network

In recent years, Transformer has been widely used in the crossing task of computer vision (CV) and natural language processing (NLP), e.g., image captioning. The prior works of image captioning based on Transformer have achieved remarkable progress. In order to further improve the model’s ability to describe detail features and generate high-quality sentences, in the paper, we first construct a Multi-stage Transformer Feature Enhancement Network (MT-FEN), which obtains more semantics by fusing features at different scales. Furthermore, we crucially propose a novel second-attention (SA) that can focus more on valuable features and filter out noises in MT-FEN. Besides that, to generate more sensible sentences, we use multiple decoding layers to choose the best-described word via the Max-out module. The overall model is dubbed as SAMT-Generator. After conducting extensive experiments on two well-known datasets: MS COCO and Flickr8k, the results demonstrate the effectiveness of our proposed model, and it is comparable with SOTA methods.

MSTCA-Net: A Novel Impervious Surface Extraction Method Based on a Multistage Transformer

MSTCA-Net: A Novel Impervious Surface Extraction Method Based on a Multistage Transformer

MAEMOT: Pretrained MAE-Based Antiocclusion 3-D Multiobject Tracking for Autonomous Driving.

The existing 3-D multiobject tracking (MOT) methods suffer from object occlusion in real-world traffic scenes. However, previous works have faced challenges in providing a reasonable solution to the fundamental question: "How can the interference of the perception data loss caused by occlusion be overcome?" Therefore, this article attempts to provide a reasonable solution by developing a novel pretrained movement-constrained masked autoencoder (M-MAE) for an antiocclusion 3-D MOT called MAEMOT. Specifically, for the pretrained M-MAE, this article adopts an efficient multistage transformer (MST) encoder and a spatiotemporal-based motion decoder to predict and reconstruct occluded point cloud data, following the properties of object motion. Afterward, the well-trained M-MAE model extracts the global features of occluded objects, ensuring that the features of the intraobjects between interframes are as consistent as possible throughout the spatiotemporal sequence. Next, a proposal-based geometric graph aggregation (PG 2 A) module is utilized to extract and fuse the spatial features of each proposal, producing refined region-of-interest (RoI) components. Finally, this article designs an object association module that combines geometric and corner affinities, which helps to match the predicted occlusion objects more robustly. According to an extensive evaluation, the proposed MAEMOT method can effectively overcome the interference of occlusion and achieve improved 3-D MOT performance under challenging conditions.

Global-Shared Text Representation Based Multi-Stage Fusion Transformer Network for Multi-Modal Dense Video Captioning

Global-Shared Text Representation Based Multi-Stage Fusion Transformer Network for Multi-Modal Dense Video Captioning

Progressive convolutional transformer for image restoration

In the past few years, convolutional neural networks (CNNs) have become the primary workhorse for image restoration tasks. However, the deficiency of modeling long-range dependencies due to the local computational property of convolution greatly limits the restoration performance. To overcome this limitation, we propose a novel multi-stage progressive convolutional Transformer to recursively restore the degraded images, termed PCformer, which enjoys a high capability for capturing local context and global dependencies with friendly computational cost. Specifically, each stage of PCformer is an asymmetric encoder–decoder network whose bottleneck is built upon a tailored Transformer block with convolution operation deployed to avoid any loss of local context. Both encoder and decoder are convolution-based modules, thus allowing to explore rich contextualized information for image recovery. Taking the low-resolution features encoded by the encoder as tokens input into the Transformer bottleneck guarantees that long-range pixel interactions are captured while reducing the computational burden. Meanwhile, we apply a gated module for filtering redundant information propagation between every two phases. In addition, long-range enhanced inpainting is further introduced to mining the ability of PCformer to exploit distant complementary features. Extensive experiments yield superior results and in particular establishing new state-of-the-art results on several image restoration tasks, including deraining (+0.37 dB on Rain13K), denoising (+0.11 dB on DND), dehazing (+0.56 dB on I-HAZE), enhancement (+0.72 dB on SICE), and shadow removal (+0.65 RMSE on ISTD). The implementation code is available at https://github.com/Jeasco/PCformer.

MSHT: Multi-stage Hybrid Transformer for the ROSE Image Analysis of Pancreatic Cancer.

Pancreatic cancer is one of the most malignant cancers with high mortality. The rapid on-site evaluation (ROSE) technique can significantly accelerate the diagnostic workflow of pancreatic cancer by immediately analyzing the fast-stained cytopathological images with on-site pathologists. However, the broader expansion of ROSE diagnosis has been hindered by the shortage of experienced pathologists. Deep learning has great potential for the automatic classification of ROSE images in diagnosis. But it is challenging to model the complicated local and global image features. The traditional convolutional neural network (CNN) structure can effectively extract spatial features, while it tends to ignore global features when the prominent local features are misleading. In contrast, the Transformer structure has excellent advantages in capturing global features and long-range relations, while it has limited ability in utilizing local features. We propose a multi-stage hybrid Transformer (MSHT) to combine the strengths of both, where a CNN backbone robustly extracts multi-stage local features at different scales as the attention guidance, and a Transformer encodes them for sophisticated global modeling. Going beyond the strength of each single method, the MSHT can simultaneously enhance the Transformer global modeling ability with the local guidance from CNN features. To evaluate the method in this unexplored field, a dataset of 4240 ROSE images is collected where MSHT achieves 95.68% in classification accuracy with more accurate attention regions. The distinctively superior results compared to the state-of-the-art models make MSHT extremely promising for cytopathological image analysis. The codes and records are available at: https://github.com/sagizty/ Multi-Stage-Hybrid-Transformer.

Multi-Stage Spatio-Temporal Aggregation Transformer for Video Person Re-Identification

In recent years, the Transformer architecture has shown its superiority in the video-based person re-identification task. Inspired by video representation learning, these methods mainly focus on designing modules to extract informative spatial and temporal features. However, they are still limited in extracting local attributes and global identity information, which are critical for the person re-identification task. In this paper, we propose a novel Multi-Stage Spatial-Temporal Aggregation Transformer (MSTAT) with two novel designed proxy embedding modules to address the above issue. Specifically, MSTAT consists of three stages to encode the attribute-associated, the identity-associated, and the attribute-identity-associated information from the video clips, respectively, achieving the holistic perception of the input person. We combine the outputs of all the stages for the final identification. In practice, to save the computational cost, the Spatial-Temporal Aggregation (STA) modules are first adopted in each stage to conduct the self-attention operations along the spatial and temporal dimensions separately. We further introduce the Attribute-Aware and Identity-Aware Proxy embedding modules (AAP and IAP) to extract the informative and discriminative feature representations at different stages. All of them are realized by employing newly designed self-attention operations with specific meanings. Moreover, temporal patch shuffling is also introduced to further improve the robustness of the model. Extensive experimental results demonstrate the effectiveness of the proposed modules in extracting the informative and discriminative information from the videos, and illustrate the MSTAT can achieve state-of-the-art accuracies on various standard benchmarks.

Composed Image Retrieval via Cross Relation Network With Hierarchical Aggregation Transformer.

Composing Text and Image to Image Retrieval (CTI-IR) aims at finding the target image, which matches the query image visually along with the query text semantically. However, existing works ignore the fact that the reference text usually serves multiple functions, e.g., modification and auxiliary. To address this issue, we put forth a unified solution, namely Hierarchical Aggregation Transformer incorporated with Cross Relation Network (CRN). CRN unifies modification and relevance manner in a single framework. This configuration shows broader applicability, enabling us to model both modification and auxiliary text or their combination in triplet relationships simultaneously. Specifically, CRN includes: 1) Cross Relation Network comprehensively captures the relationships of various composed retrieval scenarios caused by two different query text types, allowing a unified retrieval model to designate adaptive combination strategies for flexible applicability; 2) Hierarchical Aggregation Transformer aggregates top-down features with Multi-layer Perceptron (MLP) to overcome the limitations of edge information loss in a window-based multi-stage Transformer. Extensive experiments demonstrate the superiority of the proposed CRN over all three fashion-domain datasets. Code is available at github.com/yan9qu/crn.

Conv-Former: A Novel Network Combining Convolution and Self-Attention for Image Quality Assessment.

To address the challenge of no-reference image quality assessment (NR-IQA) for authentically and synthetically distorted images, we propose a novel network called the Combining Convolution and Self-Attention for Image Quality Assessment network (Conv-Former). Our model uses a multi-stage transformer architecture similar to that of ResNet-50 to represent appropriate perceptual mechanisms in image quality assessment (IQA) to build an accurate IQA model. We employ adaptive learnable position embedding to handle images with arbitrary resolution. We propose a new transformer block (TB) by taking advantage of transformers to capture long-range dependencies, and of local information perception (LIP) to model local features for enhanced representation learning. The module increases the model's understanding of the image content. Dual path pooling (DPP) is used to keep more contextual image quality information in feature downsampling. Experimental results verify that Conv-Former not only outperforms the state-of-the-art methods on authentic image databases, but also achieves competing performances on synthetic image databases which demonstrate the strong fitting performance and generalization capability of our proposed model.

Multi-Stage Transformer 3D Object Detection Method

With the development of autonomous driving, 3D object detection has experience great expectations. As the light detection and ranging (LiDAR) sensor can precisely measure the distance between environments and themselves, it has become the key component of current 3D object detection methods. However, the varing density and unstructure storage of LiDAR points cloud make it hard for feature learning. To tackle this problem, this paper proposes a multi-task transformer 3D object detection method.This method include a fast transformer based 3D encoder and a multi-stage transformer decoder. Extensive experiments demonstrate that our method can supress current other 3D object detection methods with a clear margin.

ARET-IQA: An Aspect-Ratio-Embedded Transformer for Image Quality Assessment

Image quality assessment (IQA) aims to automatically evaluate image perceptual quality by simulating the human visual system, which is an important research topic in the field of image processing and computer vision. Although existing deep-learning-based IQA models have achieved significant success, these IQA models usually require input images with a fixed size, which varies the perceptual quality of images. To this end, this paper proposes an aspect-ratio-embedded Transformer-based image quality assessment method, which can implant the adaptive aspect ratios of input images into the multihead self-attention module of the Swin Transformer. In this way, the proposed IQA model can not only relieve the variety of perceptual quality caused by size changes in input images but also leverage more global content correlations to infer image perceptual quality. Furthermore, to comprehensively capture the impact of low-level and high-level features on image quality, the proposed IQA model combines the output features of multistage Transformer blocks for jointly inferring image quality. Experimental results on multiple IQA databases show that the proposed IQA method is superior to state-of-the-art methods for assessing image technical and aesthetic quality.

Grouped Bidirectional LSTM Network and Multistage Fusion Convolutional Transformer for Hyperspectral Image Classification

The efficiently and effectively discriminative spectral-spatial feature representation is essential for hyperspectral image (HSI) classification. However, most of the existing methods rely on the patch-based convolutional neural networks (CNNs) whose ability of extracting the global spatial information is very limited. To address this issue, in this paper, we propose a two-branch network consisting of a grouped bidirectional long short-term memory (GBiLSTM) network and multi-stage fusion convolutional transformer (MFCT) for HSI classification. In the proposed GBiLSTM-MFCT, to extract the spectral features of HSI efficiently, a GBiLSTM network is designed by dividing the sequence features and hidden units of BiLSTM network into several separate groups. To simultaneously extract the global and local spatial features of HSI, a MFCT is proposed by fusing the features of different levels obtained from the multiple phases of convolutional vision transformer. Moreover, in the multi-headed attention module of each stage, blueprint separable convolution based self-attention (BSCA) module is designed which is able to model the global and local spatial information effectively. The outputs of GBiLSTM network and MFCT are fused to generate discriminative and robust spectral-spatial features for HSI classification. Experiments on three benchmark data sets of IN, UP and KSC demonstrate that the proposed GBiLSTM-MFCT exhibits higher classification performance with very limited labeled samples than eight state-of-the-art methods.

Energy transmission control for a Gridconnected modern power system Non-Linear loads with a Series Multi-Stage Transformer Voltage Reinjection with controlled converters

The effective way of energy transmission plays a key factor in improving the overall transmission systems efficiency. Many methods are proposed to control the reactive power flow, voltage fluctuations and power factor improvement, The proposed converter topology gives a much significant improvement in transmission systems performance which includes multistage transformers control with the controlled converters along with the series active filters. The overall control strategy which involves the Multistage Voltage Re-Injection Transformer Controlled Converters (MSVRITCC) to reinject the voltages into the grid to compensate the voltages and remaining parameters and power flow control. The proposed topology improves the grid security, flexibility in reaching the desired load requirements with grid adaptability and reduces THD values into a significant values and made the control of power conditioning circuit flexible and easy to perform the voltage compensations in grid to load connected applications. The binary control is used to trigger the power converter circuits which made the controlling much simpler.

A Repetitive High-Current Pulse Generator Circuit Based on Multistage Pulse Transformers

The application of inductive energy storage in the generation of high-current pulses has attracted considerable attention during recent years. In this article, a new inductive high-current pulse generator circuit is proposed based on XRAM (MARX spelled backword) current multiplier converter concept and multistage pulse transformers by using power electronic switches. This circuit is capable to recover the residual energy and generate repetitive high-current pulses with a flexible output pattern. To verify the circuit operation, simulation and experimental results of a two-stage laboratory prototype are presented. The results confirm the theoretical analysis and show the validity of the converter scheme.

An in-line coaxial-to-circular waveguide transition at X band

Abstract A wideband transition from the coaxial TEM mode to the TM01 mode of the circular waveguide with in-line input-output architecture is proposed in this paper. The transition employs multi-stage stepped impedance transformer to achieve wideband responses. Both mode transition and impedance matching characteristics are discussed in the paper. A demonstrator is developed and validated experimentally. As compared with some recently published reports, a better performance is observed. Moreover, the developed transition could facilitate the mechanical fabrication more easily. It can be found wide applications such as the circular waveguide based slot array antenna, the nondestructive detection, cracks inside the conductive pipes, and so on.

Design and Experiment of Via-Less and Small-Radiation Waveguide to Microstrip Line Transitions for Millimeter Wave Radar Modules

We propose waveguide to microstrip line transitions for automotive millimeter wave radar modules. The transitions perpendicularly connect one waveguide and one or two microstrip lines. The configuration is simple because it consists of a waveguide and a dielectric substrate with copper foils. Additionally the transitions do not need via holes on the substrate. It leads to lower costs and improved reliability. We have already proposed a via-less transition by using multi-stage impedance transformers. The impedance transformers are used for suppressing undesirable radiation from the transition as well as impedance matching. In this paper, we propose a new transition with the microstrip lines on the long axis of the waveguide while most transitions place the microstrip lines on the minor axis (electric field direction) of the waveguide. Though our transition uses bend structures of microstrip lines, which basically cause radiation, our optimized configuration can keep small radiation. We also design a transition with a single microstrip line. The proposed transition with 2 microstrip lines can be modified to the 1 microstrip line version with minimum radiation loss. Electromagnetic simulations confirm the small radiation levels expected. Additionally we fabricate the transitions with back to back structure and determine the transmission and radiation performance. We also fabricates the transition for a patch array antenna. We confirm that the undesirable radiation from the proposed transition is small and the radiation pattern of the array antenna is not worsen by the transition.

Trapezoidal Approximation of LCC Resonant Converter and Design of a Multistage Capacitor Charger for a Solid-State Marx Modulator

This paper describes the circuit of a capacitor charger for a solid-state Marx modulator. A high-efficiency LCC resonant inverter is proposed for the simultaneous charging of many capacitors in parallel using a multistage transformer and rectifier. Using a high-voltage insulation cable, the charging loop, which represents the primary winding of transformer, is implemented to transfer the power from the resonant inverter to each stage. In addition to the simultaneous charging of separate capacitors, the advantages of the proposed circuit and structure include compact design and reliable insulation performance against high-voltage pulses. Based on the relevant approximation, the simplified analysis of an LCC resonant converter with a trapezoidal shape of resonant current is provided and a 50-kW high-voltage capacitor charger is designed for a 40-kV solid-state Marx modulator. The power stage, which consists of a transformer with four voltage-doubled rectifiers, is designed, and six power stages are configured for generating a pulse output of 40 kV. The detailed implementation of the multistage transformer and rectifier is presented in accordance with the design requirements of a high voltage modulator. Finally, the developed charger achieves 96% of the maximum efficiency and 0.96 of the maximum power factor. The experimental results verify that the proposed circuit and structure can be effectively used for a solid-state Marx modulator.

Optimal capacity planning of substation transformers by demand response combined with network automation

The inclusion of smart grid features such as demand response (DR) and network automation for capacity planning of substation transformers may provide substantial monetary savings. This paper proposes an optimization model for quantification of the savings in capacity management of substation transformers over long-run. The proposed model incorporates the DR as a resource to decrease the outage cost during contingencies while considering existing switching types for load transfer between substations. The model provides optimal selection and scheduling of multistage transformer installations and their refurbishments by considering all the costs associated with them including investment, losses, maintenance, reliability, and the salvage value. For a realistic study, numerical value of the savings in transformers’ cost is calculated for a typical Finnish two-transformer primary distribution substation planning over a period of forty years. Case studies are performed based on situations encountered by utilities and type of load transfer switching (manual and remote) between substations. A sensitivity analysis based on DR penetration and load curtailment (LC) cost is also performed. The results indicate that substantial monetary benefits can be obtained in substation transformers’ cost by utilities through employing DR. The benefit of DR is superior for cases where it is used in combination with remote switching of load transfer between neighbouring substations (NSS).