Hybrid Transformer Research Articles

VQ-InfraTrans: A Unified Framework for RGB-IR Translation with Hybrid Transformer

Infrared (IR) images containing rich spectral information are essential in many fields. Most RGB-IR transfer work currently relies on conditional generative models to learn and train IR images for specific devices and scenes. However, these models only establish an empirical mapping relationship between RGB and IR images in a single dataset, which cannot achieve the multi-scene and multi-band (0.7–3 μm and 8–15 μm) transfer task. To address this challenge, we propose VQ-InfraTrans, a comprehensive framework for transferring images from the visible spectrum to the infrared spectrum. Our framework incorporates a multi-mode approach to RGB-IR image transferring, encompassing both unconditional and conditional transfers, achieving diverse and flexible image transformations. Instead of training individual models for each specific condition or dataset, we propose a two-stage transfer framework that integrates diverse requirements into a unified model that utilizes a composite encoder–decoder based on VQ-GAN, and a multi-path transformer to translate multi-modal images from RGB to infrared. To address the issue of significant errors in transferring specific targets due to their radiance, we have developed a hybrid editing module to precisely map spectral transfer information for specific local targets. The qualitative and quantitative comparisons conducted in this work reveal substantial enhancements compared to prior algorithms, as the objective evaluation metric SSIM (structural similarity index) was improved by 2.24% and the PSNR (peak signal-to-noise ratio) was improved by 2.71%.

Hybrid UNet transformer architecture for ischemic stoke segmentation with MRI and CT datasets.

A hybrid UNet and Transformer (HUT) network is introduced to combine the merits of the UNet and Transformer architectures, improving brain lesion segmentation from MRI and CT scans. The HUT overcomes the limitations of conventional approaches by utilizing two parallel stages: one based on UNet and the other on Transformers. The Transformer-based stage captures global dependencies and long-range correlations. It uses intermediate feature vectors from the UNet decoder and improves segmentation accuracy by enhancing the attention and relationship modeling between voxel patches derived from the 3D brain volumes. In addition, HUT incorporates self-supervised learning on the transformer network. This allows the transformer network to learn by maintaining consistency between the classification layers of the different resolutions of patches and augmentations. There is an improvement in the rate of convergence of the training and the overall capability of segmentation. Experimental results on benchmark datasets, including ATLAS and ISLES2018, demonstrate HUT's advantage over the state-of-the-art methods. HUT achieves higher Dice scores and reduced Hausdorff Distance scores in single-modality and multi-modality lesion segmentation. HUT outperforms the state-the-art network SPiN in the single-modality MRI segmentation on Anatomical Tracings of lesion After Stroke (ATLAS) dataset by 4.84% of Dice score and a large margin of 40.7% in the Hausdorff Distance score. HUT also performed well on CT perfusion brain scans in the Ischemic Stroke Lesion Segmentation (ISLES2018) dataset and demonstrated an improvement over the recent state-of-the-art network USSLNet by 3.3% in the Dice score and 12.5% in the Hausdorff Distance score. With the analysis of both single and multi-modalities datasets (ATLASR12 and ISLES2018), we show that HUT can perform and generalize well on different datasets. Code is available at: https://github.com/vicsohntu/HUT_CT.

Watermarking Techniques with Yuv Layered Approach

Watermarking is a technique used to embed hidden data or identification information within digital multimedia. With the rise of the internet, the creation and distribution of digital data (images, video, and audio files) have significantly increased, leading to issues concerning content protection and ownership proof. Digital image watermarking addresses these issues by embedding a watermark into the data to assert ownership and protect rights. To safeguard the watermark from counterfeiters, it must be embedded in locations resilient to various attacks (rotation, scaling, compression, cropping, filtering, and blurring). Effective watermarking hides a message signal within a host signal without perceptual distortion. This paper adopts a hybrid transformation approach to modify the cover image's singular values rather than the DWT sub-bands, making the watermark resistant to attacks while maintaining the image's original state. Simulation results support the effectiveness of the proposed methods.

Effect of the Combination of the Growth Regulators and Putrescine on the Somatic Embryogenesis of Wheat (Triticum Aestivum L.) on Some Types of Explants

Plant breeding programs need to be carried out in order to improve the genetics of wheat that is able to adapt to tropical environments through hybridization, mutation induction, tissue culture, and genetic transformation. In vitro culture through somatic embryogenesis pathways plays an important role in genetic improvement and its integration with other breeding programs can positively affect the improvement of wheat quality, quantity, and development in Indonesia. The purpose of this study was to obtain an embryogenic callus induction method from the Dewata variety using five different types of explants, namely mature Seeds, immature embryos, immature seeds, leaf, stem, and to obtain combination of plant growth regulators and putrescine on somatic embryogenesis of wheat. The experimental design was prepared based on a complete randomized design with a combination treatment of embryogenic callus induction media consisting of 9 levels, namely: 1 ppm 2.4- D, 1 ppm 2.4-D + 1 ppm Picloram, ppm 2.4-D + 1 mg / L Picloram, 1 ppm 2.4-D + 10-4 M Putrescine, 1 ppm 2.4-D + 1 ppm Picloram +10-4 M Putrescine, 2 ppm 2.4-D + 1 ppm Picloram + 10-4 M Putrescine, 1 ppm 2.4 D + 10-3 M Putrescine, 1 ppm 2.4 D + 1 ppm Picloram + 10-3 M Putrescine, 2 ppm 2.4 D + 1 ppm Picloram + 10-3 M Putrescine. The results showed that the media used was able to induce embryogenic callus using mature seed and immature embryo, but immature seed and leaf were not able to produce embryogenic callus. The best media that produced the highest percentage of embryogenic callus was 2 ppm 2.4-D + 1 ppm Picloram + 10-4 M Putrescine with as much as 85.9% in young embryo explants.

SignalFormer: Hybrid Transformer for Automatic Drone Identification Based on Drone RF Signals.

With the growing integration of drones into various civilian applications, the demand for effective automatic drone identification (ADI) technology has become essential to monitor malicious drone flights and mitigate potential threats. While numerous convolutional neural network (CNN)-based methods have been proposed for ADI tasks, the inherent local connectivity of the convolution operator in CNN models severely constrains RF signal identification performance. In this paper, we propose an innovative hybrid transformer model featuring a CNN-based tokenization method that is capable of generating T-F tokens enriched with significant local context information, and complemented by an efficient gated self-attention mechanism to capture global time/frequency correlations among these T-F tokens. Furthermore, we underscore the substantial impact of incorporating phase information into the input of the SignalFormer model. We evaluated the proposed method on two public datasets under Gaussian white noise and co-frequency signal interference conditions, The SignalFormer model achieved impressive identification accuracy of 97.57% and 98.03% for coarse-grained identification tasks, and 97.48% and 98.16% for fine-grained identification tasks. Furthermore, we introduced a class-incremental learning evaluation to demonstrate SignalFormer's competence in handling previously unseen categories of drone signals. The above results collectively demonstrate that the proposed method is a promising solution for supporting the ADI task in reliable ways.

A hybrid transformer–sequencer approach for age and gender classification from in-wild facial images

A hybrid transformer–sequencer approach for age and gender classification from in-wild facial images

VLP2MSA: Expanding vision-language pre-training to multimodal sentiment analysis

Large-scale vision-and-language representation learning has improved performance on various joint vision-language downstream tasks. In this work, our objective is to extend it effectively to multimodal sentiment analysis tasks and address two urgent challenges in this field: (1) the low contribution of the visual modality (2) the design of an effective multimodal fusion architecture. To overcome the imbalance between the visual and textual modalities, we propose an inter-frame hybrid transformer, which extends the recent CLIP and Timesformer architectures. This module extracts spatio-temporal features from sparsely sampled video frames, not only focusing on facial expressions but also capturing body movement information, providing a more comprehensive visual representation compared to the traditional direct use of pre-extracted facial information. Additionally, we tackle the challenge of modality heterogeneity in the fusion architecture by introducing a new scheme that prompts and aligns the video and text information before fusing them. Specifically, We generate discriminative text prompts based on the video content information to enhance the text representation and align the unimodal video-text features using a video-text contrastive loss. Our proposed end-to-end trainable model demonstrates state-of-the-art performance on three widely-used datasets using only two modalities: MOSI, MOSEI, and CH-SIMS. These experimental results validate the effectiveness of our approach in improving multimodal sentiment analysis tasks.

Hybrid transformation with 1’st order statistics for medical image analysis and classification

Skin cancer, one of the most critical forms of cancer, required early detection and documentation for efficient treatment, especially as certain types are fatal. In this study, an artificial neural network (ANN) was utilized to discover and index diverse melanomas using the ISIC 2018 dataset. The pre-processing phase is stringent as it insulates the cancerous fraction of a skin image, involving removing, trimming, thinning, and normalizing. In this phase, unwanted hair pieces on the image are eliminated in this phase. Feature extraction from the clipped image is achieved using a discrete cosine transform (DCT) and a gradient transform to transform it into frequency domain coefficients. Statistical feature extraction is used to reduce the amount of data required for ANN training. A dataset from ISIC 2018 that consists of seven different images from dermoscopic procedures for classification purposes is used in the empirical investigation. An accuracy of 85.44% for DCT in the sub-bands and 76.07% for the sub-band gradient transform was achieved by the applied ANN. The hybrid system's mean squared error (MSE) was discovered to be 3.52×10-4. The work highlights the potential of ANN in the early detection of skin cancer, supporting more efficient treatment and preventing advanced cases.

Hybrid transformation with first order statistics for medical image analysis and classification

Skin cancer, one of the most critical forms of cancer, required early detection and documentation for efficient treatment, especially as certain types are fatal. In this study, an artificial neural network (ANN) was utilized to discover and index diverse melanomas using the ISIC 2018 dataset. The pre-processing phase is stringent as it insulates the cancerous fraction of a skin image, involving removing, trimming, thinning, and normalizing. In this phase, unwanted hair pieces on the image are eliminated in this phase. Feature extraction from the clipped image is achieved using a discrete cosine transform (DCT) and a gradient transform to transform it into frequency-domain coefficients. Statistical feature extraction is used to reduce the amount of data required for ANN training. A dataset from ISIC 2018 that consists of seven different images from dermoscopic procedures for classification purposes is used in the empirical investigation. An accuracy of 85.44% for DCT in the sub-bands and 76.07% for the sub-band gradient transform was achieved by the applied ANN. The hybrid system's mean squared error (MSE) was discovered to be 3.52×10-4. The work highlights the potential of ANN in the early detection of skin cancer, supporting more efficient treatment and preventing advanced cases.

Dual attention transformer network for hyperspectral image classification

Hyperspectral image classification (HSIC) has been a significant topic in the field of remote sensing in the past few years. Convolutional neural networks have shown promising performance in HSIC applications due to their strong local feature extraction ability. However, they struggle to extract global information from HSIs, thereby resulting in classification performance limitations. Recently, vision transformers have been used to solve HSIC problems, and its advantage is to adopt the multi-head self-attention mechanism to explore global dependencies. Nevertheless, the extracted features using MHSA usually exhibit over-dispersion due to the abundance of band information hidden in HSIs. In this work, we propose a novel method, called dual attention transformer network (DATN), for HSIC problems. It consists of two types of modules, namely the spatial–spectral hybrid transformer (SSHT) module and the spectral local-conv block (SLCB) module. Specifically, the SSHT module aims to utilize the MHSA to capture spatial and spectral feature information. Therefore, it can effectively utilize global spatial–spectral features and embed the local spatial information, simultaneously. Besides, we design a SLCB module to extract the local spectral information of HSIs effectively. Then the SSHT and SLCB modules are integrated into an end-to-end framework. Finally, the global and local spatial–spectral features extracted from this framework are input into the fully connected layer, and then classification results of HSIs are obtained. A series of experiments on three HSI datasets have demonstrated that our DATN approach outperforms several state-of-the-art HSIC approaches.

AstroYOLO: A hybrid CNN–Transformer deep-learning object-detection model for blue horizontal-branch stars

Abstract Blue horizontal-branch stars (BHBs) are ideal tracers for studying the Milky Way (MW) due to their bright and nearly constant magnitude. However, an incomplete screen of BHBs from a survey would result in bias of estimation of the structure or mass of the MW. With surveys of large sky telescopes like the Sloan Digital Sky Survey (SDSS), it is possible to obtain a complete sample. Thus, detecting BHBs from massive photometric images quickly and effectually is necessary. The current acquisition methods of BHBs are mainly based on manual or semi-automatic modes. Therefore, novel approaches are required to replace manual or traditional machine-learning detection. The mainstream deep-learning-based object-detection methods are often vanilla convolutional neural networks whose ability to extract global features is limited by the receptive field of the convolution operator. Recently, a new Transformer-based method has benefited from the global receptive field advantage brought by the self-attention mechanism, exceeded the vanilla convolution model in many tasks, and achieved excellent results. Therefore, this paper proposes a hybrid convolution and Transformer model called AstroYOLO to take advantage of the convolution in local feature representation and Transformer’s easier discovery of long-distance feature dependences. We conduct a comparative experiment on the 4799 SDSS DR16 photometric image dataset. The experimental results show that our model achieves 99.25% AP@50, 93.79% AP@75, and 64.45% AP@95 on the test dataset, outperforming the YOLOv3 and YOLOv4 object-detection models. In addition, we test on larger cutout images based on the same resolution. Our model can reach 99.02% AP@50, 92.00% AP@75, and 61.96% AP@95 respectively, still better than YOLOv3 and YOLOv4. These results also suggest that an appropriate size for cutout images is necessary for the performance and computation of object detection. Compared with the previous models, our model has achieved satisfactory object-detection results and can effectively improve the accuracy of BHB detection.

The transmembrane protein LbRSG from the recretohalophyte Limonium bicolor enhances salt gland development and salt tolerance.

Salt glands are the unique epidermal structures present in recretohalophytes, plants that actively excrete excess Na+ by salt secretory structures to avoid salt damage. Here, we describe a transmembrane protein that localizes to the plasma membrane of the recretohalophyte Limonium bicolor. As virus-induced gene silencing of the corresponding gene LbRSG in L. bicolor decreased the number of salt glands, we named the gene Reduced Salt Gland. We detected LbRSG transcripts in salt glands by in situ hybridization and transient transformation. Overexpression and silencing of LbRSG in L. bicolor pointed to a positive role in salt gland development and salt secretion by interacting with Lb3G16832. Heterologous LbRSG expression in Arabidopsis enhanced salt tolerance during germination and the seedling stage by alleviating NaCl-induced ion stress and osmotic stress after replacing or deleting the (highly) negatively charged region of extramembranous loop. After screened by immunoprecipitation-mass spectrometry and verified using yeast two-hybrid, PGK1 and BGLU18 were proposed to interact with LbRSG to strengthen salt tolerance. Therefore, we identified (highly) negatively charged regions in the extramembrane loop that may play an essential role in salt tolerance, offering hints about LbRSG function and its potential to confer salt resistance.

TransQA: deep hybrid transformer network for measurement-guided volumetric dose prediction of pre-treatment patient-specific quality assurance

Objective. Performing pre-treatment patient-specific quality assurance (prePSQA) is considered an essential, time-consuming, and resource-intensive task for volumetric modulated arc radiotherapy (VMAT) which confirms the dose accuracy and ensure patient safety. Most current machine learning and deep learning approaches stack excessive convolutional/pooling operations (CPs) to predict prePSQA with two-dimensional or one-dimensional information input. However, these models generally present limitations in explicitly modeling long-range dependency for volumetric dose prediction due to the loss of spatial dose features and the inherent locality of CPs. The purpose of this work is to construct a deep hybrid network by combining the self-attention mechanism-based Transformer with modified U-Net for predicting measurement-guided volumetric dose (MDose) of prePSQA. Approach. The enrolled 307 cancer patients underwent VMAT were randomly divided into 246 and 61 cases for training and testing the model. The input data included computed tomography images, radiotherapy dose images exported from the treatment planning system, as well as the MDose distribution from the verification system. The output was the predicted high-quality voxel-wise prePSQA dose distribution. Main results: qualitative and quantitative experimental results show that the proposed prediction method could achieve comparable or better performance on MDose prediction over other approaches in terms of spatial dose distribution, dose–volume histogram metrics, gamma passing rates, mean absolute error, root mean square error, and structural similarity. Significance. The preliminary results on multiple cancer sites show that our approach can be taken as a clinical guidance tool and help medical physicists to reduce the measurement work of prePSQA.

High-Resolution ISAR Imaging Method for Maneuvering Targets Based on a Hybrid Transformer

High-Resolution ISAR Imaging Method for Maneuvering Targets Based on a Hybrid Transformer

Towards a new triple integral transform (Laplace–ARA–Sumudu) with applications

The main objective of this work is to introduce a novel generalization of double transformations called the triple Laplace–ARA–Sumudu transform (TLARAST). This hybrid transformation extends the concepts of double Laplace–Sumudu, double Laplace–ARA and double ARA–Sumudu transforms into a triple hybrid transform. The article provides the definition of TLARAST and investigates its fundamental properties, including existence, inverses and related theorems. Furthermore, new results concerning TLARAST for partial derivatives and the theorem of multi-convolution are introduced and discussed. The practicality and efficacy of TLARAST are demonstrated by applying it to solve various types of partial differential equations with significant applications in physics and other scientific fields, such as the heat equation, Laplace equation, Poisson equation and wave equation. The solutions are illustrated through figures created using Mathematica software. Overall, this study underscores the usefulness and efficiency of TLARAST in solving partial differential equations involving multiple variables.

Improving Structural MRI Preprocessing with Hybrid Transformer GANs.

Magnetic resonance imaging (MRI) is a technique that is widely used in practice to evaluate any pathologies in the human body. One of the areas of interest is the human brain. Naturally, MR images are low-resolution and contain noise due to signal interference, the patient's body's radio-frequency emissions and smaller Tesla coil counts in the machinery. There is a need to solve this problem, as MR tomographs that have the capability of capturing high-resolution images are extremely expensive and the length of the procedure to capture such images increases by the order of magnitude. Vision transformers have lately shown state-of-the-art results in super-resolution tasks; therefore, we decided to evaluate whether we can employ them for structural MRI super-resolution tasks. A literature review showed that similar methods do not focus on perceptual image quality because upscaled images are often blurry and are subjectively of poor quality. Knowing this, we propose a methodology called HR-MRI-GAN, which is a hybrid transformer generative adversarial network capable of increasing resolution and removing noise from 2D T1w MRI slice images. Experiments show that our method quantitatively outperforms other SOTA methods in terms of perceptual image quality and is capable of subjectively generalizing to unseen data. During the experiments, we additionally identified that the visual saliency-induced index metric is not applicable to MRI perceptual quality assessment and that general-purpose denoising networks are effective when removing noise from MR images.

CNN and Transformer Fusion for Remote Sensing Image Semantic Segmentation

Semantic segmentation of remote sensing images has been widely used in environmental protection, geological disaster discovery, and natural resource assessment. With the rapid development of deep learning, convolutional neural networks (CNNs) have dominated semantic segmentation, relying on their powerful local information extraction capabilities. Due to the locality of convolution operation, it can be challenging to obtain global context information directly. However, Transformer has excellent potential in global information modeling. This paper proposes a new hybrid convolutional and Transformer semantic segmentation model called CTFuse, which uses a multi-scale convolutional attention module in the convolutional part. CTFuse is a serial structure composed of a CNN and a Transformer. It first uses convolution to extract small-size target information and then uses Transformer to embed large-size ground target information. Subsequently, we propose a spatial and channel attention module in convolution to enhance the representation ability for global information and local features. In addition, we also propose a spatial and channel attention module in Transformer to improve the ability to capture detailed information. Finally, compared to other models used in the experiments, our CTFuse achieves state-of-the-art results on the International Society of Photogrammetry and Remote Sensing (ISPRS) Vaihingen and ISPRS Potsdam datasets.

Identification of haploid and diploid maize seeds using hybrid transformer model

Identification of haploid and diploid maize seeds using hybrid transformer model

In situ fabrication of Ag decorated porous ZnO photocatalyst via inorganic–organic hybrid transformation for degradation of organic pollutant and bacterial inactivation

In this study, in situ silver (Ag) - porous ZnO photocatalysts were synthesized via solvothermal and post-annealing treatment. The formation of the porous ZnO structure due to the removal of organic moieties from the inorganic–organic hybrids Ag-ZnS(en)0.5 during the annealing process. The optimal Ag–ZnO photocatalyst showed excellent photocatalytic degradation activity, with 95.5% orange II dye and 97.2% bisphenol A (BPA) degradation under visible light conditions. Additionally, the photocatalytic inactivation of Escherichia coli (E. coli) and Staphylococcus aureus (S. aureus) led to a 97% inactivation rate after 2 h under dark conditions. Trapping experiments suggest that the superoxide anion (▪O2−) radicals are the main active species to degrade the organic dye. The improved photocatalytic dye degradation activity and inactivation of bacteria were attributed to the synergistic effect of Ag and porous ZnO structure, increased surface area, and efficiently separated the photoexcited charge carriers. This work could provide an effective strategy for the synthesis of porous structures toward organic pollutant degradation and bacterial inactivation in wastewater.

Magnetic Flux Bias Compensation Based on Voltage Injection Method With an Auxiliary DC/DC Converter

DC bias will make the transformer core saturated, causing local overheating and insulation materials aging. Therefore, it is great of significance to study the suppression of DC bias in transformers. In this paper, a magnetic compensation scheme of DC bias based on a hybrid transformer (HT) integrated an auxiliary DC/DC converter is proposed. Firstly, a compensation mechanism is established by designing a novel HT topology structure, which lays the magnetic circuit foundation of DC bias compensation. Secondly, the mathematical model of transformer DC bias is established, and the DC compensation amount of HT is derived. A transient simulation model of three-phase HT is established by the finite element model, and the situation of the DC bias before and after the compensation is analyzed. Finally, the platform of the DC bias compensation scheme is built. The full compensation experiment of DC bias is simulated by two parallel HT. The experimental results show the DC bias compensation effect is obvious, the distortion of excitation current, temperature rise and flux density distribution drop obviously, which verifies the correctness of the scheme proposed in this paper. It provides a reference for solving the problem of DC bias.