Lightweight Transformer Research Articles

IR-ST: A Lightweight Transformer Network for Human Fall Detection Based on FMCW Radar

IR-ST: A Lightweight Transformer Network for Human Fall Detection Based on FMCW Radar

Weed mapping in multispectral drone imagery using lightweight vision transformers

In precision agriculture, non-invasive remote sensing can be used to observe crops and weeds in visible and non-visible spectra. This paper proposes a novel approach for weed mapping using lightweight Vision Transformers. The method uses a lightweight Transformer architecture to process high-resolution aerial images obtained from drones and performs semantic segmentation to distinguish between crops and weeds. The method also employs specific architectural designs to enable transfer learning from RGB weights in a multispectral setting. For this purpose, the WeedMap dataset, acquired by drones equipped with multispectral cameras, was used. The experimental results demonstrate the effectiveness of the proposed method, exceeding the state-of-the-art. Our approach also enables more efficient mapping, allowing farmers to quickly and easily identify infested areas and prioritize their control efforts. These results encourage using drones as versatile computer vision flying devices for herbicide management, thereby improving crop yields. The code is available at https://github.com/pasqualedem/LWViTs-for-weedmapping.

MobileRaT: A Lightweight Radio Transformer Method for Automatic Modulation Classification in Drone Communication Systems

Nowadays, automatic modulation classification (AMC) has become a key component of next-generation drone communication systems, which are crucial for improving communication efficiency in non-cooperative environments. The contradiction between the accuracy and efficiency of current methods hinders the practical application of AMC in drone communication systems. In this paper, we propose a real-time AMC method based on the lightweight mobile radio transformer (MobileRaT). The constructed radio transformer is trained iteratively, accompanied by pruning redundant weights based on information entropy, so it can learn robust modulation knowledge from multimodal signal representations for the AMC task. To the best of our knowledge, this is the first attempt in which the pruning technique and a lightweight transformer model are integrated and applied to processing temporal signals, ensuring AMC accuracy while also improving its inference efficiency. Finally, the experimental results—by comparing MobileRaT with a series of state-of-the-art methods based on two public datasets—have verified its superiority. Two models, MobileRaT-A and MobileRaT-B, were used to process RadioML 2018.01A and RadioML 2016.10A to achieve average AMC accuracies of 65.9% and 62.3% and the highest AMC accuracies of 98.4% and 99.2% at +18 dB and +14 dB, respectively. Ablation studies were conducted to demonstrate the robustness of MobileRaT to hyper-parameters and signal representations. All the experimental results indicate the adaptability of MobileRaT to communication conditions and that MobileRaT can be deployed on the receivers of drones to achieve air-to-air and air-to-ground cognitive communication in less demanding communication scenarios.

TCNet: A Transformer–CNN Hybrid Network for Marine Aquaculture Mapping from VHSR Images

Precise delineation of marine aquaculture areas is vital for the monitoring and protection of marine resources. However, due to the coexistence of diverse marine aquaculture areas and complex marine environments, it is still difficult to accurately delineate mariculture areas from very high spatial resolution (VHSR) images. To solve such a problem, we built a novel Transformer–CNN hybrid Network, named TCNet, which combined the advantages of CNN for modeling local features and Transformer for capturing long-range dependencies. Specifically, the proposed TCNet first employed a CNN-based encoder to extract high-dimensional feature maps from input images. Then, a hierarchical lightweight Transformer module was proposed to extract the global semantic information. Finally, it employed a coarser-to-finer strategy to progressively recover and refine the classification results. The results demonstrate the effectiveness of TCNet in accurately delineating different types of mariculture areas, with an IoU value of 90.9%. Compared with other state-of-the-art CNN or Transformer-based methods, TCNet showed significant improvement both visually and quantitatively. Our methods make a significant contribution to the development of precision agricultural in coastal regions.

PSLT: A Light-Weight Vision Transformer With Ladder Self-Attention and Progressive Shift.

Vision Transformer (ViT) has shown great potential for various visual tasks due to its ability to model long-range dependency. However, ViT requires a large amount of computing resource to compute the global self-attention. In this work, we propose a ladder self-attention block with multiple branches and a progressive shift mechanism to develop a light-weight transformer backbone that requires less computing resources (e.g., a relatively small number of parameters and FLOPs), termed Progressive Shift Ladder Transformer (PSLT). First, the ladder self-attention block reduces the computational cost by modelling local self-attention in each branch. In the meanwhile, the progressive shift mechanism is proposed to enlarge the receptive field in the ladder self-attention block by modelling diverse local self-attention for each branch and interacting among these branches. Second, the input feature of the ladder self-attention block is split equally along the channel dimension for each branch, which considerably reduces the computational cost in the ladder self-attention block (with nearly [Formula: see text] the amount of parameters and FLOPs), and the outputs of these branches are then collaborated by a pixel-adaptive fusion. Therefore, the ladder self-attention block with a relatively small number of parameters and FLOPs is capable of modelling long-range interactions. Based on the ladder self-attention block, PSLT performs well on several vision tasks, including image classification, objection detection and person re-identification. On the ImageNet-1 k dataset, PSLT achieves a top-1 accuracy of 79.9% with 9.2 M parameters and 1.9 G FLOPs, which is comparable to several existing models with more than 20 M parameters and 4 G FLOPs. Code is available at https://isee-ai.cn/wugaojie/PSLT.html.

Lightweight design method of orthotropic steel bridge deck with U-ribs based on multi-objective optimization

This paper proposed a lightweight design method for orthotropic steel bridge deck U-ribs. Firstly, the U-rib is analyzed in static structural properties. An analytic hierarchy model was constructed to maximize the stiffness for multi-objective topology optimization. Secondly, a parametric reconfiguration model of the topologically optimized U-rib was established. Finally, static tests are carried out and the performance of the 3D printing scale model was tested. The test results show that the method is effective. Weight of the original model with U-rib is 99.8 kg, and the weight of its optimized model is 76.8 kg. The method achieved a weight reduction of 23% for the U-ribs, and the stiffness still met the permitted conditions. The orthotropic steel bridge deck composed of the optimized U-ribs effectively reduced the weight by 150 kg on the basis of basically unchanged stiffness. The method provides an important reference for the lightweight transformation of assembled bridge structures.

Predicting the Refrigerant Amounts Across Air Conditioners With a Domain Adaptive Lightweight Transformer

Air conditioner consumers expect their products to work well without any problem. Consumers encounter problems if the amount of refrigerant in the air conditioner is insufficient. Therefore, we propose a novel deep learning approach that predicts the amount of refrigerant in advance. Our approach differs from others, as it is not limited to specific types of air conditioners and is applicable to all types. We propose a domain adaptive Transformer (DAT) that determines the amount of refrigerant using a Transformer encoder and domain adversarial training of neural networks. The proposed DAT reduces the model parameters by using a modified lightweight Transformer encoder and adds a domain classifier to build a universal model that is not limited to a specific type of air conditioner. We conduct the experiments by collecting data on the amount of refrigerant operating in the real fields. As a result, the proposed DAT achieves higher prediction performance. It is also significant that the results predicted by DAT are more accurate than those predicted by experts dealing with air conditioner refrigerant problems. As a result, DAT-based air conditioners are commercialized in real life. Hence, consumers can prevent the shortage of refrigerant without experts.

A transformer model blended with CNN and denoising autoencoder for inter-patient ECG arrhythmia classification

Researchers have proposed numerous novel features and models under the intra-patient paradigm. However, their performance suffers when considering the inter-patient paradigm. While some state-of-the-art results have been reported in recent years under the inter-patient paradigm, many of them deviate from the standard test protocol. The performance of minority classes remains unsatisfactory for practical applications under strict test protocols. This paper presents a novel framework based on a lightweight Transformer combined with CNN and a denoising autoencoder, which enhances the performance of minority classes under the standard test protocol. The proposed model includes a new seq2seq network that extracts local features from a single heartbeat using CNN or a denoising encoder, and attends to global features from neighboring heartbeats based on a lightweight Transformer encoder. In particular, we pretrained the autoencoder on the MIT-BIH dataset and an additional dataset, considering several transfer modes for feature representation. We organized multiple continuous heartbeats into a vector sequence, where each heartbeat incorporates information from its neighbors to improve feature representation. The model evaluation was conducted using the MIT-BIH inter-patient dataset, following the AAMI standard. The Transformer with CNN embedding achieved a total accuracy of 97.66% on the test set, while the Transformer with pretrained denoising autoencoder achieved a total accuracy of 97.93%. These results demonstrate the promising performance of our models for imbalanced inter-patient ECG classification under the standard test protocol.

Faster and Better: A Lightweight Transformer Network for Remote Sensing Scene Classification

Remote sensing (RS) scene classification has received considerable attention due to its wide applications in the RS community. Many methods based on convolutional neural networks (CNNs) have been proposed to classify complex RS scenes, but they cannot fully capture the context in RS images because of the lack of long-range dependencies (the dependency relationship between two distant elements). Recently, some researchers fine-tuned the large pretrained vision transformer (ViT) on small RS datasets to extract long-range dependencies effectively in RS scenes. However, it usually takes more time to fine-tune the ViT on account of high computational complexity. The lack of good local feature representation in the ViT limits classification performance improvement. To this end, we propose a lightweight transformer network (LTNet) for RS scene classification. First, a multi-level group convolution (MLGC) module is presented. It enriches the diversity of local features and requires a lower computational cost by co-representing multi-level and multi-group features in a single module. Then, based on the MLGC module, a lightweight transformer block, LightFormer, was designed to capture global dependencies with fewer computing resources. Finally, the LTNet was built using the MLGC and LightFormer. The experiments of fine-tuning the LTNet on four RS scene classification datasets demonstrate that the proposed network achieves a competitive classification performance under less training time.

BATFormer: Towards Boundary-Aware Lightweight Transformer for Efficient Medical Image Segmentation.

Transformers, born to remedy the inadequate receptive fields of CNNs, have drawn explosive attention recently. However, the daunting computational complexity of global representation learning, together with rigid window partitioning, hinders their deployment in medical image segmentation. This work aims to address the above two issues in transformers for better medical image segmentation. We propose a boundary-aware lightweight transformer (BATFormer) that can build cross-scale global interaction with lower computational complexity and generate windows flexibly under the guidance of entropy. Specifically, to fully explore the benefits of transformers in long-range dependency establishment, a cross-scale global transformer (CGT) module is introduced to jointly utilize multiple small-scale feature maps for richer global features with lower computational complexity. Given the importance of shape modeling in medical image segmentation, a boundary-aware local transformer (BLT) module is constructed. Different from rigid window partitioning in vanilla transformers which would produce boundary distortion, BLT adopts an adaptive window partitioning scheme under the guidance of entropy for both computational complexity reduction and shape preservation. BATFormer achieves the best performance in Dice of 92.84 %, 91.97 %, 90.26 %, and 96.30 % for the average, right ventricle, myocardium, and left ventricle respectively on the ACDC dataset and the best performance in Dice, IoU, and ACC of 90.76 %, 84.64 %, and 96.76 % respectively on the ISIC 2018 dataset. More importantly, BATFormer requires the least amount of model parameters and the lowest computational complexity compared to the state-of-the-art approaches. Our results demonstrate the necessity of developing customized transformers for efficient and better medical image segmentation. We believe the design of BATFormer is inspiring and extendable to other applications/frameworks.

Lightweight Transformer for Multi-Modal Object Detection (Student Abstract)

It has become a common practice for many perceptual systems to integrate information from multiple sensors to improve the accuracy of object detection. For example, autonomous vehicles use visible light, and infrared (IR) information to ensure that the car can cope with complex weather conditions. However, the accuracy of the algorithm is usually a trade-off between the computational complexity and memory consumption. In this study, we evaluate the performance and complexity of different fusion operators in multi-modal object detection tasks. On top of that, a Poolformer-based fusion operator (PoolFuser) is proposed to enhance the accuracy of detecting targets without compromising the efficiency of the detection framework.

Automatic disease detection in endoscopy with light weight transformer

Automatic disease detection in endoscopy with light weight transformer

Effectively fusing clinical knowledge and AI knowledge for reliable lung nodule diagnosis

Clinicians typically use semantic features to judge the malignant status of nodules, while artificial intelligence systems (AI) tend to extract unknown features to diagnose nodules. The former relies on clinical knowledge, while the latter explores AI knowledge. Although many studies indicate that fusing clinical and AI knowledge can help computer-aided diagnosis (CAD) systems improve diagnostic accuracy and gain clinician approval, how to effectively fuse them is still an open question. This paper proposes a simple and effective pipeline (abbreviated as CKAK), which fuses clinical and AI knowledge at both feature and decision levels for accurate lung nodule malignancy classification and semantic attributes characterization. The feature-level fusion can retain rich information in high-dimensional features and improve the model’s accuracy; the decision-level fusion can provide some interpretability for the model’s decision-making process, which is expected in clinical applications. Specifically, the proposed CKAK consists of two sequential stages: (i) the initial prediction stage (IPS); and (ii) the prediction refine stage (PRS). The IPS predicts eight radiologist-interpreted semantic attributes and an initial malignancy diagnosis in parallel. Then, these results are fed to the subsequent PRS to refine the diagnosis further by fully fusing them at feature and decision levels. Besides, to enhance the ability of feature learning, we propose a novel scale-aware feature extraction block (SAFE). It integrates multi-scale contextual features with a lightweight Transformer rather than adding or concatenating them roughly. Extensive experiments at the LIDC-IDRI data set show that the proposed CKAK can achieve superior benign-malignant classification accuracy with minor radiologist-interpreted semantic scores error, meeting the need for a reliable CAD system.

MCANet: Hierarchical cross-fusion lightweight transformer based on multi-ConvHead attention for object detection

The visual Transformer model based on self-attention has achieved better performance than convolutional neural networks in object detection tasks. However, existing visual Transformer models are typically heavy-weight to extract global features. In contrast, CNNs can extract features with fewer parameters and computational costs. To combine the advantages of convolutional processing at the local level with the advantages of the Transformer's global interaction, this paper proposes MCANet, a Hierarchical Cross-Fusion Lightweight Transformer Based on Multi-ConvHead Attention for Object Detection. To bi-directionally fuse local and global features, MCANet adds two improved transformers (MCA-Former) for global interaction and two novel feature fusion modules MCA-CSP. MCA-Former uses a novel self-attention computation method named Multi-ConvHead Attention(MCA) based on multi-scale depth-separable convolution, which reduces the computational cost by 2/3. Meanwhile, the number of model parameters is reduced to 9.49 M by using channel segmentation and multi-layer cross- fusion strategies. On the Pascal VOC and COCO datasets, the proposed model outperforms YOLOv4-Tiny in terms of AP by 2.43% and 1.8%, respectively. Additionally, MCANet is also superior to many latest lightweight object detection models. Results of various ablation experiments also verify the effectiveness of the proposed method.

Single‐image super‐resolution using lightweight transformer‐convolutional neural network hybrid model

AbstractWith constant advances in deep learning methods as applied to image processing, deep convolutional neural networks (CNNs) have been widely explored in single‐image super‐resolution (SISR) problems and have attained significant success. These CNN‐based methods cannot fully use the internal and external information of the image. The authors add a lightweight Transformer structure to capture this information. Specifically, the authors apply a dense block structure and residual connection to build a residual dense convolution block (RDCB) that reduces the parameters somewhat and extracts shallow features. The lightweight transformer block (LTB) further extracts features and learns the texture details between the patches through the self‐attention mechanism. The LTB comprises an efficient multi‐head transformer (EMT) with small graphics processing unit (GPU) memory footprint, and benefits from feature preprocessing by multi‐head attention (MA), reduction, and expansion. The EMT significantly reduces the use of GPU resources. In addition, a detail‐purifying attention block (DAB) is proposed to explore the context information in the high‐resolution (HR) space to recover more details. Extensive evaluations of four benchmark datasets demonstrate the effectiveness of the authors’ proposed model in terms of quantitative metrics and visual effects. The proposed EMT only uses about 40% as much GPU memory as other methods, with better performance.

Automatic pixel-level detection of tire defects based on a lightweight Transformer architecture

Efficient and accurate visual inspection of products is of great significance for intelligent manufacturing. In this paper, a visual inspection framework based on a lightweight Transformer is proposed for the pixel-level inspection of tire defects. A dual-path-Transformer feature encoder was proposed based on a hybrid architecture of convolutional neural network and Transformer, which is used to learn the local and global relationship of defect features. Moreover, multi-scale fusion Transformer (MFT) and spatial cross Transformer (SCT) were proposed, based on which a feature decoder was built. In the decoder, the MFT provides valuable spatial information for SCT, such that different levels of feature maps can refine the dependency of pixels through SCT. The proposed method is tested on the tire radiographic image dataset. Experimental results show that the proposed method reaches a detection accuracy of 98.57% and mIoU of 85.56%. Moreover, the method achieves the balance between accuracy and efficiency with satisfactory characterization ability for the defect geometric shape, and provides theoretical support for the future industrial deployment of Transformer.

LCANet: A Lightweight Context-Aware Network for Bladder Tumor Segmentation in MRI Images

Accurate segmentation of the lesion area from MRI images is essential for diagnosing bladder cancer. However, the precise segmentation of bladder tumors remains a massive challenge due to their similar intensity distributions, various tumor morphologies, and blurred boundaries. While some seminal studies, such as those using CNNs combined with transformer segmentation methods, have made significant progress, (1) how to reduce the computational complexity of the self-attention mechanism in the transformer while maintaining performance and (2) how to build a better global feature fusion process to improve segmentation performance still require further exploration. Considering the complexity of bladder MRI images, we developed a lightweight context-aware network (LCANet) to automatically segment bladder lesions from MRI images. Specifically, the local detail encoder generates local-level details of the lesion, the lightweight transformer encoder models the global-level features with different resolutions, the pyramid scene parsing module extracts high-level and multiscale semantic features, and the decoder provides high-resolution segmentation results by fusing local-level details with global-level cues at the channel level. A series of empirical studies on T2-weighted MRI images from 86 patients show that LCANet achieves an overall Jaccard index of 89.39%, a Dice similarity coefficient of 94.08%, and a Class pixel accuracy of 94.10%. These advantages show that our method is an efficient tool that can assist in reducing the heavy workload of radiologists.

A Lightweight Transformer With Strong Robustness Application in Portable Bearing Fault Diagnosis

Although Transformers have achieved excellent results in various tasks in industrial scenes, owing to the environmental noise and cost limitation, the fault diagnosis approaches based on Transformer are facing two serious challenges, i.e. robustness and lightweight. With the original intention of promoting the transformation of Transformer from theoretical design to practical engineering application, we designed a lightweight framework with strong robustness, named XACNN, to meet these challenges. Through the adjustment of FFT, the effectiveness of the pre-processed signal is improved to enhance enhances the robustness to noise, and the goal of lightweight is achieved (FLOP: 0.136 M, Params: 7.663 k) via the optimization of self-attention. To demonstrate the effectiveness, the performance of XACNN in various noise environments was tested on a self-made dataset (average accuracy: 88.525%), which is superior to other methods. Simultaneously, we deployed XACNN on a smart phone as a portable fault diagnosis device, which verified the feasibility. As the first attempt to build a portable mobile detection device based on the deep learning model, this paper provides a new detection scheme for the relevant practitioners of mechanical fault diagnosis.

Mixture lightweight transformer for scene understanding

In adapting Transformer from language to computer vision, the major obstacles are the high computational complexity and large model size of Transformer blocks, derived from the great quantity of visual tokens and high resolutions of input entities. To address these challenges, this paper presents a mixture lightweight Transformer (MLT) backbone for image understanding, where each Transformer block, called SH-Transformer, adopts Single-Head Self-Attention (SHSA) and Convolutional Inception Module (CIM). Unlike previous Transformers that compute Multi-Head Self-Attention (MHSA), SHSA limits the representation of input token into a single head, resulting in a low-dimensional embedding that greatly reduces computational complexity. In spite of adding a small number of model parameters, SHSA greatly minimizes the number of input tokens. As complementary of SHSA that only investigates global interactions, CIM is designed to explore multi-scale local information using lightweight convolutions in a multi-path parallel manner. Experimental results reveal that MLT yields competitive or state-of-the-art results with respect to recent transformers while keeping smaller model size and lower computational costs for different visual tasks, including image classification, semantic segmentation and object detection. Particularly, the proposed method has 4.2% improvement to the tiny version of Pyramid Vision Transformer on image classification of top-1 accuracy on ImageNet-1K.

Memristor-Based Light-Weight Transformer Circuit Implementation for Speech Recognizing

Transformer network (TN) is a promising model widely used for natural language processing (NLP), computer vision (CV), and audio processing (AP). However, the large number of multiples and additions (multi-adds) in TN leads to a massive number of parameters in the TN model, causing an increasing number of hardware resource overhead and thus making it difficult to deploy the TN model to edge devices. To alleviate this problem, this paper proposes a circuit implementation scheme for a memristor-based lightweight transformer network (MLTN). In addition, the speech recognition task based on the speech command dataset is implemented through the modular hardware design approach, which provides an effective lightweight hardware implementation scheme for the edge devices. Further, the retention, noise immunity, variability, and stability of the designed circuit are analyzed by simulation experiments with LTSPICE. The results show that the proposed MLTN circuit not only has a lighter network model but also has significant recognition accuracy with different hardware scales. Specifically, MLTN-40 can achieve 96.18% simulation accuracy, while the energy overhead is only 3.8% of the ARM framework, which is commonly used for edge devices, and 39.9% of the traditional server (GPU), making it a promising solution for implementing speech recognition on edge devices.