Transformer Architecture Research Articles

Medical Image Segmentation Review: The Success of U-Net.

Automatic medical image segmentation is a crucial topic in the medical domain and successively a critical counterpart in the computer-aided diagnosis paradigm. U-Net is the most widespread image segmentation architecture due to its flexibility, optimized modular design, and success in all medical image modalities. Over the years, the U-Net model has received tremendous attention from academic and industrial researchers who have extended it to address the scale and complexity created by medical tasks. These extensions are commonly related to enhancing the U-Net's backbone, bottleneck, or skip connections, or including representation learning, or combining it with a Transformer architecture, or even addressing probabilistic prediction of the segmentation map. Having a compendium of different previously proposed U-Net variants makes it easier for machine learning researchers to identify relevant research questions and understand the challenges of the biological tasks that challenge the model. In this work, we discuss the practical aspects of the U-Net model and organize each variant model into a taxonomy. Moreover, to measure the performance of these strategies in a clinical application, we propose fair evaluations of some unique and famous designs on well-known datasets. Furthermore, we provide a comprehensive implementation library with trained models. In addition, for ease of future studies, we created an online list of U-Net papers with their possible official implementation.

OOD-CV-v2 : An Extended Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images.

Enhancing the robustness of vision algorithms in real-world scenarios is challenging. One reason is that existing robustness benchmarks are limited, as they either rely on synthetic data or ignore the effects of individual nuisance factors. We introduce OOD-CV-v2, a benchmark dataset that includes out-of-distribution examples of 10 object categories in terms of pose, shape, texture, context and the weather conditions, and enables benchmarking of models for image classification, object detection, and 3D pose estimation. In addition to this novel dataset, we contribute extensive experiments using popular baseline methods, which reveal that: 1) Some nuisance factors have a much stronger negative effect on the performance compared to others, also depending on the vision task. 2) Current approaches to enhance robustness have only marginal effects, and can even reduce robustness. 3) We do not observe significant differences between convolutional and transformer architectures. We believe our dataset provides a rich test bed to study robustness and will help push forward research in this area.

Dual-Grained Lightweight Strategy.

Removing redundant parameters and computations before the model training has attracted a great interest as it can effectively reduce the storage space of the model, speed up the training and inference of the model, and save energy consumption during the running of the model. In addition, the simplification of deep neural network models can enable high-performance network models to be deployed to resource-constrained edge devices, thus promoting the development of the intelligent world. However, current pruning at initialization methods exhibit poor performance at extreme sparsity. In order to improve the performance of the model under extreme sparsity, this paper proposes a dual-grained lightweight strategy-TEDEPR. This is the first time that TEDEPR has used tensor theory in the pruning at initialization method to optimize the structure of a sparse sub-network model and improve its performance. Specifically, first, at the coarse-grained level, we represent the weight matrix or weight tensor of the model as a low-rank tensor decomposition form and use multi-step chain operations to enhance the feature extraction capability of the base module to construct a low-rank compact network model. Second, unimportant weights are pruned at a fine-grained level based on the trainability of the weights in the low-rank model before the training of the model, resulting in the final compressed model. To evaluate the superiority of TEDEPR, we conducted extensive experiments on MNIST, UCF11, CIFAR-10, CIFAR-100, Tiny-ImageNet and ImageNet datasets with LeNet, LSTM, VGGNet, ResNet and Transformer architectures, and compared with state-of-the-art methods. The experimental results show that TEDEPR has higher accuracy, faster training and inference, and less storage space than other pruning at initialization methods under extreme sparsity.

A Survey on Efficient Vision Transformers: Algorithms, Techniques, and Performance Benchmarking.

Vision Transformer (ViT) architectures are becoming increasingly popular and widely employed to tackle computer vision applications. Their main feature is the capacity to extract global information through the self-attention mechanism, outperforming earlier convolutional neural networks. However, ViT deployment and performance have grown steadily with their size, number of trainable parameters, and operations. Furthermore, self-attention's computational and memory cost quadratically increases with the image resolution. Generally speaking, it is challenging to employ these architectures in real-world applications due to many hardware and environmental restrictions, such as processing and computational capabilities. Therefore, this survey investigates the most efficient methodologies to ensure sub-optimal estimation performances. More in detail, four efficient categories will be analyzed: compact architecture, pruning, knowledge distillation, and quantization strategies. Moreover, a new metric called Efficient Error Rate has been introduced in order to normalize and compare models' features that affect hardware devices at inference time, such as the number of parameters, bits, FLOPs, and model size. Summarizing, this paper first mathematically defines the strategies used to make Vision Transformer efficient, describes and discusses state-of-the-art methodologies, and analyzes their performances over different application scenarios. Toward the end of this paper, we also discuss open challenges and promising research directions.

MT-DSNet: Mix-mask teacher-student strategies and dual dynamic selection plug-in module for fine-grained image recognition

The fine-grained image recognition (FGIR) task aims to classify and distinguish subtle differences between subcategories with visually similar appearances, such as bird species and the makes or models of vehicles. However, subtle interclass differences and significant intraclass variances lead to poor model recognition performance. To address these challenges, we developed a mixed-mask teacher–student cooperative training strategy. A mixed masked image is generated and embedded into a knowledge distillation network by replacing one image’s visible marker with another’s masked marker. Collaborative reinforcement between teachers and students is used to improve the recognition performance of the network. We chose the classic transformer architecture as a baseline to better explore the contextual relationships between features. Additionally, we suggest a dual dynamic selection plug-in for choosing features with discriminative capabilities in the spatial and channel dimensions and filter out irrelevant interference information to efficiently handle background and noise features in fine-grained images. The proposed feature suppression module is used to enhance the differences between different features, thereby motivating the network to mine more discriminative features. We validated our method using two datasets: CUB-200-2011 and Stanford Cars. The experimental results show that the proposed MT-DSNet can significantly improve the feature representation for FGIR tasks. Moreover, by applying it to different fine-grained networks, the FGIR accuracy can be improved without changing the original network structure. We hope that this work provides a promising approach for improving the feature representation of networks in the future.

Optical Flow as Spatial-Temporal Attention Learners.

Optical flow is an indispensable building block for various important computer vision tasks, including motion estimation, object tracking, and disparity measurement. To date, the dominant methods are CNN-based, leaving plenty of room for improvement. In this work, we propose TransFlow, a transformer architecture for optical flow estimation. Compared to dominant CNN-based methods, TransFlow demonstrates three advantages. First, it provides more accurate correlation and trustworthy matching in flow estimation by utilizing spatial self-attention and cross-attention mechanisms between adjacent frames to effectively capture global dependencies; Second, it recovers more compromised information (e.g., occlusion and motion blur) in flow estimation through long-range temporal association in dynamic scenes; Third, it introduces a concise self-learning paradigm, eliminating the need for complex and laborious multi-stage pre-training procedures. The versatility and superiority of TransFlow extend seamlessly to 3D scene motion, yielding competitive outcomes in 3D scene flow estimation. Our approach attains state-of-the-art results on benchmark datasets such as Sintel and KITTI-15, while also exhibiting exceptional performance on downstream tasks, including video object detection using the ImageNet VID dataset, video frame interpolation using the GoPro dataset, and video stabilization using the DeepStab dataset. We believe that the effectiveness of TransFlow positions it as a flexible baseline for both optical flow and scene flow estimation, offering promising avenues for future research and development.

Artionyms and machine learning: auto naming of the paintings

Image captioning is a question of great interest in a wide range of applications. In the art market there is a particularly acute shortage of specialized machine learning methods for accelerated and at the same time in-depth study of often too specific aspects of art. One of the main difficulties is caused by ambiguous names of art works, as well as clarifying (in practice, often complicating under- standing and perception) signatures of the authors to them. Although previous research has established that captioning of photos can be done with high efficacy, there is little published data about generation of captions for artistic paintings. In this research, we utilize a transformer architecture to generate an artionym for a given painting in author’s manner. We describe the model and report its performance on different art styles. We assess the model performance with an expert evaluation and image captioning metrics, and then discuss their capacity to analyze art-related names.

A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective.

Graph Neural Networks (GNNs) have gained momentum in graph representation learning and boosted the state of the art in a variety of areas, such as data mining (e.g., social network analysis and recommender systems), computer vision (e.g., object detection and point cloud learning), and natural language processing (e.g., relation extraction and sequence learning), to name a few. With the emergence of Transformers in natural language processing and computer vision, graph Transformers embed a graph structure into the Transformer architecture to overcome the limitations of local neighborhood aggregation while avoiding strict structural inductive biases. In this paper, we present a comprehensive review of GNNs and graph Transformers in computer vision from a task-oriented perspective. Specifically, we divide their applications in computer vision into five categories according to the modality of input data, i.e., 2D natural images, videos, 3D data, vision + language, and medical images. In each category, we further divide the applications according to a set of vision tasks. Such a task-oriented taxonomy allows us to examine how each task is tackled by different GNN-based approaches and how well these approaches perform. Based on the necessary preliminaries, we provide the definitions and challenges of the tasks, in-depth coverage of the representative approaches, as well as discussions regarding insights, limitations, and future directions.

PRITrans: A Transformer-Based Approach for the Prediction of the Effects of Missense Mutation on Protein–RNA Interactions

Protein–RNA interactions are essential to many cellular functions, and missense mutations in RNA-binding proteins can disrupt these interactions, often leading to disease. To address this, we developed PRITrans, a specialized computational method aimed at predicting the effects of missense mutations on protein–RNA interactions, which is vital for understanding disease mechanisms and advancing molecular biology research. PRITrans is a novel deep learning model designed to predict the effects of missense mutations on protein–RNA interactions, which employs a Transformer architecture enhanced with multiscale convolution modules for comprehensive feature extraction. Its primary innovation lies in integrating protein language model embeddings with a deep feature fusion strategy, effectively handling high-dimensional feature representations. By utilizing multi-layer self-attention mechanisms, PRITrans captures nuanced, high-level sequence information, while multiscale convolutions extract features across various depths, thereby enhancing predictive accuracy. Consequently, this architecture enables significant improvements in ΔΔG prediction compared to traditional approaches. We validated PRITrans using three different cross-validation strategies on two newly reconstructed mutation datasets, S315 and S630 (containing 315 forward and 315 reverse mutations). The results consistently demonstrated PRITrans’s strong performance on both datasets. PRITrans demonstrated strong predictive capability, achieving a Pearson correlation coefficient of 0.741 and a root mean square error (RMSE) of 1.168 kcal/mol on the S630 dataset. Moreover, its robust performance extended to independent test sets, achieving a Pearson correlation of 0.699 and an RMSE of 1.592 kcal/mol. These results underscore PRITrans’s potential as a powerful tool for protein-RNA interaction studies. Moreover, when tested against existing prediction methods on an independent dataset, PRITrans showed improved predictive accuracy and robustness.

Architecture of new generation research institutions under reconstruction conditions

In the article, the authors focus on key methods and approaches that ensure optimal organization of space for scientific and research work. This contributes to achieving versatility, flexibility and the ability to transform the internal environment. Modern compositional techniques for the architectural transformation of objects of the type under consideration and methods of their application at various levels of architectural formation and development have been studied. The main criteria for the architectural and spatial formation of new generation research institutions have been identified, thanks to which an architectural ensemble is formed, the concept of spatial organization, the creation of a unified compositional structure, the organization of recognizable architecture. The basic principles and features that form the nature of sustainability, visual perception, and create a spatial structure of perception have been identified.

Image enhancement with art design: a visual feature approach with a CNN-transformer fusion model

Graphic design, as a product of the burgeoning new media era, has seen its users’ requirements for images continuously evolve. However, external factors such as light and noise often cause graphic design images to become distorted during acquisition. To enhance the definition of these images, this paper introduces a novel image enhancement model based on visual features. Initially, a histogram equalization (HE) algorithm is applied to enhance the graphic design images. Subsequently, image feature extraction is performed using a dual-flow network comprising convolutional neural network (CNN) and Transformer architectures. The CNN employs a residual dense block (RDB) to embed spatial local structure information with varying receptive fields. An improved attention mechanism module, attention feature fusion (AFF), is then introduced to integrate the image features extracted from the dual-flow network. Finally, through image perception quality guided adversarial learning, the model adjusts the initial enhanced image’s color and recovers more details. Experimental results demonstrate that the proposed algorithm model achieves enhancement effects exceeding 90% on two large image datasets, which represents a 5%–10% improvement over other models. Furthermore, the algorithm exhibits superior performance in terms of peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) image quality evaluation metrics. Our findings indicate that the fusion model significantly enhances image quality, thereby advancing the field of graphic design and showcasing its potential in cultural and creative product design.

EchoPT: A Pretrained Transformer Architecture that Predicts 2D In-Air Sonar Images for Mobile Robotics

The predictive brain hypothesis suggests that perception can be interpreted as the process of minimizing the error between predicted perception tokens generated via an internal world model and actual sensory input tokens. When implementing working examples of this hypothesis in the context of in-air sonar, significant difficulties arise due to the sparse nature of the reflection model that governs ultrasonic sensing. Despite these challenges, creating consistent world models using sonar data is crucial for implementing predictive processing of ultrasound data in robotics. In an effort to enable robust robot behavior using ultrasound as the sole exteroceptive sensor modality, this paper introduces EchoPT (Echo-Predicting Pretrained Transformer), a pretrained transformer architecture designed to predict 2D sonar images from previous sensory data and robot ego-motion information. We detail the transformer architecture that drives EchoPT and compare the performance of our model to several state-of-the-art techniques. In addition to presenting and evaluating our EchoPT model, we demonstrate the effectiveness of this predictive perception approach in two robotic tasks.

Diversifying Multi-Head Attention in the Transformer Model

Recent studies have shown that, due to redundancy, some heads of the Transformer model can be pruned without diminishing the efficiency of the model. In this paper, we propose a constrained optimization algorithm based on Hebbian learning, which trains specific layers in the Transformer architecture in order to enforce diversification between the different heads in the multi-head attention module. The diversification of the heads is achieved through a single-layer feed-forward neural network that is added to the Transformer architecture and is trained with the proposed algorithm. We utilize the algorithm in three different architectural variations of the baseline Transformer model. In addition to the diversification of the heads, the proposed methodology can be used to prune the heads that capture redundant information. Experiments on diverse NLP tasks, including machine translation, text summarization, question answering and large language modeling, show that our proposed approach consistently improves the performance of baseline Transformer models.

MolE: a foundation model for molecular graphs using disentangled attention.

Models that accurately predict properties based on chemical structure are valuable tools in the chemical sciences. However, for many properties, public and private training sets are typically small, making it difficult for models to generalize well outside of the training data. Recently, this lack of generalization has been mitigated by using self-supervised pretraining on large unlabeled datasets, followed by finetuning on smaller, labeled datasets. Inspired by these advances, we report MolE, a Transformer architecture adapted for molecular graphs together with a two-step pretraining strategy. The first step of pretraining is a self-supervised approach focused on learning chemical structures trained on ~842 million molecular graphs, and the second step is a massive multi-task approach to learn biological information. We show that finetuning models that were pretrained in this way perform better than the best published results on 10 of the 22 ADMET (absorption, distribution, metabolism, excretion and toxicity) tasks included in the Therapeutic Data Commons leaderboard (c. September 2023).

Harnessing Artificial Intelligence for Wildlife Conservation

The rapid decline in global biodiversity demands innovative conservation strategies. This paper examines the use of artificial intelligence (AI) in wildlife conservation, focusing on the Conservation AI platform. Leveraging machine learning and computer vision, Conservation AI detects and classifies animals, humans, and poaching-related objects using visual spectrum and thermal infrared cameras. The platform processes these data with convolutional neural networks (CNNs) and transformer architectures to monitor species, including those that are critically endangered. Real-time detection provides the immediate responses required for time-critical situations (e.g., poaching), while non-real-time analysis supports long-term wildlife monitoring and habitat health assessment. Case studies from Europe, North America, Africa, and Southeast Asia highlight the platform’s success in species identification, biodiversity monitoring, and poaching prevention. The paper also discusses challenges related to data quality, model accuracy, and logistical constraints while outlining future directions involving technological advancements, expansion into new geographical regions, and deeper collaboration with local communities and policymakers. Conservation AI represents a significant step forward in addressing the urgent challenges of wildlife conservation, offering a scalable and adaptable solution that can be implemented globally.

Continuous Evolution of Digital Twins using the DarTwin Notation

AbstractDespite best efforts, various challenges remain in the creation and maintenance processes of digital twins (DTs). One of those primary challenges is the constant, continuous and omnipresent evolution of systems, their user’s needs and their environment, demanding the adaptation of the developed DT systems. DTs are developed for a specific purpose, which generally entails the monitoring, analysis, simulation or optimisation of a specific aspect of an actual system, referred to as the actual twin (AT). As such, when the twin system changes, that is either the AT itself changes, or the scope/purpose of a DT is modified, the DTs usually evolve in close synchronicity with the AT. As DTs are software systems, the best practices or methodologies for software evolution can be leveraged. This paper tackles the challenge of maintaining a (set of) DT(s) throughout the evolution of the user’s requirements and priorities and tries to understand how this evolution takes place. In doing so, we provide two contributions: (i) we develop , a visual notation form that enables reasoning on a twin system, its purposes, properties and implementation, and (ii) we introduce a set of architectural transformations that describe the evolution of DT systems. The development of these transformations is driven and illustrated by the evolution and transformations of a family home’s DT, whose purpose is expanded, changed and re-prioritised throughout its ongoing lifecycle. Additionally, we evaluate the transformations on a laboratory-scale gantry crane’s DT.

DC-Mamba: A Novel Network for Enhanced Remote Sensing Change Detection in Difficult Cases

Remote sensing change detection (RSCD) aims to utilize paired temporal remote sensing images to detect surface changes in the same area. Traditional CNN-based methods are limited by the size of the receptive field, making it difficult to capture the global features of remote sensing images. In contrast, Transformer-based methods address this issue with their powerful modeling capabilities. However, applying the Transformer architecture to image processing introduces a quadratic complexity problem, significantly increasing computational costs. Recently, the Mamba architecture based on state-space models has gained widespread application in the field of RSCD due to its excellent global feature extraction capabilities and linear complexity characteristics. Nevertheless, existing Mamba-based methods lack optimization for complex change areas, making it easy to lose shallow features or local features, which leads to poor performance on challenging detection cases and high-difficulty datasets. In this paper, we propose a Mamba-based RSCD network for difficult cases (DC-Mamba), which effectively improves the model’s detection capability in complex change areas. Specifically, we introduce the edge-feature enhancement (EFE) block and the dual-flow state-space (DFSS) block, which enhance the details of change edges and local features while maintaining the model’s global feature extraction capability. We propose a dynamic loss function to address the issue of sample imbalance, giving more attention to difficult samples during training. Extensive experiments on three change detection datasets demonstrate that our proposed DC-Mamba outperforms existing state-of-the-art methods overall and exhibits significant performance improvements in detecting difficult cases.

CovTransformer: A trans former model for SARS-CoV-2 lineage frequency forecasting

Abstract With hundreds of SARS-CoV-2 lineages circulating in the global population, there is an ongoing need for predicting and forecasting lineage frequencies and thus identifying rapidly expanding lineages. Accu4 rate prediction would allow for more focused experimental efforts to understand pathogenicity of future dominating lineages and characterize the extent of their immune escape. Here, we first show that the inherent noise and biases in lineage frequency data make a commonly-used regression-based approach unreliable. To address this weakness, we constructed a machine learning model for SARS-CoV-2 lineage frequency forecasting, called CovTransformer, based on the transformer architecture. We designed our model to navigate challenges such as a limited amount of data with high levels of noise and bias. We first trained and tested the model using data from the UK and the US, and then tested the generalization ability of the model to other many countries. Remarkably, the trained model makes accurate predictions two months into the future with high levels of accuracy both globally (in 31 countries with high levels of sequencing effort) and at the US-state level. Our model performed substantially better than a widely used forecasting tool, the multinomial regression model implemented in Nextstrain, demonstrating its utility in SARS-CoV-2 monitoring. Assuming a newly emerged lineage is identified and assigned, our test using retrospective data shows that our model is able to identify the dominating lineages 7 weeks in advance on average before they became dominant. Overall, our work demonstrates that transformer models represent a promising approach for SARS-CoV-2 forecasting and pandemic monitoring.

LEM-Detector: An Efficient Detector for Photovoltaic Panel Defect Detection

Photovoltaic panel defect detection presents significant challenges due to the wide range of defect scales, diverse defect types, and severe background interference, often leading to a high rate of false positives and missed detections. To address these challenges, this paper proposes the LEM-Detector, an efficient end-to-end photovoltaic panel defect detector based on the transformer architecture. To address the low detection accuracy for Crack and Star crack defects and the imbalanced dataset, a novel data augmentation method, the Linear Feature Augmentation (LFA) module, specifically designed for linear features, is introduced. LFA effectively improves model training performance and robustness. Furthermore, the Efficient Feature Enhancement Module (EFEM) is presented to enhance the receptive field, suppress redundant information, and emphasize meaningful features. To handle defects of varying scales, complementary semantic information from different feature layers is leveraged for enhanced feature fusion. A Multi-Scale Multi-Feature Pyramid Network (MMFPN) is employed to selectively aggregate boundary and category information, thereby improving the accuracy of multi-scale target recognition. Experimental results on a large-scale photovoltaic panel dataset demonstrate that the LEM-Detector achieves a detection accuracy of 94.7% for multi-scale defects, outperforming several state-of-the-art methods. This approach effectively addresses the challenges of photovoltaic panel defect detection, paving the way for more reliable and accurate defect identification systems. This research will contribute to the automatic detection of surface defects in industrial production, ultimately enhancing production efficiency.

Data-Based Prediction of Redox Potentials via Introducing Chemical Features into the Transformer Architecture.

Rapid and accurate prediction of basic physicochemical parameters of molecules will greatly accelerate the target-orientated design of novel reactions and materials but has been long challenging. Herein, a chemical language model-based deep learning method, TransChem, has been developed for the prediction of redox potentials of organic molecules. Embedding an effective molecular characterization (combining spatial and electronic features), a nonlinear molecular messaging approach (Mol-Attention), and a perturbation learning method, TransChem, shows high accuracy in predicting the redox potential of organic radicals comprising over 100,000 data (R2 > 0.97, MAE <0.09 V) and is generalized to the smaller 2,1,3-benzothiadiazole data set (<3000 data points) and electron affinity data set (660 data) with low MAE of 0.07 V and 0.18 eV, respectively. In this context, a self-developed data set, i.e., the oxidation potential (OP) of a full-space disubstituted phenol data set (OPP-data set, total set: 74,529), has been predicted by TransChem with a high-throughput, and active learning strategy. The rapid and reliable prediction of OP could hopefully accelerate the screening of plausible reagents in highly selective cross-coupling of phenol derivatives. This study presents an important attempt to guide language modeling with chemical knowledge, while TransChem demonstrates state-of-the-art (SOTA) predictive performance on redox potential prediction benchmark data sets for its better understanding of molecular design and conformational relationships.