Transformer Architecture Research Articles

Implementing data lineage frameworks in financial institutions: A systematic analysis of compliance, efficiency, and risk management

This article comprehensively analyzes modern data lineage frameworks within financial institutions, examining their implementation challenges, operational impacts, and regulatory compliance benefits. The article investigates the key components of enterprise-scale data lineage systems, including source integration, transformation architecture, storage mechanisms, and governance structures. The article systematically examines implementation strategies across multiple financial organizations and identifies critical success factors and common impediments in establishing robust data lineage frameworks. The findings demonstrate significant improvements in regulatory compliance efficiency, risk management capabilities, and operational transparency when implementing comprehensive data lineage solutions. The article highlights the importance of integrated governance policies, automated monitoring systems, and scalable architectural designs in achieving successful outcomes. Furthermore, the article reveals that financial institutions implementing structured data lineage frameworks show an enhanced ability to meet evolving regulatory requirements while maintaining operational efficiency. This article contributes to the growing knowledge of data governance in financial services. It provides practical insights for organizations seeking to strengthen their data management capabilities through modern lineage frameworks.

Modernizing enterprise architecture: A guide to microservices migration and hybrid cloud integration

The migration from monolithic to microservices architecture represents a transformative journey in enterprise software development, complemented by the adoption of hybrid cloud technologies. This comprehensive article explores the strategic approaches, implementation patterns, and best practices for successful architectural transformation. The article examines key aspects, including preliminary analysis requirements, domain-driven design implementation, technical considerations for database and API management, and the integration of hybrid cloud technologies. Through a detailed examination of containerization, orchestration, and observability practices, this research provides organizations with a structured framework for modernizing their enterprise architecture while maintaining operational efficiency and system reliability. The article incorporates real-world implementation cases across various sectors, demonstrating measurable improvements in development agility, system performance, and business outcomes.

Optimizing Robotic Manipulation With Decision-RWKV: A Recurrent Sequence Modeling Approach for Lifelong Learning

Abstract Models based on the transformer architecture have seen widespread application across fields such as natural language processing (NLP), computer vision, and robotics, with large language models (LLMs) like ChatGPT revolutionizing machine understanding of human language and demonstrating impressive memory capacity and reproduction capabilities. Traditional machine learning algorithms struggle with catastrophic forgetting, detrimental to the diverse and generalized abilities required for robotic deployment. This article investigates the receptance weighted key value (RWKV) framework, known for its advanced capabilities in efficient and effective sequence modeling, integration with the decision transformer (DT), and experience replay architectures. It focuses on potential performance enhancements in sequence decision-making and lifelong robotic learning tasks. We introduce the decision-RWKV (DRWKV) model and conduct extensive experiments using the D4RL database within the OpenAI Gym environment and on the D’Claw platform to assess the DRWKV model’s performance in single-task tests and lifelong learning scenarios, showing its ability to handle multiple subtasks efficiently. The code for all algorithms, training, and image rendering in this study is available online (open source).

Systematic Review of Hybrid Vision Transformer Architectures for Radiological Image Analysis.

Vision transformer (ViT)and convolutional neural networks (CNNs) each possess distinct strengths in medical imaging: ViT excels in capturing long-range dependencies through self-attention, while CNNs are adept at extracting local features via spatial convolution filters. While ViT may struggle with capturing detailed local spatial information, critical for tasks like anomaly detection in medical imaging, shallow CNNs often fail to effectively abstract global context. This study aims to explore and evaluate hybrid architectures that integrate ViT and CNN to leverage their complementary strengths for enhanced performance in medical vision tasks, such as segmentation, classification, reconstruction, and prediction. Following PRISMA guideline, a systematic review was conducted on 34 articles published between 2020 and Sept. 2024. These articles proposed novel hybrid ViT-CNN architectures specifically for medical imaging tasks in radiology. The review focused on analyzing architectural variations, merging strategies between ViT and CNN, innovative applications of ViT, and efficiency metrics including parameters, inference time (GFlops), and performance benchmarks. The review identified that integrating ViT and CNN can mitigate the limitations of each architecture offering comprehensive solutions that combine global context understanding with precise local feature extraction. We benchmarked the articles based on architectural variations, merging strategies, innovative uses of ViT, and efficiency metrics (number of parameters, inference time (GFlops), and performance), and derived a ranked list. By synthesizing current literature, this review defines fundamental concepts of hybrid vision transformers and highlights emerging trends in the field. It provides a clear direction for future research aimed at optimizing the integration of ViT and CNN for effective utilization in medical imaging, contributing to advancements in diagnostic accuracy and image analysis. We performed systematic review of hybrid vision transformer architecture using PRISMA guideline and performed thorough comparative analysis to benchmark the architectures.

Masked autoencoder for multiagent trajectories

Automatically labeling trajectories of multiple agents is key to behavioral analyses but usually requires a large amount of manual annotations. This also applies to the domain of team sport analyses. In this paper, we specifically show how pretraining transformer models improves the classification performance on tracking data from professional soccer. For this purpose, we propose a novel self-supervised masked autoencoder for multiagent trajectories to effectively learn from only a few labeled sequences. Our approach builds upon a factorized transformer architecture for multiagent trajectory data and employs a masking scheme on the level of individual agent trajectories. As a result, our model allows for a reconstruction of masked trajectory segments while being permutation equivariant with respect to the agent trajectories. In addition to experiments on soccer, we demonstrate the usefulness of the proposed pretraining approach on multiagent pose data from entomology. In contrast to related work, our approach is conceptually much simpler, does not require handcrafted features and naturally allows for permutation invariance in downstream tasks.

RPM-ETC: A Risk Prediction Model for Elevators Based on Transformer and Self-Temporal Compression Mechanism

Elevators play an indispensable role in modern urban life. Ensuring the safe operation of elevators is crucial due to the severe consequences of malfunctions. Traditional maintenance methods are costly and may not comprehensively capture potential faults. Leveraging deep learning technologies, this study proposes a Risk Prediction Model based on Elevator Transformer and Self-temporal Compression Mechanism (RPM-ETC). By analyzing rich operational data, the model predicts potential faults before significant issues arise. The model utilizes the Transformer architecture to effectively capture temporal relationships and employs a temporal compression mechanism to enhance prediction efficiency. Additionally, it uses Enhanced Positional Encoding to prevent the the loss of temporal information as network depth increases. Based on the obtained performance results, the model achieves an accuracy of 86.3% and a frame-per-second (FPS) rate of 388.7, accurately and rapidly predicting elevator faults. Additionally, this paper provides a comprehensive dataset for elevator operation prediction to facilitate further research.

QuanFormer: A Transformer-Based Precise Peak Detection and Quantification Tool in LC-MS-Based Metabolomics.

In metabolomic analysis based on liquid chromatography coupled with mass spectrometry, detecting and quantifying intricate objects is a massive job. Current peak picking methods still cause high rates of incorrectly picked peaks to influence the reliability and reproducibility of results. To address these challenges, we developed QuanFormer, a deep learning method based on object detection designed to accurately quantify peak signals. Our algorithm combines the feature extraction capabilities of convolutional neural networks (CNNs) with the global computation capability of Transformer architecture. Data training in QuanFormer by using nearly 20,000 annotated regions-of-interest (ROIs) ensures unique prediction via bipartite matching, achieving 96.5% of the average precision value on the test set. Even without retraining, QuanFormer achieves over 90% accuracy in distinguishing true from false peaks. Performance was further analyzed using visualization techniques applied to the encoder and decoder layers. We also demonstrated that QuanFormer could correct retention time shifts for peak alignment and generally surpass the existing methods, including MZmine 3 and PeakDetective, to obtain a larger number of picked peaks and higher accurate quantification. Finally, we also carried out metabolomic analysis in a clinical cohort of breast cancer patients and utilized QuanFormer to detect and quantify the potential biomarkers. QuanFormer is open-source and available at https://github.com/LinShuhaiLAB/QuanFormer.

Spatio-temporal transformers for decoding neural movement control.

Deep learning tools applied to high-resolution neurophysiological data have significantly progressed, offering enhanced decoding, real-time processing, and readability for practical applications. However, the design of artificial neural networks to analyze neural activity in vivo remains a challenge, requiring a delicate balance between efficiency in low-data regimes and the interpretability of the results.
Approach:
To address this challenge, we introduce a novel specialized transformer architecture to analyze single-neuron spiking activity. The model is tested on multi-electrode recordings from the dorsal premotor cortex (PMd) of non-human primates performing a motor inhibition task.
Main Results
The proposed architecture provides an early prediction of the correct movement direction, achieving accurate results no later than 230 ms after the Go signal presentation across animals. Additionally, the model can forecast whether the movement will be generated or withheld before a Stop signal, unattended, is actually presented. To further understand the internal dynamics of the model, we compute the predicted correlations between time steps and between neurons at successive layers of the architecture, with the evolution of these correlations mirrors findings from previous theoretical analyses.
Significance
Overall, our framework provides a comprehensive use case for the practical implementation of deep learning tools in motor control research, highlighting both the predictive capabilities and interpretability of the proposed architecture.

Transformer Architecture for Micromotion Target Detection Based on Multi-Scale Subaperture Coherent Integration

In recent years, long-time coherent integration techniques have gained significant attention in maneuvering target detection due to their ability to effectively enhance the signal-to-noise ratio (SNR) and improve detection performance. However, for space targets, challenges such as micromotion phenomena and complex scattering characteristics make envelope alignment and phase compensation difficult, thereby limiting integration gain. To address these issues, in this study, we conducted an in-depth analysis of the echo model of cylindrical space targets (CSTs) based on different types of scattering centers. Building on this foundation, the multi-scale subaperture coherent integration Transformer (MsSCIFormer) was proposed, which integrates MsSCI with a Transformer architecture to achieve precise detection and motion parameter estimation of space targets in low-SNR environments. The core of the method lies in the introduction of a convolutional neural network (CNN) feature extractor and a dual-attention mechanism, covering both intra-subaperture attention (Intra-SA) and inter-subaperture attention (Inter-SA). This design efficiently captures the spatial distribution and motion patterns of the scattering centers of space targets. By aggregating multi-scale features, MsSCIFormer significantly enhances the detection performance and improves the accuracy of motion parameter estimation. Simulation experiments demonstrated that MsSCIFormer outperforms traditional moving target detection (MTD) methods and other deep learning-based algorithms in both detection and estimation tasks. Furthermore, each module proposed in this study was proven to contribute positively to the overall performance of the network.

FPE–Transformer: A Feature Positional Encoding-Based Transformer Model for Attack Detection

The increase in cybersecurity threats has made attack detection systems critically important. Traditional deep learning methods often require large amounts of data and struggle to understand relationships between features effectively. With their self-attention mechanism, Transformers excel in modeling complex relationships and long-term dependencies. They are also adaptable to various data types and sources, making them advantageous in large-scale attack detection scenarios. This paper introduces the FPE–Transformer framework, leveraging the strengths of the Transformer architecture. FPE–Transformer incorporates an innovative feature positional encoding mechanism that encodes the positional information of each feature separately, enabling a deeper understanding of feature relationships and more precise attack detection. Additionally, the model includes a ClassificationHead for enhanced accuracy and complex pattern recognition. The framework’s performance was validated using the NSL-KDD and CIC-IDS2017 datasets, demonstrating its superiority over traditional methods in detecting diverse attack types and improving overall performance. This study highlights FPE–Transformer’s innovative approach and ability to address key limitations of traditional deep learning methods, establishing it as a robust solution for modern attack detection challenges.

Leveraging two-dimensional pre-trained vision transformers for three-dimensional model generation via masked autoencoders

Although the Transformer architecture has established itself as the industry standard for jobs involving natural language processing, it still has few uses in computer vision. In vision, attention is used in conjunction with convolutional networks or to replace individual convolutional network elements while preserving the overall network design. Differences between the two domains, such as significant variations in the scale of visual things and the higher granularity of pixels in images compared to words in the text, make it difficult to transfer Transformer from language to vision. Masking autoencoding is a promising self-supervised learning approach that greatly advances computer vision and natural language processing. For robust 2D representations, pre-training with large image data has become standard practice. On the other hand, the low availability of 3D datasets significantly impedes learning high-quality 3D features because of the high data processing cost. We present a strong multi-scale MAE prior training architecture that uses a trained ViT and a 3D representation model from 2D images to let 3D point clouds learn on their own. We employ the adept 2D information to direct a 3D masking-based autoencoder, which uses an encoder-decoder architecture to rebuild the masked point tokens through self-supervised pre-training. To acquire the input point cloud’s multi-view visual characteristics, we first use pre-trained 2D models. Next, we present a two-dimensional masking method that preserves the visibility of semantically significant point tokens. Numerous tests demonstrate how effectively our method works with pre-trained models and how well it generalizes to a range of downstream tasks. In particular, our pre-trained model achieved 93.63% accuracy for linear SVM on ScanObjectNN and 91.31% accuracy on ModelNet40. Our approach demonstrates how a straightforward architecture solely based on conventional transformers may outperform specialized transformer models from supervised learning.

MammoViT: A Custom Vision Transformer Architecture for Accurate BIRADS Classification in Mammogram Analysis

Background: Breast cancer screening through mammography interpretation is crucial for early detection and improved patient outcomes. However, the manual classification of mammograms using the BIRADS (Breast Imaging-Reporting and Data System) remains challenging due to subtle imaging features, inter-reader variability, and increasing radiologist workload. Traditional computer-aided detection systems often struggle with complex feature extraction and contextual understanding of mammographic abnormalities. To address these limitations, this study proposes MammoViT, a novel hybrid deep learning framework that leverages both ResNet50’s hierarchical feature extraction capabilities and Vision Transformer’s ability to capture long-range dependencies in images. Methods: We implemented a multi-stage approach utilizing a pre-trained ResNet50 model for initial feature extraction from mammogram images. To address the significant class imbalance in our four-class BIRADS dataset, we applied SMOTE (Synthetic Minority Over-sampling Technique) to generate synthetic samples for minority classes. The extracted feature arrays were transformed into non-overlapping patches with positional encodings for Vision Transformer processing. The Vision Transformer employs multi-head self-attention mechanisms to capture both local and global relationships between image patches, with each attention head learning different aspects of spatial dependencies. The model was optimized using Keras Tuner and trained using 5-fold cross-validation with early stopping to prevent overfitting. Results: MammoViT achieved 97.4% accuracy in classifying mammogram images across different BIRADS categories. The model’s effectiveness was validated through comprehensive evaluation metrics, including a classification report, confusion matrix, probability distribution, and comparison with existing studies. Conclusions: MammoViT effectively combines ResNet50 and Vision Transformer architectures while addressing the challenge of imbalanced medical imaging datasets. The high accuracy and robust performance demonstrate its potential as a reliable tool for supporting clinical decision-making in breast cancer screening.

Integrating Transformer Architecture and Householder Transformations for Enhanced Temporal Knowledge Graph Embedding in DuaTHP

The rapid advancement of knowledge graph (KG) technology has led to the emergence of temporal knowledge graphs (TKGs), which represent dynamic relationships over time. Temporal knowledge graph embedding (TKGE) techniques are commonly employed for link prediction and knowledge graph completion, among other tasks. However, existing TKGE models mainly rely on basic arithmetic operations, such as addition, subtraction, and multiplication, which limits their capacity to capture complex, non-linear relationships between entities. Moreover, many neural network-based TKGE models focus on static entities and relationships, overlooking the temporal dynamics of entity neighborhoods and their potential for encoding relational patterns, which can result in significant semantic loss. To address these limitations, we propose DuaTHP, a novel model that integrates Transformer blocks with Householder projections in the dual quaternion space. DuaTHP utilizes Householder projections to map head-to-tail entity relations, effectively capturing key relational patterns. The model incorporates two Transformer blocks: the entity Transformer, which models entity–relationship interactions, and the context Transformer, which aggregates relational and temporal information. Additionally, we introduce a time-restricted neighbor selector, which focuses on neighbors interacting within a specific time frame to enhance domain-specific analysis. Experimental results demonstrate that DuaTHP significantly outperforms existing methods in link prediction and knowledge graph completion, effectively addressing both semantic loss and time-related issues in TKGs.

Deep Learning-Based Multi-View Projection Synthesis Approach for Improving the Quality of Sparse-View CBCT in Image-Guided Radiotherapy.

While radiation hazards induced by cone-beam computed tomography (CBCT) in image-guided radiotherapy (IGRT) can be reduced by sparse-view sampling, the image quality is inevitably degraded. We propose a deep learning-based multi-view projection synthesis (DLMPS) approach to improve the quality of sparse-view low-dose CBCT images. In the proposed DLMPS approach, linear interpolation was first applied to sparse-view projections and the projections were rearranged into sinograms; these sinograms were processed with a sinogram restoration model and then rearranged back into projections. The sinogram restoration model was modified from the 2D U-Net by incorporating dynamic convolutional layers and residual learning techniques. The DLMPS approach was trained, validated, and tested on CBCT data from 163, 30, and 30 real patients respectively. Sparse-view projection datasets with 1/4 and 1/8 of the original sampling rate were simulated, and the corresponding full-view projection datasets were restored via the DLMPS approach. Tomographic images were reconstructed using the Feldkamp-Davis-Kress algorithm. Quantitative metrics including root-mean-square error (RMSE), peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and feature similarity (FSIM) were calculated in both the projection and image domains to evaluate the performance of the DLMPS approach. The DLMPS approach was compared with 11 state-of-the-art (SOTA) models, including CNN and Transformer architectures. For 1/4 sparse-view reconstruction task, the proposed DLMPS approach achieved averaged RMSE, PSNR, SSIM, and FSIM values of 0.0271, 45.93 dB, 0.9817, and 0.9587 in the projection domain, and 0.000885, 37.63 dB, 0.9074, and 0.9885 in the image domain, respectively. For 1/8 sparse-view reconstruction task, the DLMPS approach achieved averaged RMSE, PSNR, SSIM, and FSIM values of 0.0304, 44.85 dB, 0.9785, and 0.9524 in the projection domain, and 0.001057, 36.05 dB, 0.8786, and 0.9774 in the image domain, respectively. The DLMPS approach outperformed all the 11 SOTA models in both the projection and image domains for 1/4 and 1/8 sparse-view reconstruction tasks. The proposed DLMPS approach effectively improves the quality of sparse-view CBCT images in IGRT by accurately synthesizing missing projections, exhibiting potential in substantially reducing imaging dose to patients with minimal loss of image quality.

Application of Transformers to Chemical Synthesis

Efficient chemical synthesis is critical for the production of organic chemicals, particularly in the pharmaceutical industry. Leveraging machine learning to predict chemical synthesis and improve the development efficiency has become a significant research focus in modern chemistry. Among various machine learning models, the Transformer, a leading model in natural language processing, has revolutionized numerous fields due to its powerful feature-extraction and representation-learning capabilities. Recent applications demonstrated that Transformer models can also significantly enhance the performance in chemical synthesis tasks, particularly in reaction prediction and retrosynthetic planning. This article provides a comprehensive review of the applications and innovations of Transformer models in the qualitative prediction tasks of chemical synthesis, with a focus on technical approaches, performance advantages, and the challenges associated with applying the Transformer architecture to chemical reactions. Furthermore, we discuss the future directions for improving the applications of Transformer models in chemical synthesis.

P0400 Automated real-time detection of intestinal tract and inflammatory bowel disease in intestinal ultrasound with the vision transformer (ViT)

Abstract Background Intestinal ultrasound is a powerful tool for diagnosing and monitoring inflammatory bowel disease (IBD). Although access to training is improving, the detection and interpretation of ultrasound images in IBD remains complex. The aim of this study was to develop and validate an artificial intelligence module to automatically detect the intestinal tract and active IBD in order to simplify the use of ultrasound in IBD. Methods Vision Transformers (ViTs), recently introduced as a competitive alternative to Convolutional Neural Networks (CNNs), were used to automate the real time detection and segmentation of the normal and inflamed intestinal tract. ViTs were chosen for their ability to better capture global relationships in images, which is particularly useful for recognizing complex patterns in medical images. The ViT model was trained on a dataset consisting of 17,488 intestinal ultrasound images from 46 patients, including 11,396 images of active IBD and 6,092 inactive IBD evaluated by two gastroenterologists. Results The sensitivity, specificity and accuracy for the detection of inactive and active IBD calculated from two intestinal ultrasound loop containing a total of 617 images of inactive IBD and 339 images of active IBD were 95%, 94% and 95% respectively. Some limitations were identified, notably a decrease in sensitivity in the detection of minor anomalies, in areas of low contrast and during variations in scale. These points suggest that the model could benefit from further refinement of its spatial processing capabilities, particularly in complex clinical situations. Conclusion We have developed a model based on the pre-trained Vision Transformer (ViT) architecture that is able to detect the active and inactive IBD. This approach could eventually make it easier for inexperienced operators to use intestinal ultrasound to detect and monitor IBD. Further in-depth tests on a larger set of images and videos are under way to validate the model’s performance.

Hybrid Neural Network Models to Estimate Vital Signs from Facial Videos

Introduction: Remote health monitoring plays a crucial role in telehealth services and the effective management of patients, which can be enhanced by vital sign prediction from facial videos. Facial videos are easily captured through various imaging devices like phone cameras, webcams, or surveillance systems. Methods: This study introduces a hybrid deep learning model aimed at estimating heart rate (HR), blood oxygen saturation level (SpO2), and blood pressure (BP) from facial videos. The hybrid model integrates convolutional neural network (CNN), convolutional long short-term memory (convLSTM), and video vision transformer (ViViT) architectures to ensure comprehensive analysis. Given the temporal variability of HR and BP, emphasis is placed on temporal resolution during feature extraction. The CNN processes video frames one by one while convLSTM and ViViT handle sequences of frames. These high-resolution temporal features are fused to predict HR, BP, and SpO2, capturing their dynamic variations effectively. Results: The dataset encompasses 891 subjects of diverse races and ages, and preprocessing includes facial detection and data normalization. Experimental results demonstrate high accuracies in predicting HR, SpO2, and BP using the proposed hybrid models. Discussion: Facial images can be easily captured using smartphones, which offers an economical and convenient solution for vital sign monitoring, particularly beneficial for elderly individuals or during outbreaks of contagious diseases like COVID-19. The proposed models were only validated on one dataset. However, the dataset (size, representation, diversity, balance, and processing) plays an important role in any data-driven models including ours. Conclusions: Through experiments, we observed the hybrid model’s efficacy in predicting vital signs such as HR, SpO2, SBP, and DBP, along with demographic variables like sex and age. There is potential for extending the hybrid model to estimate additional vital signs such as body temperature and respiration rate.

Pre-trained Maldi Transformers improve MALDI-TOF MS-based prediction.

For the last decade, matrix-assisted laser desorption/ionization time-of-flight mass spectrometry (MALDI-TOF MS) has been the reference method for species identification in clinical microbiology. Hampered by a historical lack of open data, machine learning research towards models specifically adapted to MALDI-TOF MS remains in its infancy. Given the growing complexity of available datasets (such as large-scale antimicrobial resistance prediction), a need for models that (1) are specifically designed for MALDI-TOF MS data, and (2) have high representational capacity, presents itself. Here, we introduce Maldi Transformer, an adaptation of the state-of-the-art transformer architecture to the MALDI-TOF mass spectral domain. We propose the first self-supervised pre-training technique specifically designed for mass spectra. The technique is based on shuffling peaks across spectra, and pre-training the transformer as a peak discriminator. Extensive benchmarks confirm the efficacy of this novel design. The final result is a model exhibiting state-of-the-art (or competitive) performance on downstream prediction tasks. In addition, we show that Maldi Transformer's identification of noisy spectra may be leveraged towards higher predictive performance. All code supporting this study is distributed on PyPI and is packaged under: https://github.com/gdewael/maldi-nn.

Augmenting LLMs to Securely Retrieve Information for Construction and Facility Management

In the past few years, generative AI has seen remarkable progress. The emergence of the transformer architecture has facilitated the creation of highly advanced language models that generate text, summarize content, and translate languages with impressive accuracy. Our study introduces a retrieval-augmented generation system tailored to the dynamic needs of facility management. The proposed system aims to provide instant, accurate access to essential information by integrating advanced techniques from natural language processing and information retrieval paradigms. The implementation leverages the Mixtral 8x7B model for multilingual text processing and the Milvus vector database for efficient document storage and retrieval. The dataset used includes documents such as images, operation manuals, inspection results, blueprints, and technical drawings, in various file formats. This diverse dataset reflects the variety of information encountered in construction and facility management. The evaluation involved generating question–answer pairs pertinent to facility management tasks and assessing the system’s performance using metrics such as ROUGE, BLEU, and semantic similarity. The findings suggest that retrieval-augmented generation systems can significantly enhance operational efficiency by reducing the time and effort required to access information while maintaining high security and data privacy standards.

Aggregate global features into separable hierarchical lane detection transformer

Lane detection is one of the key functions to ensure the safe driving of autonomous vehicles, and it is a challenging task. In real driving scenarios, external factors inevitably interfere with the lane detection system, such as missing lane markings, harsh weather conditions, and vehicle occlusion. To enhance the accuracy and detection speed of lane detection in complex road environments, this paper proposes an end-to-end lane detection model with a pure Transformer architecture, which exhibits excellent detection performance in complex road scenes. Firstly, a separable lane multi-head attention mechanism based on window self-attention is proposed. This mechanism can establish the attention relationship between each window faster and more effectively, reducing the computational cost and improving the detection speed. Then, an extended and overlapping strategy is designed, which solves the problem of insufficient information interaction between two adjacent windows of the standard multi-head attention mechanism, thereby obtaining more global information and effectively improving the detection accuracy in complex road environments. Finally, experiments are carried out on four data sets. The experimental results indicate that the proposed method is superior to the existing state of the arts method in terms of both effectiveness and efficiency.