Proposed Transformer Model Research Articles

Infectious disease time series modelling using transformer self-attention based network

Abstract Researchers are focusing on improving time series forecasting methods to address real-world problems like COVID-19. Current methods show de-creased accuracy due to unpredictable seasonality, and enhancing models to handle long-term dependencies is crucial for better forecasting accuracy. This paper presents a Transformer self-attention based novel approach for infec-tious disease time series forecasting, specifically for COVID-19. The proposed method utilizes the Ensemble Empirical Mode Decomposition (EEMD) and Local Outlier Factor (LOF) methods for data pre-processing and to detect outliers. Next, a modified self-attention model based on Transformer neural network is introduced for predicting COVID-19 time series forecasting for the first time. The research specifically investigates the application of encod-er/decoder networks with an enhanced Positional Encoding approach. This involves using a novel time encoding technique on the input pattern to achieve more precise intended output. Consequently, the parameters in the Transformer model are adjusted using the Arithmetic Optimization Algorithm (AOA) to enhance the accuracy of the prediction. The model generates more accurate predictions over broader time intervals, with the lowest MAE of 371.92 and RMSE of 674.61, indicating superior predictive accuracy by ap-proximately 30% compared to other state-of-the-art methods. The proposed Transformer model has demonstrated significant improvements in robustness and forecasting accuracy compared to standard approaches such as LSTM, RNN, Exponential Smoothing, AutoARIMA, and TBATS for the COVID-19 time series of India, USA, and Brazil. The suggested model, due to its superior predictive accuracy, is applicable in diverse time series forecasting domains such as stock market trends, sales, and industrial consumption forecasting etc.

Modeling and Evaluation of Attention Mechanism Neural Network Based on Industrial Time Series Data

Chemical process control systems are complex, and modeling the controlled object is the first task in automatic control and optimal design. Most chemical process modeling experiments require test signals to be applied to the process, which may lead to production interruptions or cause safety accidents. Therefore, this paper proposes an improved transformer model based on a self-attention mechanism for modeling industrial processes. Then, an evaluation mechanism based on root mean square error (RMSE) and Kullback–Leibler divergence (KLD) metrics is designed to obtain more appropriate model parameters. The Variational Auto-Encoder (VAE) network is used to compute the associated KLD. Finally, a real nonlinear dynamic process in the petrochemical industry is modeled and evaluated using the proposed methodology to predict the time series data of the process. This study demonstrates the validity of the proposed transformer model and illustrates the versatility of using an integrated modeling, evaluation, and prediction scheme for nonlinear dynamic processes in process industries. The scheme is of great importance for the field of industrial soft measurements as well as for deep learning-based time series prediction. In addition, the issue of a suitable time domain for the prediction is discussed.

Segmentation for mammography classification utilizing deep convolutional neural network

BackgroundMammography for the diagnosis of early breast cancer (BC) relies heavily on the identification of breast masses. However, in the early stages, it might be challenging to ascertain whether a breast mass is benign or malignant. Consequently, many deep learning (DL)-based computer-aided diagnosis (CAD) approaches for BC classification have been developed.MethodsRecently, the transformer model has emerged as a method for overcoming the constraints of convolutional neural networks (CNN). Thus, our primary goal was to determine how well an improved transformer model could distinguish between benign and malignant breast tissues. In this instance, we drew on the Mendeley data repository’s INbreast dataset, which includes benign and malignant breast types. Additionally, the segmentation anything model (SAM) method was used to generate the optimized cutoff for region of interest (ROI) extraction from all mammograms. We implemented a successful architecture modification at the bottom layer of a pyramid transformer (PTr) to identify BC from mammography images.ResultsThe proposed PTr model using a transfer learning (TL) approach with a segmentation technique achieved the best accuracy of 99.96% for binary classifications with an area under the curve (AUC) score of 99.98%, respectively. We also compared the performance of the proposed model with other transformer model vision transformers (ViT) and DL models, MobileNetV3 and EfficientNetB7, respectively.ConclusionsIn this study, a modified transformer model is proposed for BC prediction and mammography image classification using segmentation approaches. Data segmentation techniques accurately identify the regions affected by BC. Finally, the proposed transformer model accurately classified benign and malignant breast tissues, which is vital for radiologists to guide future treatment.

TST-Refrac: A Novel Transformer Variant for Prediction of Production of Re-fractured Oil Wells

Abstract The accurate prediction of post-fracturing production for re-fractured wells is crucial for selecting suitable targets for re-fracturing. Given that the post-fracturing production of re-fractured wells is essentially time series data, this study aims to provide an accurate deep neural network approach for production prediction of re-fractured wells. Traditional deep neural networks, such as Recurrent Neural Networks (RNN) and Long Short-Term Memory Networks (LSTM), are not adept at capturing long-range dependencies. To address this limitation, a new Transformer Variant for the Prediction of Production of Re-fractured Oil Wells is proposed, based on the time window method and the classic Transformer architecture. Experimental results demonstrate that the performance of the Transformer shows significant improvement compared to RNN and LSTM. Utilizing production data from Block W in the J Basin, a random sample of nine wells was selected for model fitting and prediction. The time series Transformer model exhibited the lowest Root Mean Square Error. The successful implementation of the proposed Transformer model for time series demonstrates its capability to effectively capture long-range dependencies, enabling more precise predictions than traditional deep learning methods.

Predicting pedestrian-vehicle interaction severity at unsignalized intersections

Objectives This study aims to develop and validate a novel deep-learning model that predicts the severity of pedestrian-vehicle interactions at unsignalized intersections, distinctively integrating Transformer-based models with Multilayer Perceptrons (MLP). This approach leverages advanced feature analysis capabilities, offering a more direct and interpretable method than traditional models. Methods High-resolution optical cameras recorded detailed pedestrian and vehicle movements at study sites, with data processed to extract trajectories and convert them into real-world coordinates via precise georeferencing. Trained observers categorized interactions into safe passage, critical event, and conflict based on movement patterns, speeds, and accelerations. Fleiss Kappa statistic measured inter-rater agreement to ensure evaluator consistency. This study introduces a novel deep-learning model combining Transformer-based time series data capabilities with the classification strengths of a Multilayer Perceptron (MLP). Unlike traditional models, this approach focuses on feature analysis for greater interpretability. The model, trained on dynamic input variables from trajectory data, employs attention mechanisms to evaluate the significance of each input variable, offering deeper insights into factors influencing interaction severity. Results The model demonstrated high performance across different severity categories: safe interactions achieved a precision of 0.78, recall of 0.91, and F1-score of 0.84. In more severe categories like critical events and conflicts, precision and recall were even higher. Overall accuracy stood at 0.87, with both macro and weighted averages for precision, recall, and F1-score also at 0.87. The variable importance analysis, using attention scores from the proposed transformer model, identified ‘Vehicle Speed’ as the most significant input variable positively influencing severity. Conversely, ‘Approaching Angle’ and ‘Vehicle Distance from Conflict Point’ negatively impacted severity. Other significant factors included ‘Type of Vehicle’, ‘Pedestrian Speed’, and ‘Pedestrian Yaw Rate’, highlighting the complex interplay of behavioral and environmental factors in pedestrian-vehicle interactions. Conclusions This study introduces a deep-learning model that effectively predicts the severity of pedestrian-vehicle interactions at crosswalks, utilizing a Transformer-MLP hybrid architecture with high precision and recall across severity categories. Key factors influencing severity were identified, paving the way for further enhancements in real-time analysis and broader safety assessments in urban settings.

Forest Fire Prediction Based on Time Series Networks and Remote Sensing Images

Protecting forest resources and preventing forest fires are vital for social development and public well-being. However, current research studies on forest fire warning systems often focus on extensive geographic areas like states, counties, and provinces. This approach lacks the precision and detail needed for predicting fires in smaller regions. To address this gap, we propose a Transformer-based time series forecasting model aimed at improving the accuracy of forest fire predictions in smaller areas. Our study focuses on Quanzhou County, Guilin City, Guangxi Province, China. We utilized time series data from 2021 to 2022, along with remote sensing images and ArcGIS technology, to identify various factors influencing forest fires in this region. We established a time series dataset containing twelve influencing factors, each labeled with forest fire occurrences. By integrating these data with the Transformer model, we generated forest fire danger level prediction maps for Quanzhou County. Our model’s performance is compared with other deep learning methods using metrics such as RMSE, and the results reveal that the proposed Transformer model achieves higher accuracy (ACC = 0.903, MAPE = 0.259, MAE = 0.053, RMSE = 0.389). This study demonstrates that the Transformer model effectively takes advantage of spatial background information and the periodicity of forest fire factors, significantly enhancing predictive accuracy.

IoT-Based Energy Consumption Prediction Using Transformers

With the advancement of various IoT-based systems, the amount of data is steadily increasing. The increase of data on a daily basis is essential for decision-makers to assess current situations and formulate future policies. Among the various types of data, time-series data presents a challenging relationship between current and future dependencies. Time-series prediction aims to forecast future values of target variables by leveraging insights gained from past data points. Recent advancements in deep learning-based algorithms have surpassed traditional machine learning-based algorithms for time-series in IoT systems. In this study, we employ Enc&Dec Transformer, the latest advancements in neural networks for time-series prediction problems. The obtained results were compared with Encoder-only and Decoder-only Transformer blocks as well as well-known recurrent based algorithms, including 1D-CNN, RNN, LSTM, and GRU. To validate our approach, we utilize three different univariate time-series datasets collected on an hourly basis, focusing on energy consumption within IoT systems. Our results demonstrate that our proposed Transformer model outperforms its counterparts, achieving a minimum Mean Squared Error (MSE) of 0.020 on small, 0.008 on medium, and 0.006 on large-sized datasets.

Feasibility of decoding covert speech in ECoG with a Transformer trained on overt speech

Several attempts for speech brain–computer interfacing (BCI) have been made to decode phonemes, sub-words, words, or sentences using invasive measurements, such as the electrocorticogram (ECoG), during auditory speech perception, overt speech, or imagined (covert) speech. Decoding sentences from covert speech is a challenging task. Sixteen epilepsy patients with intracranially implanted electrodes participated in this study, and ECoGs were recorded during overt speech and covert speech of eight Japanese sentences, each consisting of three tokens. In particular, Transformer neural network model was applied to decode text sentences from covert speech, which was trained using ECoGs obtained during overt speech. We first examined the proposed Transformer model using the same task for training and testing, and then evaluated the model’s performance when trained with overt task for decoding covert speech. The Transformer model trained on covert speech achieved an average token error rate (TER) of 46.6% for decoding covert speech, whereas the model trained on overt speech achieved a TER of 46.3% (p>0.05;d=0.07)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$( p > 0.05; d=0.07)$$\\end{document}. Therefore, the challenge of collecting training data for covert speech can be addressed using overt speech. The performance of covert speech can improve by employing several overt speeches.

A framework for automated landslide dating utilizing SAR-Derived Parameters Time-Series, An Enhanced Transformer Model, and Dynamic Thresholding

Determining the timing of landslide occurrence is crucial for establishing an accurate, comprehensive and systematic landslide inventory while assessing the potential for reducing landslide risk. Unfortunately, many existing landslide inventories lack temporal information such as the precise time of landslide events. Optical and Synthetic Aperture Radar (SAR) sensors are the most commonly used remote sensing technologies for landslide detection. Unlike optical sensors, SAR sensors are not affected by cloudy conditions and provide valuable imagery regardless of sunlight availability. Therefore, SAR-derived parameters, i.e., SAR amplitude, interferometric coherence, and polarimetric features (alpha and entropy), offer a higher temporal resolution for detecting landslide occurrence times compared to optical data. Despite the advantages, there is currently no universally accepted automatic method for determining the time of landslide events using SAR data. This is due to the lack of anomaly labels and the high time-series volatility in detecting landslide occurrence times. Despite advances in deep-learning methods for anomaly detection in time-series, only a few of them can address these challenges in our case. In this paper, we propose an unsupervised multivariate transformed-based deep-learning model to automatically and efficiently estimate landslide occurrence times using multivariate SAR-derived parameters time-series analysis. The designed gated relative position can increase robustness and temporal context information, by learning global temporal trends in the time-series. Subsequently, the time-series of the anomaly score derived from the proposed Transformer model is analyzed using an adaptive thresholding strategy to dynamically and automatically mark anomalies related to the landslide occurrence. Our research focuses on collapsed landslides characterized by dramatic changes in ground surface topography, with a particular attention for the need of a prior knowledge about landslide boundaries. We assess the performance of the proposed methodology for several collapsed landslides including the July 21, 2020 Shaziba and 23 July, 2019 Shuicheng landslides in China, March 19, 2019 Takht landslide in Iran, June 15, 2018 Jalgyz-Jangak and May 25, 2018 Kugart landslides in Kyrgyzstan, July 7, 2018 Hitardalur landslide in Iceland, and January 25, 2019 Brumadinho landslide in Brazil. In comparison to commonly used neural networks like the LSTM algorithm, our proposed framework leads to a more accurate estimate for the time of landslide failure using time-series of SAR-derived parameters. Furthermore, our results suggest the great potential of SAR data to narrow the time period detected from optical data when used in conjunction with them.

A Deep Learning Model for Markerless Pose Estimation Based on Keypoint Augmentation: What Factors Influence Errors in Biomechanical Applications?

In biomechanics, movement is typically recorded by tracking the trajectories of anatomical landmarks previously marked using passive instrumentation, which entails several inconveniences. To overcome these disadvantages, researchers are exploring different markerless methods, such as pose estimation networks, to capture movement with equivalent accuracy to marker-based photogrammetry. However, pose estimation models usually only provide joint centers, which are incomplete data for calculating joint angles in all anatomical axes. Recently, marker augmentation models based on deep learning have emerged. These models transform pose estimation data into complete anatomical data. Building on this concept, this study presents three marker augmentation models of varying complexity that were compared to a photogrammetry system. The errors in anatomical landmark positions and the derived joint angles were calculated, and a statistical analysis of the errors was performed to identify the factors that most influence their magnitude. The proposed Transformer model improved upon the errors reported in the literature, yielding position errors of less than 1.5 cm for anatomical landmarks and 4.4 degrees for all seven movements evaluated. Anthropometric data did not influence the errors, while anatomical landmarks and movement influenced position errors, and model, rotation axis, and movement influenced joint angle errors.

Lightweight Transformer Model for Mobile Application Classification.

Recently, realistic services like virtual reality and augmented reality have gained popularity. These realistic services require deterministic transmission with end-to-end low latency and high reliability for practical applications. However, for these real-time services to be deterministic, the network core should provide the requisite level of network. To deliver differentiated services to each real-time service, network service providers can classify applications based on traffic. However, due to the presence of personal information in headers, application classification based on encrypted application data is necessary. Initially, we collected application traffic from four well-known applications and preprocessed this data to extract encrypted application data and convert it into model input. We proposed a lightweight transformer model consisting of an encoder, a global average pooling layer, and a dense layer to categorize applications based on the encrypted payload in a packet. To enhance the performance of the proposed model, we determined hyperparameters using several performance evaluations. We evaluated performance with 1D-CNN and ET-BERT. The proposed transformer model demonstrated good performance in the performance evaluation, with a classification accuracy and F1 score of 96% and 95%, respectively. The time complexity of the proposed transformer model was higher than that of 1D-CNN but performed better in application classification. The proposed transformer model had lower time complexity and higher classification performance than ET-BERT.

An efficient methodology for simulating multivariate non-Gaussian stochastic processes

Simulation of multivariate non-Gaussian stochastic processes is still a challenging task, where the efficiency and accuracy may not be balanced. In this paper, an efficient methodology is presented to address this challenge. This methodology makes use of a probability weighted moments-based Hermite polynomial model and a wavenumber–frequency spectrum density-based spectral representation approach. First, a probability weighted moments-based Hermite polynomial model is employed to establish a transformation relationship between Gaussian and non-Gaussian samples. Importantly, this model only requires the solution of a solvable system of equations. Second, the target non-Gaussian wavenumber–frequency spectrum density can be promptly transformed into the Gaussian one, where a fast root matching strategy and fast Fourier transform are utilized. Finally, the underlying Gaussian samples are generated by the spectral representation method, and the non-Gaussian samples are obtained via the proposed transformation model. The computational results from numerical examples demonstrate that the proposed methodology is considered to be an accurate, applicable, and efficient approach for simulating multivariate non-Gaussian stochastic processes.

A deep learning-based transformer model for photovoltaic fault forecasting and classification

According to the US-based National Renewable Energy Lab (NREL), solar energy losses due to faults were 3.5 % in 2004, which increased to 17.5 % in 2018. Therefore, the fault prediction mechanism will enable PV practitioners to reduce losses effectively, enhancing the solar system's efficiency and power output. This paper proposes a deep learning-based Transformer model for robust fault prediction in photovoltaic. Transformer uses attention mechanism that considers data points as a language units “word” and learn dependencies among them to predict upcoming data points. Unlike other forecasting algorithms, our proposed approach does not rely on previous trends. In case of PV faults, trends do not exist. The proposed algorithm utilizes rate of change of solar cell parameters for establishing a trend to forecast faults, enabling proactive fault mitigation. It also classifies faults with different severity levels to identify the level of predictive maintenance required. The proposed approach is extensively evaluated using MATLAB on datasets of several faults with low, medium, and high severity levels. The proposed Transformer model achieves a forecasting mean average error (MAE) of 0.09377. Performance of the proposed forecasting and classification algorithm is compared with existing machine learning-based regression and classification techniques such as KNN, SVM, and NN, where proposed approach outperforms state-of-the-art approaches.

Bio-inspired affordance learning for 6-DoF robotic grasping: A transformer-based global feature encoding approach

The 6-Degree-of-Freedom (6-DoF) robotic grasping is a fundamental task in robot manipulation, aimed at detecting graspable points and corresponding parameters in a 3D space, i.e affordance learning, and then a robot executes grasp actions with the detected affordances. Existing research works on affordance learning predominantly focus on learning local features directly for each grid in a voxel scene or each point in a point cloud scene, subsequently filtering the most promising candidate for execution. Contrarily, cognitive models of grasping highlight the significance of global descriptors, such as size, shape, and orientation, in grasping. These global descriptors indicate a grasp path closely tied to actions. Inspired by this, we propose a novel bio-inspired neural network that explicitly incorporates global feature encoding. In particular, our method utilizes a Truncated Signed Distance Function (TSDF) as input, and employs the recently proposed Transformer model to encode the global features of a scene directly. With the effective global representation, we then use deconvolution modules to decode multiple local features to generate graspable candidates. In addition, to integrate global and local features, we propose using a skip-connection module to merge lower-layer global features with higher-layer local features. Our approach, when tested on a recently proposed pile and packed grasping dataset for a decluttering task, surpassed state-of-the-art local feature learning methods by approximately 5% in terms of success and declutter rates. We also evaluated its running time and generalization ability, further demonstrating its superiority. We deployed our model on a Franka Panda robot arm, with real-world results aligning well with simulation data. This underscores our approach’s effectiveness for generalization and real-world applications.

Transformer network for data imputation in electricity demand data

Load forecasting necessitates a significant amount of smart meter data. Several elements in this process, including device malfunctions and signal transmission issues, produce missing data gaps. Missing values in the dataset significantly influence the learning ability of machine learning algorithms, and they must be infilled before proceeding with any statistical analysis. This paper investigates the handling of missing values in demand data, and a new approach is developed for improving the performance of demand analytics, such as energy forecasting. The proposed model uses a transformer neural network to impute the missing values at various rates in the demand profile. Our model uses a k-means algorithm to fill in the missing values with proxy values in the dataset. The model is applied to two case-study residential house located in Cornwall and Fintry, United Kingdom. The developed algorithm is assessed for it potential for infilling missing values for three widely understood missing value scenarios: missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR). The proposed model's imputed outputs are compared to the original dataset to assess model performance. The performance of the framework is compared with a selection of widely used statistical and machine learning models. The proposed transformer model shows significant improvements over the common linear method in all three scenarios (with 30% missing values), with percentage improvements ranging from approximately 49.71% to 57.52% for Cornwall dataset.

Classification of attention deficit/hyperactivity disorder based on EEG signals using a EEG-Transformer model ∗ ∗This work was supported in part by the National Key R&D Program of China (2022YFE0197500), National Natural Science Foundation of China (#81927804, #62101538), Science and Technology Planning Project of Shenzhen (#JSGG20210713091808027, #JSGG20211029095801002), China Postdoctoral Science Foundation (2022M710968), and the Science and Technology Program of Guangdong Province

Objective. Attention-deficit/hyperactivity disorder (ADHD) is the most common neurodevelopmental disorder in adolescents that can seriously impair a person’s attention function, cognitive processes, and learning ability. Currently, clinicians primarily diagnose patients based on the subjective assessments of the Diagnostic and Statistical Manual of Mental Disorders-5, which can lead to delayed diagnosis of ADHD and even misdiagnosis due to low diagnostic efficiency and lack of well-trained diagnostic experts. Deep learning of electroencephalogram (EEG) signals recorded from ADHD patients could provide an objective and accurate method to assist physicians in clinical diagnosis. Approach. This paper proposes the EEG-Transformer deep learning model, which is based on the attention mechanism in the traditional Transformer model, and can perform feature extraction and signal classification processing for the characteristics of EEG signals. A comprehensive comparison was made between the proposed transformer model and three existing convolutional neural network models. Main results. The results showed that the proposed EEG-Transformer model achieved an average accuracy of 95.85% and an average AUC value of 0.9926 with the fastest convergence speed, outperforming the other three models. The function and relationship of each module of the model are studied by ablation experiments. The model with optimal performance was identified by the optimization experiment. Significance. The EEG-Transformer model proposed in this paper can be used as an auxiliary tool for clinical diagnosis of ADHD, and at the same time provides a basic model for transferable learning in the field of EEG signal classification.

Probabilistic generic transformation model between two rock mass properties: specific fracture energy and P-wave velocity

This study proposed a probabilistic generic transformation model between two rock mass properties, specific fracture energy, Ev, and P-wave velocity, VP. To build the transformation model, 12 pairwise data sets of Ev and VP were collected from six different construction sites involving construction of mountain tunnels in Japan. This database is labeled as “RockMass/2/350”. A probabilistic transformation model was built based on a bivariate standard normal distribution with these 350 data points. The model is generic, because it is based on a variety of sites. The performance of the constructed transformation model was evaluated through a cross-validation. It was found that 98.2% of the validation data fell within the computed 95% confidence interval of the model estimation, and this result provides a preliminary validation of the probabilistic transformation model. Unlike existing deterministic transformation models for estimating VP from Ev, the proposed model can explicitly evaluate the transformation uncertainty with a quantitative metric such as a percentile. For practical application, a 3D model of the spatial distribution for Young’s modulus, E, was visualized based on the proposed transformation model. Since the proposed model is probabilistic, it can provide the spatial distribution for percentiles of E values. The constructed 3D model presented in this paper can be directly used as an input data for finite element or finite difference analysis, and probabilistic evaluation of excavation simulation is feasible based on the proposed probabilistic model. The quantitative information on such uncertainty can be useful in decision-making for tunnel constructions, such as selection of a cautious characteristic value.

Heartbeat information prediction based on transformer model using millimetre‐wave radar

AbstractMillimetre‐wave radar offers high ranging accuracy and can capture subtle vibration information of the human heart. This study proposes a heartbeat prediction method based on the transformer model using millimetre‐wave radar. Firstly, the millimetre‐wave radar was used to collect the heartbeat data and conduct normalisation processing. Secondly, a position coding was introduced to assign sine or cosine variables to input data and extract their relative position relationship. Subsequently, the transformer encoder was adopted to allocate attention to input data through the multi‐head attention mechanism, using a mask layer before the decoding layer to prevent the leakage of future information. Finally, we employ the fully connected layer was employed in the linear decoder for regression and output the predicted results. Our experimental results demonstrate that the proposed transformer model achieves nearly 30% higher prediction accuracy than traditional long short‐term memory models while improving both the prediction accuracy and convergence rate. The proposed method has great potential in predicting the heartbeat state of elderly and sick patients.

ITran: A novel transformer-based approach for industrial anomaly detection and localization

Anomaly detection is currently an essential quality monitoring process in industrial production. It is often affected by factors such as under or over reconstruction of images and unclear criteria for feature distribution evaluation, thus making it challenging to improve detection performance. To solve the above problems, this paper proposes a novel transformer-based approach, Inductive Transformer (ITran) for industrial anomaly detection and localization, which utilizes a multi-layer pyramid structure and multi-level jump connections to extract multi-scale features of the data, putting the anomaly detection into the feature space and achieving more accurate industrial anomaly detection and localization results. It incorporates inductive bias and convolution operations into the Transformer which helps to break the myth of Transformer being “data hungry”. Compared with the common Transformers, ITran significantly reduces the computational cost and memory usage and makes it work well on small datasets. In addition, we basically eliminate the effect of positional embedding on the proposed Transformer model. Sufficient experiments have been conducted to validate global anomaly detection on three datasets MNIST, Fashion-MNST and Cifar-10, as well as local anomaly detection on the industrial datasets MVTec AD, Concrete Crack Image and BTAD. The proposed ITran achieves outstanding results on all the above datasets.

Time-series forecasting of mortality rates using transformer

Predicting mortality rates is a crucial issue in life insurance pricing and demographic statistics. Traditional approaches, such as the Lee-Carter model and its variants, predict the trends of mortality rates using factor models, which explain the variations of mortality rates from the perspective of ages, gender, regions, and other factors. Recently, deep learning techniques have achieved great success in various tasks and shown strong potential for time-series forecasting. In this paper, we propose a modified Transformer architecture for predicting mortality rates in major countries around the world. Through the multi-head attention mechanism and positional encoding, the proposed Transformer model extracts key features effectively and thus achieves better performance in time-series forecasting. By using empirical data from the Human Mortality Database, we demonstrate that our Transformer model has higher prediction accuracy of mortality rates than the Lee-Carter model and other classic neural networks. Our model provides a powerful forecasting tool for insurance companies and policy makers.