Length Of Input Sequence Research Articles

Recurrent Fourier-Kolmogorov Arnold Networks for photovoltaic power forecasting

Accurate day-ahead forecasting of photovoltaic (PV) power generation is crucial for power system scheduling. To overcome the inaccuracies and inefficiencies of current PV power generation forecasting models, this paper introduces the Recurrent Fourier-Kolmogorov Arnold Network (RFKAN). Initially, recurrent kernel nodes are employed to investigate the interdependencies within sequences. Subsequently, Fourier series are applied to extract periodic features, enhancing forecasting accuracy and training speed. Ablation studies conducted using data from a PV power plant in Tieling City, Liaoning Province, validate the effectiveness of these two structural enhancements. Comparative experiments with baseline and state-of-the-art models further underscore the efficiency of RFKAN. The results indicate that RFKAN achieves the best forecasting performance with a grid depth of 100 and an input sequence length of 2, reducing RMSE and MAE by at least 5%, increasing CORR by 2%, and decreasing training time by 24% compared to advanced models.

Unlocking Potential Score Insights of Sentimental Analysis with a Deep Learning Revolutionizes

Online hate has emerged as a rapidly growing issue worldwide, often stemming from differences in opinion. It is crucial to use appropriate language and words on social media platforms, as inappropriate communication can negatively impact others. Consequently, detecting hate speech is of significant importance. While manual methods are commonly employed to identify hate and offensive content on social media, they are time-consuming, labor-intensive, and prone to errors. Therefore, AI-based approaches are increasingly being adopted for the effective classification of hate and offensive speech. The proposed model incorporates various text preprocessing techniques, such as removing extraneous elements like URLs, emojis, and blank spaces. Following preprocessing, tokenization is applied to break down the text into smaller components or tokens. The tokenization technique utilized in this study is TF-IDF (Term Frequency–Inverse Document Frequency). After tokenization, the model performs the classification of hate and offensive speech using the proposed BiLSTM-based SM-CJ (Scalable Multi-Channel Joint) framework. The BiLSTM-based SM-CJ model is particularly effective in detecting hate, offensive, and neutral tweets due to its ability to capture both forward and backward contexts within a given text. Detecting hate speech requires a comprehensive understanding of the text and the identification of patterns spanning across multiple words or phrases. To achieve this, the LSTM component of the BiLSTM model is designed to capture long-term dependencies by utilizing information from earlier parts of the text. The proposed SM-CJ framework further aligns the input sequence lengths fetched from the input layer, enabling the model to focus on specific segments of the input sequence that are most relevant for hate speech detection. This approach allows the model to accurately capture derogatory language, and subtle nuances present in hate speech. Finally, the performance of the proposed framework is evaluated using various metrics, including accuracy, recall, F1-score, and precision. The results are compared with state-of-the-art approaches, demonstrating the effectiveness of the proposed model. Doi: 10.28991/ESJ-2025-09-01-03 Full Text: PDF

LMPTMSite: A Platform for PTM Site Prediction in Proteins Leveraging Transformer-Based Protein Language Models.

Protein post-translational modifications (PTMs) introduce new functionalities and play a critical role in the regulation of protein functions. Characterizing these modifications, especially PTM sites, is essential for unraveling complex biological systems. However, traditional experimental approaches, such as mass spectrometry, are time-consuming and expensive. Machine learning and deep learning techniques offer promising alternatives for predicting PTM sites. In this chapter, we introduce our LMPTMSite (language model-based post-translational modification site predictor) platform, which emphasizes two transformer-based protein language model (pLM) approaches: pLMSNOSite and LMSuccSite, for the prediction of S-nitrosylation sites and succinylation sites in proteins, respectively. We highlight the various methods of using pLM-based sequence encoding, explain the underlying deep learning architectures, and discuss the superior efficacy of these tools compared to other state-of-the-art tools. Subsequently, we present an analysis of runtime and memory usage for pLMSNOSite, with a focus on CPU and RAM usage as the input sequence length is scaled up. Finally, we showcase a case study predicting succinylation sites in proteins active within the tricarboxylic acid (TCA) cycle pathway using LMSuccSite, demonstrating its potential utility and efficiency in real-world biological contexts. The LMPTMSite platform, inclusive of pLMSNOSite and LMSuccSite, is freely available both as a web server ( http://kcdukkalab.org/pLMSNOSite/ and http://kcdukkalab.org/LMSuccSite/ ) and as standalone packages ( https://github.com/KCLabMTU/pLMSNOSite and https://github.com/KCLabMTU/LMSuccSite ), providing valuable tools for researchers in the field.

Entity-enhanced BERT for medical specialty prediction based on clinical questionnaire data.

A medical specialty prediction system for remote diagnosis can reduce the unexpected costs incurred by first-visit patients who visit the wrong hospital department for their symptoms. To develop medical specialty prediction systems, several researchers have explored clinical predictive models using real medical text data. Medical text data include large amounts of information regarding patients, which increases the sequence length. Hence, a few studies have attempted to extract entities from the text as concise features and provide domain-specific knowledge for clinical text classification. However, it is still insufficient to inject them into the model effectively. Thus, we propose Entity-enhanced BERT (E-BERT), which utilizes the structural attributes of BERT for medical specialty prediction. E-BERT has an entity embedding layer and entity-aware attention to inject domain-specific knowledge and focus on relationships between medical-related entities within the sequences. Experimental results on clinical questionnaire data demonstrate the superiority of E-BERT over the other benchmark models, regardless of the input sequence length. Moreover, the visualization results for the effects of entity-aware attention prove that E-BERT effectively incorporate domain-specific knowledge and other information, enabling the capture of contextual information in the text. Finally, the robustness and applicability of the proposed method is explored by applying it to other Pre-trained Language Models. These effective medical specialty predictive model can provide practical information to first-visit patients, resulting in streamlining the diagnostic process and improving the quality of medical consultations.

Comparison of time-series models for predicting physiological metrics under sedation.

This study presents a comprehensive comparison of multiple time-series models applied to physiological metric predictions. It aims to explore the effectiveness of both statistical prediction models and pharmacokinetic-pharmacodynamic prediction model and modern deep learning approaches. Specifically, the study focuses on predicting the bispectral index (BIS), a vital metric in anesthesia used to assess the depth of sedation during surgery, using datasets collected from real-life surgeries. The goal is to evaluate and compare model performance considering both univariate and multivariate schemes. Accurate BIS prediction is essential for avoiding under- or over-sedation, which can lead to adverse outcomes. The study investigates a range of models: The traditional mathematical models include the pharmacokinetic-pharmacodynamic model and statistical models such as autoregressive integrated moving average (ARIMA) and vector autoregression (VAR). The deep learning models encompass recurrent neural networks (RNNs), specifically Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU), as well as Temporal Convolutional Networks (TCNs) and Transformer models. The analysis focuses on evaluating model performance in predicting the BIS using two distinct datasets of physiological metrics collected from actual surgical procedures. It explores both univariate and multivariate prediction schemes and investigates how different combinations of features and input sequence lengths impact model accuracy. The experimental findings reveal significant performance differences among the models: In univariate prediction scenarios for predicting BIS, the LSTM model demonstrates a 2.88% improvement over the second-best performing model. For multivariate predictions, the LSTM model outperforms others by 6.67% compared to the next best model. Furthermore, the addition of Electromyography (EMG) and Mean Arterial Pressure (MAP) brings significant accuracy improvement when predicting BIS. The study emphasizes the importance of selecting and building appropriate time-series models to achieve accurate predictions in biomedical applications. This research provides insights to guide future efforts in improving vital sign prediction methodologies for clinical and research purposes. Clinically, with improvements in the prediction of physiological parameters, clinicians can be informed of interventions if an anomaly is detected or predicted.

Predicting significant wave height in the South China Sea using the SAC-ConvLSTM model

IntroductionThe precise forecasting of Significant wave height(SWH) is vital to ensure the safety and efficiency of aquatic activities such as ocean engineering, shipping, and fishing.MethodsThis paper proposes a deep learning model named SAC-ConvLSTM to perform 24-hour prediction with the SWH in the South China Sea. The long-term prediction capability of the model is enhanced by using the attention mechanism and context vectors. The prediction ability of the model is evaluated by mean absolute error (MAE), root mean square error (RMSE), mean square error (MSE), and Pearson correlation coefficient (PCC).ResultsThe experimental results show that the optimal input sequence length for the model is 12. Starting from 12 hours, the SAC-ConvLSTM model consistently outperforms other models in predictive performance. For the 24-hour prediction, this model achieves RMSE, MAE, and PCC values of 0.2117 m, 0.1083 m, and 0.9630, respectively. In addition, the introduction of wind can improve the accuracy of wave prediction. The SAC-ConvLSTM model also has good prediction performance compared to the ConvLSTM model during extreme weather, especially in coastal areas.DiscussionThis paper presents a 24-hour prediction of SWH in the South China Sea. Through comparative validation, the SAC-ConvLSTM model outperforms other models. The inclusion of wind data enhances the model's predictive capability. This model also performs well under extreme weather conditions. In physical oceanography, variables related to SWH include not only wind but also other factors such as mean wave period and sea surface air pressure. In the future, additional variables can be incorporated to further improve the model's predictive performance.

Chemical Gas Source Localization with Synthetic Time Series Diffusion Data Using Video Vision Transformer

Gas source localization is vital in emergency scenarios to enable swift and effective responses. In this study, we introduce a gas source localization model leveraging the video vision transformer (ViViT). Utilizing synthetic time series diffusion data, the source grid is predicted by classifying the grid with the highest probability of gas occurrence within the diffusion data coverage. Through extensive experimentation using the NBC-RAMS simulator, we generate large datasets of gas diffusion under varied experimental conditions and meteorological environments, enabling comprehensive model training and evaluation. Our findings demonstrate that the ViViT outperforms other deep learning models in processing time series gas data, showcasing a superior estimation performance. Leveraging a transformer architecture, the ViViT exhibits a robust classification performance even in scenarios influenced by weather conditions or incomplete observations. Furthermore, we conduct an analysis of accuracy and parameter count across various input sequence lengths, revealing the ability of the ViViT to maintain high computational efficiency while achieving accurate source localization. These results underscore the effectiveness of the ViViT as a model for gas source localization, particularly in situations demanding a rapid response in real-world environments, such as gas leaks or attacks.

An efficient CNN-transformer hybrid approach for water turbine unit failure prediction

Water Turbine Unit is the core equipment of water power generation. It is the most important research object in hydroelectric power. The analysis of daily monitoring data of Water Turbines can evaluate its running state and avoid the loss caused by failure. In this paper, A hybrid neural network architecture which CNN combines Transformer is proposed to evaluate the health state of water turbines based on multivariate long time series data. It has two core components: (i) dilated convolution applies to capture low-level and local semantic information, then the output of convolutional layers is divided into subseries-level patches by time, these patches are regarded as input tokens of Transformer layers; (ii) utilizing self-attention of Transformer to extract high-level and global semantic information. Patching design naturally has benefit as follow: (i) local semantic information is retained in Transformer input tokens and high-level semantic confirm to common sense of human understanding through low-level construction; (ii) the length of Transformer input sequence is greatly shorted and attention can be more concentrated compared with point-wise form. Meanwhile, the computation and memory usage reduces at the same time. The experimental result indicates that the hybrid architecture can achieves excellent performance in time series understanding.

Short-term multi-step-ahead sector-based traffic flow prediction based on the attention-enhanced graph convolutional LSTM network (AGC-LSTM)

AbstractAccurate sector-based air traffic flow predictions are essential for ensuring the safety and efficiency of the air traffic management (ATM) system. However, due to the inherent spatial and temporal dependencies of air traffic flow, it is still a challenging problem. To solve this problem, some methods are proposed considering the relationship between sectors, while the complicated spatiotemporal dynamics and interdependencies between traffic flow of route segments related to the sector are not taken into account. To address this challenge, the attention-enhanced graph convolutional long short-term memory network (AGC-LSTM) model is applied to improve the short-term sector-based traffic flow prediction, in which spatial structures of route segments related to the sector are considered for the first time. Specifically, the graph convolutional networks (GCN)-LSTM network model was employed to capture spatiotemporal dependencies of the flight data, and the attention mechanism is designed to concentrate on the informative features from key nodes at each layer of the AGC-LSTM model. The proposed model is evaluated through a case study of the typical enroute sector in the central–southern region of China. The prediction results show that MAE reduces by 14.4% compared to the best performing GCN-LSTM model among the other five models. Furthermore, the study involves comparative analyses to assess the influence of route segment range, input and output sequence lengths, and time granularities on prediction performance. This study helps air traffic managers predict flight situations more accurately and avoid implementing overly conservative or excessively aggressive flow management measures for the sectors.

A Joint Time-Frequency Domain Transformer for multivariate time series forecasting

In order to enhance the performance of Transformer models for long-term multivariate forecasting while minimizing computational demands, this paper introduces the Joint Time-Frequency Domain Transformer (JTFT). JTFT combines time and frequency domain representations to make predictions. The frequency domain representation efficiently extracts multi-scale dependencies while maintaining sparsity by utilizing a small number of learnable frequencies. Simultaneously, the time domain (TD) representation is derived from a fixed number of the most recent data points, strengthening the modeling of local relationships and mitigating the effects of non-stationarity. Importantly, the length of the representation remains independent of the input sequence length, enabling JTFT to achieve linear computational complexity. Furthermore, a low-rank attention layer is proposed to efficiently capture cross-dimensional dependencies, thus preventing performance degradation resulting from the entanglement of temporal and channel-wise modeling. Experimental results on eight real-world datasets demonstrate that JTFT outperforms state-of-the-art baselines in predictive performance.

Improving respiratory signal prediction with a deep neural network and simple changes to the input and output data format

Objective. To improve respiratory gating accuracy and radiation treatment throughput, we developed a generalized model based on a deep neural network (DNN) for predicting any given patient’s respiratory motion. Approach. Our model uses long short-term memory (LSTM) based on a recurrent neural network (RNN), and improves upon common techniques. The first improvement is that the data input is not a one-dimensional sequence, but two-dimensional block data. This shortens the input sequence length, reducing computation time. Second, the output is not a scalar, but a sequence prediction. This increases the amount of available data, allowing improved prediction accuracy. For training and evaluation of our model, 434 sets of real-time position management data were retrospectively collected from clinical studies. The data were separated in a ratio of 4:1, with the larger set used for training models and the remaining set used for testing. We measured the accuracy of respiratory signal prediction and amplitude-based gating with prediction windows equaling 133, 333, and 533 ms. This new model was compared with the original LSTM and a non-recurrent DNN model. Main results. The mean absolute errors with the prediction window at 133, 333 and 533 ms were 0.036, 0.084, 0.119 with our model; 0.049, 0.14, 0.246 with the original LSTM-based model; and 0.041, 0.119, 0.16 with the non-recurrent DNN model, respectively. The computation time were 0.66 ms with our model; 0.63 ms the original LSTM-based model; 1.60 ms the non-recurrent DNN model, respectively. The accuracies of amplitude-based gating with the same prediction window settings and a duty cycle of approximately 50% were 98.3%, 95.8% and 92.7% with our model, 97.6%, 93.9% and 87.2% with the original LSTM-based model; and 97.9%, 94.3% and 89.5% with the non-recurrent DNN model, respectively. Significance. Our RNN algorithm for respiratory signal prediction successfully estimated tumor positions. We believe it will be useful in respiratory signal prediction technology.

Proxyformer: Nyström-Based Linear Transformer with Trainable Proxy Tokens

Transformer-based models have demonstrated remarkable performance in various domains, including natural language processing, image processing and generative modeling. The most significant contributor to the successful performance of Transformer models is the self-attention mechanism, which allows for a comprehensive understanding of the interactions between tokens in the input sequence. However, there is a well-known scalability issue, the quadratic dependency (i.e. O(n^2)) of self-attention operations on the input sequence length n, making the handling of lengthy sequences challenging. To address this limitation, there has been a surge of research on efficient transformers, aiming to alleviate the quadratic dependency on the input sequence length. Among these, the Nyströmformer, which utilizes the Nyström method to decompose the attention matrix, achieves superior performance in both accuracy and throughput. However, its landmark selection exhibits redundancy, and the model incurs computational overhead when calculating the pseudo-inverse matrix. We propose a novel Nyström method-based transformer, called Proxyformer. Unlike the traditional approach of selecting landmarks from input tokens, the Proxyformer utilizes trainable neural memory, called proxy tokens, for landmarks. By integrating contrastive learning, input injection, and a specialized dropout for the decomposed matrix, Proxyformer achieves top-tier performance for long sequence tasks in the Long Range Arena benchmark.

TS-Fastformer: Fast Transformer for Time-series Forecasting

Many real-world applications require precise and fast time-series forecasting. Recent trends in time-series forecasting models are shifting from LSTM-based models to Transformer-based models. However, the Transformer-based model has a limited ability to represent sequential relationships in time-series data. In addition, the transformer-based model suffers from slow training and inference speed due to the bottleneck incurred by a deep encoder and step-by-step decoder inference. To address these problems, we propose a time-series forecasting optimized Transformer model, called TS-Fastformer. TS-Fastformer introduces three new optimizations: First, we propose a Sub Window Tokenizer for compressing input in a simple manner. The Sub Window Tokenizer reduces the length of input sequences to mitigate the complexity of self-attention and enables both single and multi-sequence learning. Second, we propose Time-series Pre-trained Encoder to extract effective representations through pre-training. This optimization enables TS-Fastformer to capture both seasonal and trend representations as well as to mitigate bottlenecks of conventional transformer models. Third, we propose the Past Attention Decoder to forecast target by incorporating past long short-term dependency patterns. Furthermore, Past Attention Decoder achieves high performance improvement by removing a trend distribution that changes over a long period. We evaluate the efficiency of our model with extensive experiments using seven real-world datasets and compare our model to six representative time-series forecasting approaches. The results show that the proposed TS-Fastformer reduces MSE by 10.1% compared to state-of-the-art model and demonstrates 21.6% faster training time compared to the existing fastest transformer, respectively.

Improving Stock Price Prediction Accuracy with StacBi LSTM

This research aimed to enhance stock price prediction accuracy using the Stacked Bidirectional Long Short-Term Memory (StacBi LSTM) model. The study addressed the challenge of capturing long-term dependencies and temporal patterns inherent in stock price data. The research objectives were to evaluate the model's performance across different input sequence lengths and identify the optimal length for prediction. Leveraging a dataset from the Indonesian Stock Exchange, the model's predictions were evaluated using key metrics such as RMSE, MAE, MAPE, and R2. Results indicated that the StacBi LSTM model excelled in capturing stock price trends and demonstrated strengths over traditional methods. The optimal input sequence length was identified, balancing computational efficiency and prediction accuracy. This research contributes valuable insights into improving stock price prediction techniques and offers practical implications for traders and investors. Future research directions encompass hybrid models and integrating external factors to enhance predictive capabilities further.

Language model-based B cell receptor sequence embeddings can effectively encode receptor specificity.

High throughput sequencing of B cell receptors (BCRs) is increasingly applied to study the immense diversity of antibodies. Learning biologically meaningful embeddings of BCR sequences is beneficial for predictive modeling. Several embedding methods have been developed for BCRs, but no direct performance benchmarking exists. Moreover, the impact of the input sequence length and paired-chain information on the prediction remains to be explored. We evaluated the performance of multiple embedding models to predict BCR sequence properties and receptor specificity. Despite the differences in model architectures, most embeddings effectively capture BCR sequence properties and specificity. BCR-specific embeddings slightly outperform general protein language models in predicting specificity. In addition, incorporating full-length heavy chains and paired light chain sequences improves the prediction performance of all embeddings. This study provides insights into the properties of BCR embeddings to improve downstream prediction applications for antibody analysis and discovery.

Intelligent prediction for support resistance in working faces of coal mine: A research based on deep spatiotemporal sequence models

Accurately predicting support resistance in coal mine working faces is a pressing matter in the field of intelligent mining research, which holds immense importance in enhancing the safety of coal production and the overall level of intelligent mining. The current research in this field primarily focuses on utilizing deep temporal sequence models for prediction. While there have been promising advancements in this area, concerns persist regarding the dependability of predicted outcomes. In fact, the movement of overburden in mining faces varies over time and space, and the changes in the support resistance that reflect the instability of overburden movement also have temporal and spatial characteristics. Therefore, in order to improve the reliability of model prediction, this paper proposes the application of deep spatiotemporal sequence models for support resistance prediction, and identifies the PredRNN and PredRNN++ models as suitable options. The exploration on the application of deep spatiotemporal sequence models for predicting the support resistance has been conducted. The proposed application involves the establishment of a fully working face area grid model with support number and mining cycle as coordinate units. Different from deep temporal sequence models that take the studied grid features as observations, the 25 adjacent spatiotemporal feature grids corresponding to the studied grid are considered as observations for the deep spatiotemporal sequence models. Based on the predicted three-dimensional tensor, the required predicted support resistance data can be obtained. Based on the complete support resistance data from 9 working faces in a coal mine in Datong mining area from 2009 to 2018, a comprehensive dataset was established to experimentally analyze the predictive performance and reliability of the LSTM, PredRNN, and PredRNN++ models. The experimental results demonstrate significant improvements in predictive performance and reliability of deep spatiotemporal sequence models compared to the current mainstream deep temporal sequence models. In terms of predictive performance, the PredRNN and PredRNN++ models have improved their feature extraction ability by incorporating the convolution operations to extract the spatial correlation of the input data. Furthermore, through the incorporation of gated units and nonlinear neurons into the storage units, coupled with the integration of vertical memory state transitions, the depth of the network has significantly increased, resulting in the enhancement of the sequence modeling capabilities. In terms of reliability of prediction results: The acceptable threshold for prediction error was set at 1000 kN. According to the average prediction results of all samples in the test set, it was found that the deep temporal sequence model was unable to accurately predict support resistance ahead. While the deep spatiotemporal sequence model demonstrated a predictable sequence length of 8. According to the prediction results of all samples in the test set, the accurate prediction rate increased from 30 %∼35 % to 65 %∼85 % and the maximum MAE value decreased from 10000 ∼ 12500kN to 6500 ∼ 9000kN in more than 20,000 test samples. The PredRNN model with an input sequence length of 50 achieves an accurate prediction rate of 85.48 % and a maximum MAE value of 6698kN, making it the most reliable prediction model.

Nucleotide-level prediction of CircRNA-protein binding based on fully convolutional neural network.

Introduction: CircRNA-protein binding plays a critical role in complex biological activity and disease. Various deep learning-based algorithms have been proposed to identify CircRNA-protein binding sites. These methods predict whether the CircRNA sequence includes protein binding sites from the sequence level, and primarily concentrate on analysing the sequence specificity of CircRNA-protein binding. For model performance, these methods are unsatisfactory in accurately predicting motif sites that have special functions in gene expression. Methods: In this study, based on the deep learning models that implement pixel-level binary classification prediction in computer vision, we viewed the CircRNA-protein binding sites prediction as a nucleotide-level binary classification task, and use a fully convolutional neural networks to identify CircRNA-protein binding motif sites (CPBFCN). Results: CPBFCN provides a new path to predict CircRNA motifs. Based on the MEME tool, the existing CircRNA-related and protein-related database, we analysed the motif functions discovered by CPBFCN. We also investigated the correlation between CircRNA sponge and motif distribution. Furthermore, by comparing the motif distribution with different input sequence lengths, we found that some motifs in the flanking sequences of CircRNA-protein binding region may contribute to CircRNA-protein binding. Conclusion: This study contributes to identify circRNA-protein binding and provides help in understanding the role of circRNA-protein binding in gene expression regulation.

A semantically enhanced text retrieval framework with abstractive summarization

AbstractRecently, large pretrained language models (PLMs) have led a revolution in the information retrieval community. In most PLMs‐based retrieval frameworks, the ranking performance broadly depends on the model structure and the semantic complexity of the input text. Sequence‐to‐sequence generative models for question answering or text generation have proven to be competitive, so we wonder whether these models can improve ranking effectiveness by enhancing input semantics. This article introduces SE‐BERT, a semantically enhanced bidirectional encoder representation from transformers (BERT) based ranking framework that captures more semantic information by modifying the input text. SE‐BERT utilizes a pretrained generative language model to summarize both sides of the candidate passage and concatenate them into a new input sequence, allowing BERT to acquire more semantic information within the constraints of the input sequence's length. Experimental results from two Text Retrieval Conference datasets demonstrate that our approach's effectiveness increasing as the length of the input text increases.

A novel approach to attention mechanism using kernel functions: Kerformer.

Artificial Intelligence (AI) is driving advancements across various fields by simulating and enhancing human intelligence. In Natural Language Processing (NLP), transformer models like the Kerformer, a linear transformer based on a kernel approach, have garnered success. However, traditional attention mechanisms in these models have quadratic calculation costs linked to input sequence lengths, hampering efficiency in tasks with extended orders. To tackle this, Kerformer introduces a nonlinear reweighting mechanism, transforming maximum attention into feature-based dot product attention. By exploiting the non-negativity and non-linear weighting traits of softmax computation, separate non-negativity operations for Query(Q) and Key(K) computations are performed. The inclusion of the SE Block further enhances model performance. Kerformer significantly reduces attention matrix time complexity from O(N2) to O(N), with N representing sequence length. This transformation results in remarkable efficiency and scalability gains, especially for prolonged tasks. Experimental results demonstrate Kerformer's superiority in terms of time and memory consumption, yielding higher average accuracy (83.39%) in NLP and vision tasks. In tasks with long sequences, Kerformer achieves an average accuracy of 58.94% and exhibits superior efficiency and convergence speed in visual tasks. This model thus offers a promising solution to the limitations posed by conventional attention mechanisms in handling lengthy tasks.

Prediction of Cognitive Test Scores from Variable Length Multimodal Data in Alzheimer’s Disease

Alzheimer’s disease (AD) is a neurogenerative condition characterized by sharp cognitive decline with no confirmed effective treatment or cure. This makes it critically important to identify the symptoms of Alzheimer’s disease in its early stages before significant cognitive deterioration has taken hold and even before any brain morphology and neuropathology are noticeable. In this study, five different multimodal deep neural networks (MDNN), with different architectures, in search of an optimal model for predicting the cognitive test scores for the Mini-Mental State Examination (MMSE) and the modified Alzheimer’s Disease Assessment Scale (ADAS-CoG13) over a span of 60 months (5 years). The multimodal data utilized to train and test the proposed models were obtained from the Alzheimer’s Disease Neuroimaging Initiative study and includes cerebrospinal fluid (CSF) levels of tau and beta-amyloid, structural measures from magnetic resonance imaging (MRI), functional and metabolic measures from positron emission tomography (PET), and cognitive scores from the neuropsychological tests (Cog). The models developed herein delve into two main issues: (1) application merits of single-task vs. multitask for predicting future cognitive scores and (2) whether time-varying input data are better suited than specific timepoints for optimizing prediction results. This model yields a high of 90.27% (SD = 1.36) prediction accuracy (correlation) at 6 months after the initial visit to a lower 79.91% (SD = 8.84) prediction accuracy at 60 months. The analysis provided is comprehensive as it determines the predictions at all other timepoints and all MDNN models include converters in the CN and MCI groups (CNc, MCIc) and all the unstable groups in the CN and MCI groups (CNun and MCIun) that reverted to CN from MCI and to MCI from AD, so as not to bias the results. The results show that the best performance is achieved by a multimodal combined single-task long short-term memory (LSTM) regressor with an input sequence length of 2 data points (2 visits, 6 months apart) augmented with a pretrained Neural Network Estimator to fill in for the missing values.