Attention-based Encoder-decoder Research Articles

Dynamic modeling of post-combustion carbon capture process based on multi-gate mixture-of-experts incorporating dual-stage attention-based encoder-decoder network

Solvent-based post-combustion carbon capture (PCC) technology is a promising, near-term solution for decarbonizing power generation and industrial facilities. Model-based process simulation is crucial for the optimal design and operation of the PCC process. Recently, data-driven models have gained attention due to their adaptability, efficient computation and high accuracy. However, the nonlinearity, strong couplings and multi-time scale features of the PCC process pose significant challenges for model identification. To this end, this paper proposes a multi-gate mixture-of-experts incorporating dual-stage attention-based encoder-decoder (MMoE-DAED) network for dynamic modeling of the PCC process under wide operating conditions. An encoder-decoder composed of long short-term memory (LSTM) network is employed to extract features from the time-dependent input data and learn the complex dynamic interactions caused by the inertial and delay properties of the process. Dual-stage attention mechanism is incorporated into the encoder and decoder respectively to select the most relevant input features and their correlations within the time series data. To enhance multi-output prediction accuracy, multi-gate mixture-of-experts (MMoE) framework that considers correlations of multitask learning is implemented. Simulation results using operating data from a PCC experimental setup indicate that the proposed modeling approach accurately predicts the steady-state values and dynamic trends of the CO2 capture rate and stripper bottom temperature over a wide operating range. The RMSE, MAPE and R2 indices for the CO2 capture rate are 2.1592, 0.0295, 0.9641, respectively, and for the stripper bottom temperature are 0.1491, 0.0003, 0.9833, respectively. Validations on a PCC simulator further verify the accuracy and efficiency of the MMoE-DAED model, which enables an 80.87% reduction in computation time compared to the simulator. This paper points to a new direction for the data-driven dynamic modeling of complex energy conversion processes.

A new portfolio approach integrating three-way decision and Encoder–Decoder network

In the stock market, it can be challenging to predict stock trends and identify stocks with investment potential due to uncertainty and risk. Accurate predictive models and reliable asset preselection methods are essential for portfolio management. This paper proposes an integration of the deep learning model, the three-way decision (3WD) theory, and the Mean-Variance (MV) method for asset preselection and optimal asset allocation. In the initial stage, an attention-based Encoder–Decoder model is used to predict asset returns and calculate volatility. It outperforms other benchmark models regarding relative error, absolute error, and directional accuracy. The subsequent stage involves selecting stocks based on expected returns and volatility characteristics. The 3WD theory is then used to identify the delayed decision set for implementing additional investment strategies to hedge portfolio risk and enhance returns. Finally, the MV model is employed for investment portfolio optimization. The proposed model is validated using a substantial dataset from the A-share market’s SSE 50 Index from January 2010 to December 2022. Results indicate that the proposed model outperforms benchmark models in terms of daily average returns, annualized Information ratio, Sortino ratio, and other aspects under certain trading costs. However, further exploration is required to establish a mechanism that limits the quantity of preselected assets to ensure its broad applicability.

Decoupled structure for improved adaptability of end-to-end models

Although end-to-end (E2E) trainable automatic speech recognition (ASR) has shown great success by jointly learning acoustic and linguistic information, it still suffers from the effect of domain shifts, thus limiting potential applications. The E2E ASR model implicitly learns an internal language model (LM) which characterises the training distribution of the source domain, and the E2E trainable nature makes the internal LM difficult to adapt to the target domain with text-only data. To solve this problem, this paper proposes decoupled structures for attention-based encoder–decoder (Decoupled-AED) and neural transducer (Decoupled-Transducer) models, which can achieve flexible domain adaptation in both offline and online scenarios while maintaining robust intra-domain performance. To this end, the acoustic and linguistic parts of the E2E model decoder (or prediction network) are decoupled, making the linguistic component (i.e. internal LM) replaceable. When encountering a domain shift, the internal LM can be directly replaced during inference by a target-domain LM, without re-training or using domain-specific paired speech-text data. Experiments for E2E ASR models trained on the LibriSpeech-100h corpus showed that the proposed decoupled structure gave 15.1% and 17.2% relative word error rate reductions on the TED-LIUM 2 and AESRC2020 corpora while still maintaining performance on intra-domain data. It is also shown that the decoupled structure can be used to boost cross-domain speech translation quality while retaining the intra-domain performance.

Accurate segmentation of liver tumor from multi-modality non-contrast images using a dual-stream multi-level fusion framework

The use of multi-modality non-contrast images (i.e., T1FS, T2FS and DWI) for segmenting liver tumors provides a solution by eliminating the use of contrast agents and is crucial for clinical diagnosis. However, this remains a challenging task to discover the most useful information to fuse multi-modality images for accurate segmentation due to inter-modal interference. In this paper, we propose a dual-stream multi-level fusion framework (DM-FF) to, for the first time, accurately segment liver tumors from non-contrast multi-modality images directly. Our DM-FF first designs an attention-based encoder–decoder to effectively extract multi-level feature maps corresponding to a specified representation of each modality. Then, DM-FF creates two types of fusion modules, in which a module fuses learned features to obtain a shared representation across multi-modality images to exploit commonalities and improve the performance, and a module fuses the decision evidence of segment to discover differences between modalities to prevent interference caused by modality’s conflict. By integrating these three components, DM-FF enables multi-modality non-contrast images to cooperate with each other and enables an accurate segmentation. Evaluation on 250 patients including different types of tumors from two MRI scanners, DM-FF achieves a Dice of 81.20%, and improves performance (Dice by at least 11%) when comparing the eight state-of-the-art segmentation architectures. The results indicate that our DM-FF significantly promotes the development and deployment of non-contrast liver tumor technology.

Deep reinforcement learning approach with hybrid action space for mobile charging in wireless rechargeable sensor networks

Mobile charging is feasible to deal with the energy-constrained problem in wireless rechargeable sensor networks (WRSNs). The mobile chargers (MCs) are usually employed to charge the sensors sequentially according to the charging schemes. Existing studies assume that each sensor should be charged to its maximum energy capacity or to a fixed upper threshold before the next one can be charged. However, they neglect to control the charging time adaptively of each sensor according to the charging demand. Hence, in this paper, we assume that the charging time of each sensor can be controlled, and we study the joint optimization of charging sequence and charging time problem (JCSCT). Correspondingly, we propose a novel deep reinforcement learning with hybrid action space approach for JCSCT (DRLH-JCSCT), which utilizes deep q-network (DQN) to generate the charging sequence, and adopts deep deterministic policy gradient (DDPG) to determine the charging time. An attention-based encoder–decoder model is integrated in the actor network of DDPG, and a modified bi-directional gate recurrent unit network (MBGRU) is utilized as the decoder. We also design a novel reward function to evaluate the quality of the charging actions. Simulations demonstrate the improved charging performance of the proposed approach, with a longer network lifetime and fewer failed sensors compared with the existing mobile charging scheduling approaches.

A Dual-Stage Attention-Based Vehicle Speed Prediction Model Considering Driver Heterogeneity with Fuel Consumption and Emissions Analysis

With the development of intelligent transportation systems (ITSs), personalized driving systems are receiving more and more attention, and the development of advanced systems cannot be separated from the practical exploration of drivers’ heterogeneous driving behaviors. An important foundation for subsequent driver-targeted research is how to mine the key influencing factors that characterize drivers through real driving data and how to appropriately classify drivers as a whole. This study took heterogeneous drivers as the object, based on a dual-stage attention-based vehicle speed prediction model, and carried out research on the speed prediction of traffic flow and the impact of fuel consumption and emissions in the car-following state considering the heterogeneity of drivers. Specifically, first, Spearman’s correlation analysis and K-means clustering were used to classify different types of drivers. Then, speed predictions for different types of drivers were separated via the dual-stage attention-based encoder–decoder (DAED) model and the prediction results between models and drivers were compared. Finally, the heterogeneous drivers’ fuel consumption and emissions were further analyzed via the VT-micro model. The results show that the proposed speed prediction model can effectively discriminate the influences of heterogeneous drivers on the prediction model, and the aggressive type presents the best effect. In addition, from the experiments on traffic fuel consumption and emissions, it can be concluded that the timid driver is the friendliest to the environment. By researching individual drivers’ driving characteristics, this study may help sustainable development in traffic management.

Attention-Based Encoder-Decoder End-to-End Neural Diarization With Embedding Enhancer

Attention-Based Encoder-Decoder End-to-End Neural Diarization With Embedding Enhancer

AEDNet: An Attention-Based Encoder–Decoder Network for Urban Water Extraction From High Spatial Resolution Remote Sensing Images

AEDNet: An Attention-Based Encoder–Decoder Network for Urban Water Extraction From High Spatial Resolution Remote Sensing Images

Turning brain MRI into diagnostic PET: 15O-water PET CBF synthesis from multi-contrast MRI via attention-based encoder–decoder networks

Accurate quantification of cerebral blood flow (CBF) is essential for the diagnosis and assessment of a wide range of neurological diseases. Positron emission tomography (PET) with radiolabeled water (15O-water) is the gold-standard for the measurement of CBF in humans, however, it is not widely available due to its prohibitive costs and the use of short-lived radiopharmaceutical tracers that require onsite cyclotron production. Magnetic resonance imaging (MRI), in contrast, is more accessible and does not involve ionizing radiation. This study presents a convolutional encoder–decoder network with attention mechanisms to predict the gold-standard 15O-water PET CBF from multi-contrast MRI scans, thus eliminating the need for radioactive tracers. The model was trained and validated using 5-fold cross-validation in a group of 126 subjects consisting of healthy controls and cerebrovascular disease patients, all of whom underwent simultaneous 15O-water PET/MRI. The results demonstrate that the model can successfully synthesize high-quality PET CBF measurements (with an average SSIM of 0.924 and PSNR of 38.8 dB) and is more accurate compared to concurrent and previous PET synthesis methods. We also demonstrate the clinical significance of the proposed algorithm by evaluating the agreement for identifying the vascular territories with impaired CBF. Such methods may enable more widespread and accurate CBF evaluation in larger cohorts who cannot undergo PET imaging due to radiation concerns, lack of access, or logistic challenges.

A framework-based transformer and knowledge distillation for interior style classification

Interior style classification is an interesting problem which has potential applications both commercial and academic communities. This task aims to devise interior design styles automatically. Thus, interior designers will explore customers’ tastes and then precisely provide suggestions for decor inspiration based on their preferences. Recently, Convolutional Neural Networks (CNNs) have been considered the de-facto standard in computer vision tasks. Therefore, several current works have tended to address interior style classification using CNN-based architectures. Moreover, transformer-based architectures and attention-based encoder–decoder models have been proven successfully in computer vision and natural language processing tasks. Sequentially, more studies have been arguing the efficiency of combining CNN-based architectures and transformer-based architectures for normal image classification problems. In this project, we focus on finding an architecture network that is suitable for the interior style classification problem. We propose a robustness method to address interior style design classification, named ISC-DeIT. The proposed method is based on Data-efficient image transformer architectures and knowledge distillation, which can be trained on small datasets effectively. Especially, a proposed additional module is plugged to leverage learning feature representations for improving predictive accuracy. Experiments were carried out on a new curated dataset with five interior styles including Art-Decor, Hitech, Indochina, Industrial, and Scandinavian. Empirical results of ISC- DeiT indicated that the ability of prediction for interior style classification of the proposed method has been increased significantly, compared with other state-of-the-art methods.

An expandable machine learning-optimization framework to sequential decision-making

We present an integrated prediction-optimization (PredOpt) framework to efficiently solve sequential decision-making problems by predicting the values of binary decision variables in an optimal solution. We address the key issues of sequential dependence, infeasibility, and generalization in machine learning (ML) to make predictions for optimal solutions to combinatorial problems. The sequential nature of the combinatorial optimization problems considered is captured with recurrent neural networks and a sliding-attention window. We integrate an attention-based encoder–decoder neural network architecture with an infeasibility-elimination and generalization framework to learn high-quality feasible solutions to time-dependent optimization problems. In this framework, the required level of predictions is optimized to eliminate the infeasibility of the ML predictions. These predictions are then fixed in mixed-integer programming (MIP) problems to solve them quickly with the aid of a commercial solver. We demonstrate our approach to tackling the two well-known dynamic NP-Hard optimization problems: multi-item capacitated lot-sizing (MCLSP) and multi-dimensional knapsack (MSMK). Our results show that models trained on shorter and smaller-dimensional instances can be successfully used to predict longer and larger-dimensional problems. The solution time can be reduced by three orders of magnitude with an average optimality gap below 0.1%. We compare PredOpt with various specially designed heuristics and show that our framework outperforms them. PredOpt can be advantageous for solving dynamic MIP problems that need to be solved instantly and repetitively.

DNTC: An unsupervised Deep Networks for Temperature Compensation in non-stationary data

Temperature changes usually result in abnormal signals and hence could significantly decrease the performance of sensors. To reduce the system errors caused by temperature changes, constructing an accurate temperature compensation model from the data is a promising issue. However, there are few effective methods to address the challenges when faced with practical difficulties, including large amounts of data, high volatility, strong non-stationarity, and high training cost. In this paper, we propose an unsupervised Deep Networks for Temperature Compensation (DNTC) to compensate temperature for non-stationary data. DNTC consists of three submodules: pooling, generation, and reconstruction modules. The pooling module employs a convolutional network to effectively compress features, the generation module uses a De-stationary Attention-based encoder–decoder structure to extract non-stationary information from the sequence, and the reconstruction module solves the statistical quantity deviation problem of the output sequence to achieve temperature compensation. DNTC employs an end-to-end architecture, avoiding the bias accumulation phenomenon caused by multi-stage operations in traditional methods; and it breaks free from the awkwardness of relying on labeled learning, thus expanding its applicability. Furthermore, it effectively handles large-scale sequences while considering their non-stationary characteristics. Extensive experiments on 9 datasets demonstrate that DNTC exhibits excellent compensation effects and is expected to be widely used in the engineering field. Code and data are available at: https://github.com/tietoucun/Temperature-Compensation.

A multitask encoder–decoder model for quality prediction in injection moulding

It is desirable in injection moulding that every yielded product is of high quality and precision. Several studies are conducted on modelling the relationship between the process parameters e.g., holding pressure, mould temperature, etc. that are either time series or non-time series, and the output product quality characteristics e.g., weight, dimensions, and warpage. To build a more comprehensive/generalized quality prediction model with online learning, this study for the first time takes time series process parameters from 2 pairs of in-mould pressure and temperature sensors simultaneously with 18 non-time series parameters and introduces a single deep learning architecture to perform multitask learning i.e., to leverage information from the related sources to simultaneously predict multiple product quality characteristics on a real injection moulding machine. This along with the discussed modern containerization practices develops a fully automated software part of the quality control process. The non-time series process parameters are learned by fully connected neural network layers of the model while the time series process parameters are related through an attention-based encoder–decoder architecture. This model is ablated as well as compared with a benchmark model to validate the performance advantage in terms of the reduced error asymptote. The proposed model offers an order of magnitude lower average mean squared error than the benchmark model, and 1.2 to 3 times smaller mean squared errors than its ablated versions.

Dual attention-based multi-step ahead prediction enhancement for monitoring systems in industrial processes

In industrial processes, the ability to predict future steps is essential as it offers long-term insights, benefiting strategic decision-making. However, traditional sequence-to-sequence models designed to predict dynamic behaviors suffer from accumulating errors during recurrent predictions which use previous outputs as inputs for the next time step. In this article, we propose a dual attention-based encoder–decoder framework, specifically designed to enhance multi-step ahead predictions in industrial processes. The dual attention model strategically minimizes the error accumulation of output sequence by leveraging a temporal attention mechanism, which focuses on relevant time-steps in the input sequence, and a supervised attention mechanism that assigns different weights to output sequence errors during training. The supervised attention method, in particular, provides a significant improvement by focusing on minimizing the error of earlier steps during backpropagation using predefined attention weights, resulting in enhanced overall multistep prediction performance. Experiments on real-world industrial datasets demonstrate that our approach outperforms baseline models, specifically simple sequence-to-sequence and single attention-based sequence-to-sequence models. In fact, our dual attention framework consistently surpasses single attention models, currently regarded as state-of-the-art, at all prediction stages. The suggested approach has potential applications in the field of process monitoring and model predictive control.

Aircraft Bleed Air System Fault Prediction based on Encoder-Decoder with Attention Mechanism

The engine bleed air system (BAS) is one of the important systems for civil aircraft, and fault prediction of BAS is necessary to improve aircraft safety and the operator's profit. A dual-stage two-phase attention-based encoder-decoder (DSTP-ED) prediction model is proposed for BAS normal state estimation. Unlike traditional ED networks, the DSTP-ED combines spatial and temporal attention to better capture the spatiotemporal relationships to achieve higher prediction accuracy. Five data-driven algorithms, autoregressive integrated moving average (ARIMA), support vector regression (SVR), long short-term memory (LSTM), ED, and DSTP-ED, are applied to build prediction models for BAS. The comparison experiments show that the DSTP-ED model outperforms the other four data-driven models. An exponentially weighted moving average (EWMA) control chart is used as the evaluation criterion for the BAS failure warning. An empirical study based on Quick Access Recorder (QAR) data from Airbus A320 series aircraft demonstrates that the proposed method can effectively predict failures.

A Neural Attention-Based Encoder-Decoder Approach for English to Bangla Translation

Machine translation (MT) is the process of translating text from one language to another using bilingual data sets and grammatical rules. Recent works in the field of MT have popularized sequence-to-sequence models leveraging neural attention and deep learning. The success of neural attention models is yet to be construed into a robust framework for automated English-to-Bangla translation due to a lack of a comprehensive dataset that encompasses the diverse vocabulary of the Bangla language. In this study, we have proposed an English-to-Bangla MT system using an encoder-decoder attention model using the CCMatrix corpus. Our method shows that this model can outperform traditional SMT and RBMT models with a Bilingual Evaluation Understudy (BLEU) score of 15.68 despite being constrained by the limited vocabulary of the corpus. We hypothesize that this model can be used successfully for state-of-the-art machine translation with a more diverse and accurate dataset. This work can be extended further to incorporate several newer datasets using transfer learning techniques.

Attention-based encoder–decoder network for depth estimation from color-coded light fields

Compressive light field cameras have attracted notable attention over the past few years because they can efficiently determine redundancy from light fields. However, much of the research has only concentrated on reconstructing the entire light field from compressed sampling, which ignores the possibility of directly extracting information such as depth from it. In this paper, we introduce a light field camera configuration with a random color-coded microlens array. Considering the color-coded light fields, we propose a novel attention-based encoder–decoder network. Specifically, the encoder part compresses the coded measurement into a low-dimensional representation that removes most redundancy, and the decoder part constructs the depth map directly from the latent representation. The attention mechanism enables the network to process spatial and angular features dynamically and effectively, thus significantly improving performance. Extensive experiments on synthetic and real-world datasets show that our method outperforms the state-of-the-art light field depth estimation method designed for non-coded light fields. To our knowledge, this is the first study that combines the color-coded light field with the attention-based deep learning approach, which provides a crucial insight into the design of enhanced light field photography systems.

Hybrid Attention-Based Encoder–Decoder Fully Convolutional Network for PolSAR Image Classification

Recently, methods based on convolutional neural networks (CNNs) achieve superior performance in polarimetric synthetic aperture radar (PolSAR) image classification. However, the current CNN-based classifiers follow patch-based frameworks, which need input images to be divided into overlapping patches. Consequently, these classification approaches have the drawback of requiring repeated calculations and only relying on local information. In addition, the receptive field size in conventional CNN-based methods is fixed, which limits the potential to extract features. In this paper, a hybrid attention-based encoder–decoder fully convolutional network (HA-EDNet) is presented for PolSAR classification. Unlike traditional CNN-based approaches, the encoder–decoder fully convolutional network (EDNet) can use an arbitrary-size image as input without dividing. Then, the output is the whole image classification result. Meanwhile, the self-attention module is used to establish global spatial dependence and extract context characteristics, which can improve the performance of classification. Moreover, an attention-based selective kernel module (SK module) is included in the network. In the module, softmax attention is employed to fuse several branches with different receptive field sizes. Consequently, the module can capture features with different scales and further boost classification accuracy. The experiment results demonstrate that the HA-EDNet achieves superior performance compared to CNN-based and traditional fully convolutional network methods.

A CTC Alignment-Based Non-Autoregressive Transformer for End-to-End Automatic Speech Recognition

Recently, end-to-end models have been widely used in automatic speech recognition (ASR) systems. Two of the most representative approaches are connectionist temporal classification (CTC) and attention-based encoder-decoder (AED) models. Autoregressive transformers, variants of AED, adopt an autoregressive mechanism for token generation and thus are relatively slow during inference. In this paper, we present a comprehensive study of a CTC Alignment-based Single-Step Non-Autoregressive Transformer (CASS-NAT) for end-to-end ASR. In CASS-NAT, word embeddings in the autoregressive transformer (AT) are substituted with token-level acoustic embeddings (TAE) that are extracted from encoder outputs with the acoustical boundary information offered by the CTC alignment. TAE can be obtained in parallel, resulting in a parallel generation of output tokens. During training, Viterbi-alignment is used for TAE generation, and multiple training strategies are further explored to improve the word error rate (WER) performance. During inference, an error-based alignment sampling method is investigated in depth to reduce the alignment mismatch in the training and testing processes. Experimental results show that the CASS-NAT has a WER that is close to AT on various ASR tasks, while providing a ~24x inference speedup. With and without self-supervised learning, we achieve new state-of-the-art results for non-autoregressive models on several datasets. We also analyze the behavior of the CASS-NAT decoder to explain why it can perform similarly to AT. We find that TAEs have similar functionality to word embeddings for grammatical structures, which might indicate the possibility of learning some semantic information from TAEs without a language model.

Alignment Knowledge Distillation for Online Streaming Attention-Based Speech Recognition

This article describes an efficient training method for online streaming attention-based encoder-decoder (AED) automatic speech recognition (ASR) systems. AED models have achieved competitive performance in offline scenarios by jointly optimizing all components. They have recently been extended to an online streaming framework via models such as monotonic chunkwise attention (MoChA). However, the elaborate attention calculation process is not robust for long-form speech utterances. Moreover, the sequence-level training objective and time-restricted streaming encoder cause a nonnegligible delay in token emission during inference. To address these problems, we propose CTC synchronous training (CTC-ST), in which CTC alignments are leveraged as a reference for token boundaries to enable a MoChA model to learn optimal monotonic input-output alignments. We formulate a purely end-to-end training objective to synchronize the boundaries of MoChA to those of CTC. The CTC model shares an encoder with the MoChA model to enhance the encoder representation. Moreover, the proposed method provides alignment information learned in the CTC branch to the attention-based decoder. Therefore, CTC-ST can be regarded as self-distillation of alignment knowledge from CTC to MoChA. Experimental evaluations on a variety of benchmark datasets show that the proposed method significantly reduces recognition errors and emission latency simultaneously, especially for long-form and noisy speech. We also compare CTC-ST with several methods that distill alignment knowledge from a hybrid ASR system and show that the CTC-ST can achieve a comparable tradeoff of accuracy and latency without relying on external alignment information. The best MoChA system shows performance comparable to that of RNN-transducer (RNN-T).