Bidirectional Long Short-term Memory Research Articles

Leveraging Hybrid Deep Learning Models for Enhanced Multivariate Time Series Forecasting

Time series forecasting is crucial in various domains, ranging from finance and economics to weather prediction and supply chain management. Traditional statistical methods and machine learning models have been widely used for this task. However, they often face limitations in capturing complex temporal dependencies and handling multivariate time series data. In recent years, deep learning models have emerged as a promising solution for overcoming these limitations. This paper investigates how deep learning, specifically hybrid models, can enhance time series forecasting and address the shortcomings of traditional approaches. This dual capability handles intricate variable interdependencies and non-stationarities in multivariate forecasting. Our results show that the hybrid models achieved lower error rates and higher R2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$R^2$$\\end{document} values, signifying their superior predictive performance and generalization capabilities. These architectures effectively extract spatial features and temporal dynamics in multivariate time series by combining convolutional and recurrent modules. This study evaluates deep learning models, specifically hybrid architectures, for multivariate time series forecasting. On two real-world datasets - Traffic Volume and Air Quality - the TCN-BiLSTM model achieved the best overall performance. For Traffic Volume, the TCN-BiLSTM model achieved an R2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$R^2$$\\end{document} score of 0.976, and for Air Quality, it reached an R2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$R^2$$\\end{document} score of 0.94. These results highlight the model’s effectiveness in leveraging the strengths of Temporal Convolutional Networks (TCNs) for capturing multi-scale temporal patterns and Bidirectional Long Short-Term Memory (BiLSTMs) for retaining contextual information, thereby enhancing the accuracy of time series forecasting.

Robust In-Vehicle Signal Quality Assessment Using Multimodal Signal Fusion.

Continuous monitoring of physiological signals such as electrocardiogram (ECG) in driving environments has the potential to reduce the need for frequent health check-ups by providing real-time information on cardiovascular health. However, capturing ECG from sensors mounted on steering wheels creates difficulties due to motion artifacts, noise, and dropouts. To address this, we propose a novel method for reliable and accurate detection of heartbeats using sensor fusion with a bidirectional long short-term memory (BiLSTM) model. Our dataset contains reference ECG, steering wheel ECG, photoplethysmogram (PPG), and imaging PPG (iPPG) signals, which are more feasible to capture in driving scenarios. We combine these signals for R-wave detection. We conduct experiments with individual signals and signal fusion techniques to evaluate the performance of detected heartbeat positions. The BiLSTMs model achieves a performance of 62.69% in the driving scenario city. The model can be integrated into the system to detect heartbeat positions for further analysis.

DeepTimeNet: A novel architecture for precise surface temperature estimation of lithium-ion batteries across diverse ambient conditions

With the growing demand for battery-powered devices and electric vehicles, the need for improved battery performance and safety is paramount. A key determinant of battery health is the accurate monitoring of surface temperature (ST). Conventional ST estimation often depends on direct sensor measurements, which may not be cost-effective and can impact system reliability. This paper presents DeepTimeNet, a novel approach leveraging deep learning (DL) architectures for sensorless ST prediction in lithium-ion batteries. DeepTimeNet combines Convolutional Neural Networks (CNN), ResNet blocks, Inception modules, Bidirectional LSTM, and GRU layers to precisely model the time-dependent behaviour of batteries. A comprehensive evaluation against traditional models, across temperatures ranging from -20 °C to 25 °C and under various driving profiles, including US06 and Urban Dynamometer Driving Schedule (UDDS), is conducted. DeepTimeNet’s performance is quantified by metrics such as mean absolute error (MAE), surpassing that of models like Gated Recurrent Unit-Recurrent Neural Network (GRU-RNN), Convolutional Neural Network-Long Short Term Memory Network (CNN-LSTM), and Long Short Term Memory Network (LSTM). The results demonstrate DeepTimeNet’s superior performance, with an RMSE of 0.0971, MSE of 0.0099, MAE of 0.0912, and MAXE of 0.3963, validating it as an advanced tool for enhancing the efficacy of battery management systems and underscoring its potential as a benchmark for future innovations.

AI-Driven Model for Contract Law Cases

Abstract: The subjective nature of human analysis results in inconsistent decision-making, which seriously jeopardizes the accuracy and fairness of the verdicts in contract disputes. Legal practitioners confront the difficult task of organizing and evaluating numerous precedent/statutes case materials promptly as the amount of contracts keeps increasing. This time constraint not only makes it more difficult to resolve contract issues on time but also makes the legal system more complicated and unclear. There has never been a greater need for contract litigation to undergo an extensive change. The researcher proposed two complementary methods for retrieving legal documents: BM25 and an aggregated Bidirectional Long Short-Term Memory (BiLSTM) model with a Convolution Neural Network (CNN)

A novel lithium-ion battery state-of-health estimation method for fast-charging scenarios based on an improved multi-feature extraction and bagging temporal attention network

Accurately estimating the state of health (SOH) of fast-charging lithium-ion batteries is crucial for safely and reliably operating battery systems. However, handling data scarcity and rapid charging scenarios under diverse operational conditions is challenging. In this paper, a novel approach for estimating the SOH of lithium-ion batteries (LIBs) is introduced based on an improved multisegment feature extraction and bagging temporal attention network. First, four health features, including the time differences observed at equal voltages, the cumulative integral of voltage changes, the total circuit charge variation and the levels of slope peaks, are identified. Second, a residual bidirectional long short-term memory (Bi-LSTM) attention mechanism is designed to focus on the temporal dimension of battery data by incorporating a multilayered complex neural network design comprising convolutional layers, pooling layers, Bi-LSTM layers and fully connected layers. This design effectively captures the features and relationships contained in battery data. The predictions derived from randomly initialized parameters and multiple submodels are stacked to improve the generalizability of the model. Finally, comprehensive evaluations are conducted through comparative experiments, ablation studies and noise experiments, with six evaluation metrics considered across three datasets. The MAE, RMSE, MAEP and MAXE of the proposed model reach 0.366, 0.605, 0.519 and 1.56, respectively. The results indicate that the proposed method enhances the robustness, resilience and generalizability of the estimates produced under different conditions and noise scenarios.

Short-Term Power Load Forecasting Based on Secondary Cleaning and CNN-BILSTM-Attention

Accurate power load forecasting can provide crucial insights for power system scheduling and energy planning. In this paper, to address the problem of low accuracy of power load prediction, we propose a method that combines secondary data cleaning and adaptive variational mode decomposition (VMD), convolutional neural networks (CNN), bi-directional long short-term memory (BILSTM), and adding attention mechanism (AM). The Inner Mongolia electricity load data were first cleaned use the K-means algorithm, and then further refined with the density-based spatial clustering of applications with the noise (DBSCAN) algorithm. Subsequently, the parameters of the VMD algorithm were optimized using a multi-strategy Cubic-T dung beetle optimization algorithm (CTDBO), after which the VMD algorithm was employed to decompose the twice-cleaned load sequences into a number of intrinsic mode functions (IMFs) with different frequencies. These IMFs were then used as inputs to the CNN-BILSTM-Attention model. In this model, a CNN is used for feature extraction, BILSTM for extracting information from the load sequence, and AM for assigning different weights to different features to optimize the prediction results. It is proved experimentally that the model proposed in this paper achieves the highest prediction accuracy and robustness compared to other models and exhibits high stability across different time periods.

Crested Porcupine Optimizer-Optimized CNN-BiLSTM-Attention Model for Predicting Main Girder Temperature in Bridges

Stage-built long-span bridges deform with temperature, affecting alignment to design needs. In this paper, a model for predicting temperature time series is proposed, which can predict temperatures in engineering practice and utilize the predicted results to adjust the elevation of stage construction. The model employs convolutional neural networks (CNNs) for initial feature extraction, followed by bidirectional long short-term memory (BiLSTM) layers to capture temporal dependencies. An attention mechanism is applied to the LSTM output, enhancing the model’s ability to focus on the most relevant parts of the sequence. The Crested Porcupine Optimizer (CPO) is used to fine-tune parameters like the number of LSTM units, dropout rate, and learning rate. The experiments on the measured temperature data of an under-construction cable-stayed bridge are conducted to validate our model. The results indicate that our model outperforms the other five models in comparison, with all the R2 values exceeding 0.97. The average of the mean absolute error (MAE) on the 30 measure points is 0.19095, and the average of the root mean square error (RMSE) is 0.28283. Furthermore, the model’s low sensitivity to data makes it adaptable and effective for predicting temperatures and adjusting the elevation in large-span bridge construction.

Pipeline Leak Detection System for a Smart City: Leveraging Acoustic Emission Sensing and Sequential Deep Learning

This study explores a novel approach utilizing acoustic emission (AE) signaling technology for pipeline leakage detection and analysis. Pipeline leaks are a significant concern in the liquids and gases industries, prompting the development of innovative detection methods. Unlike conventional methods, which often require contact and visual inspection with the pipeline surface, the proposed time-series-based deep learning approach offers real-time detection with higher safety and efficiency. In this study, we propose an automatic detection system of pipeline leakage for efficient transportation of liquid (water) and gas across the city, considering the smart city approach. We propose an AE-based framework combined with time-series deep learning algorithms to detect pipeline leaks through time-series features. The time-series AE signal detection module is designed to capture subtle changes in the AE signal state caused by leaks. Sequential deep learning models, including long short-term memory (LSTM), bi-directional LSTM (Bi-LSTM), and gated recurrent units (GRUs), are used to classify the AE response into normal and leakage detection from minor seepage, moderate leakage, and major ruptures in the pipeline. Three AE sensors are installed at different configurations on a pipeline, and data are acquired at 1 MHz sample/sec, which is decimated to 4K sample/second for efficiently utilizing the memory constraints of a remote system. The performance of these models is evaluated using metrics, namely accuracy, precision, recall, F1 score, and convergence, demonstrating classification accuracies of up to 99.78%. An accuracy comparison shows that BiLSTM performed better mostly with all hyperparameter settings. This research contributes to the advancement of pipeline leakage detection technology, offering improved accuracy and reliability in identifying and addressing pipeline integrity issues.

Comparison of deep learning algorithms for site detection of false data injection attacks in smart grids

False Data Injection Attacks (FDIA) pose a significant threat to the stability of smart grids. Traditional Bad Data Detection (BDD) algorithms, deployed to remove low-quality data, can easily be bypassed by these attacks which require minimal knowledge about the parameters of the power bus systems. This makes it essential to develop defence approaches that are generic and scalable to all types of power systems. Deep learning algorithms provide state-of-the-art detection for FDIA while requiring no knowledge about system parameters. However, there are very few works in the literature that evaluate these models for FDIA detection at the level of an individual node in the power system. In this paper, we compare several recent deep learning-based model that proven their high performance and accuracy in detecting the exact location of the attack node, which are convolutional neural networks (CNN), Long Short-Term Memory (LSTM), attention-based bidirectional LSTM, and hybrid models. We, then, compare their performance with baseline multi-layer perceptron (MLP)., All the models are evaluated on IEEE-14 and IEEE-118 bus systems in terms of row accuracy (RACC), computational time, and memory space required for training the deep learning model. Each model was further investigated through a manual grid search to determine the optimal architecture of the deep learning model, including the number of layers and neurons in each layer. Based on the results, CNN model exhibited consistently high performance in very short training time. LSTM achieved the second highest accuracy; however, it had required an averagely higher training time. The attention-based LSTM model achieved a high accuracy of 94.53 during hyperparameter tuning, while the CNN model achieved a moderately lower accuracy with only one-fourth of the training time. Finally, the performance of each model was quantified on different variants of the dataset—which varied in their l2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${\ ext{l}}_{2}$$\\end{document}-norm. Based on the results, LSTM, CNN obtained the highest accuracy followed by CNN-LSTM and lastly MLP.

An optimized model based on adaptive convolutional neural network and grey wolf algorithm for breast cancer diagnosis.

Medical image classification (IC) is a method for categorizing images according to the appropriate pathological stage. It is a crucial stage in computer-aided diagnosis (CAD) systems, which were created to help radiologists with reading and analyzing medical images as well as with the early detection of tumors and other disorders. The use of convolutional neural network (CNN) models in the medical industry has recently increased, and they achieve great results at IC, particularly in terms of high performance and robustness. The proposed method uses pre-trained models such as Dense Convolutional Network (DenseNet)-121 and Visual Geometry Group (VGG)-16 as feature extractor networks, bidirectional long short-term memory (BiLSTM) layers for temporal feature extraction, and the Support Vector Machine (SVM) and Random Forest (RF) algorithms to perform classification. For improved performance, the selected pre-trained CNN hyperparameters have been optimized using a modified grey wolf optimization method. The experimental analysis for the presented model on the Mammographic Image Analysis Society (MIAS) dataset shows that the VGG16 model is powerful for BC classification with overall accuracy, sensitivity, specificity, precision, and area under the ROC curve (AUC) of 99.86%, 99.9%, 99.7%, 97.1%, and 1.0, respectively, on the MIAS dataset and 99.4%, 99.03%, 99.2%, 97.4%, and 1.0, respectively, on the INbreast dataset.

Understanding the Dynamics of Ocean Wave-Current Interactions Through Multivariate Multi-Step Time Series Forecasting

ABSTRACT Understanding ocean wave-current interactions’ complex dynamics is crucial for coastal engineering, marine operations, and climate research applications. This study introduces a pioneering data-driven approach by employing advanced deep learning techniques, specifically Long Short-Term Memory (LSTM), and Bidirectional LSTM (BiLSTM) models, to forecast both wave and current parameters at varying depths. The models are designed to capture the complex temporal relationships inherent in ocean dynamics, considering wave speed and direction, current speed, and direction as multivariate time series inputs. Two comprehensive experiments are conducted, one utilizing historical values of all parameters and another focusing on using wave parameters to forecast current parameters. Model performance is rigorously evaluated across forecast horizons of 5, 12, and 24 hours ahead using metrics such as Mean Absolute Error (MAE), Mean Squared Error (MSE), and Root Mean Squared Error (RMSE). BiLSTM emerges as the superior model, demonstrating lower errors, particularly at higher depths, while nearshore predictions reveal challenges in shallower waters. Furthermore, the methodology incorporates hyperparameter optimization and cross-validation techniques to enhance the model’s robustness. Ultimately, this work represents a transformative leap toward smarter oceans, emphasizing the fusion of fluid dynamics and bathymetry to advance our understanding of coupled wave-current dynamics. The results showcase high accuracy and reliability across various forecast horizons and depths, signifying the method’s potential applications in oceanography, hydrodynamics, coastal engineering, and ocean renewable energy.

Phase-Based Gait Prediction after Botulinum Toxin Treatment Using Deep Learning.

Gait disorders in neurological diseases are frequently associated with spasticity. Intramuscular injection of Botulinum Toxin Type A (BTX-A) can be used to treat spasticity. Providing optimal treatment with the highest possible benefit-risk ratio is a crucial consideration. This paper presents a novel approach for predicting knee and ankle kinematics after BTX-A treatment based on pre-treatment kinematics and treatment information. The proposed method is based on a Bidirectional Long Short-Term Memory (Bi-LSTM) deep learning architecture. Our study's objective is to investigate this approach's effectiveness in accurately predicting the kinematics of each phase of the gait cycle separately after BTX-A treatment. Two deep learning models are designed to incorporate categorical medical treatment data corresponding to the injected muscles: (1) within the hidden layers of the Bi-LSTM network, (2) through a gating mechanism. Since several muscles can be injected during the same session, the proposed architectures aim to model the interactions between the different treatment combinations. In this study, we conduct a comparative analysis of our prediction results with the current state of the art. The best results are obtained with the incorporation of the gating mechanism. The average prediction root mean squared error is 2.99° (R2 = 0.85) and 2.21° (R2 = 0.84) for the knee and the ankle kinematics, respectively. Our findings indicate that our approach outperforms the existing methods, yielding a significantly improved prediction accuracy.

Fusion-based modeling of an intelligent algorithm for enhanced object detection using a Deep Learning Approach on radar and camera data

Object detection, the process of detecting and classifying objects within a given environment, forms the foundational element. Multisensory fusion incorporates data from diverse sensors, like radar and cameras, to refine the reliability and accuracy of detection. Further, Radar and camera data fusion refine this process by integrating the unique strength of both technologies, which leverage the radar's proficiency in adverse weather conditions and the camera's high-resolution imaging. This incorporation enhances the object detection systems, which enables them to effectively operate across the spectrum of scenarios, from autonomous vehicles navigating challenging weather to surveillance systems monitoring critical infrastructure. Deep learning (DL), a branch of machine learning (ML), empowers this system with the capability to learn complex representations and patterns directly from the data, which enables them to generalize and adapt to new situations. By integrating the advanced methodology, we can develop strong perception system capable of interpreting and detecting objects accurately in dynamic and diverse environments, from autonomous vehicles navigating urban landscapes to surveillance systems monitoring complex environments. This study designs an Intelligent Algorithm for Enhanced Object Detection Using Deep Learning Approach on the Radar and Camera Data Fusion (IAEOD-DLRCDF) technique. The presented IAEOD-DLRCDF technique uses multi-angle joint calibration where the spatial sparse alignment of the heterogeneous data of the camera and Radar is realized with image falsification disregarded. Besides, the IAEOD-DLRCDF technique applies YOLOv8 object detector for radar and camera target detection individually which are then integrated with the image plane. Moreover, the detected objects are then classified via the bidirectional long short-term memory (BiLSTM) model. Furthermore, the Adam optimizer is used for the optimum hyperparameter selection of the BiLSTM network which results in a better recognition rate. The performance assessment of the IAEOD-DLRCDF method is tested under benchmark dataset. The empirical analysis stated that the IAEOD-DLRCDF method gains better performance over other models.

COVID-19 IgG antibodies detection based on CNN-BiLSTM algorithm combined with fiber-optic dataset

The urgent need for efficient and accurate automated screening tools for COVID-19 detection has led to research efforts exploring various approaches. In this study, we present pioneering research on COVID-19 detection using a hybrid model that combines convolutional neural networks (CNN) with a bi-directional long short-term memory (Bi-LSTM) network, in conjunction with fiber optic data for SARS-CoV-2 Immunoglobulin G (IgG) antibodies. Our research introduces a comprehensive data preprocessing pipeline and evaluates the performance of four different deep learning (DL) algorithms: CNN, CNN-RNN, BiLSTM, and CNN-BiLSTM, in classifying samples as positive or negative for the COVID-19 virus. Among these, the CNN-BiLSTM classifier demonstrated superior performance on the training datasets, achieving an accuracy of 89 %, a recall of 88 %, a precision of 90 %, an F1-score of 89 %, a specificity of 90 %, a geometric mean (G-mean) of 89 %, and a receiver operating characteristic (ROC) of 96 %. In addition, the achieved classification results were compared with those reported in the literature. The findings indicate that the proposed model has promising potential for classifying COVID-19 and could serve as a valuable tool for healthcare professionals. The use of IgG antibodies to detect the virus enhances the specificity and accuracy of the diagnostic tool.

Is the U.S. energy independence and Security Act of 2022 associated with stock market volatility?

This study investigates the influence of the U.S. Energy Independence and Security Act (EISA) of 2022 on stock market volatility. Considering potential policy impacts on market dynamics, it employs a mean-reverting jump-diffusion model for volatility analysis, supported by rigorous hypothesis testing to validate empirical findings. For investor sentiment analysis, a pre-trained model (BERT-base-multilingual-uncased) is used to assess sentiment, with logistic regression and Bidirectional LSTM utilized for sentiment prediction. Contrary to conventional findings, logistic regression outperforms Bidirectional LSTM for the small dataset. Overall, the study estimates show no statistically noteworthy difference in volatility before and after implementing the EISA of 2022. This study provides valuable insights for navigating financial decisions within the intricate landscape of energy markets.

A comparative analysis of classical and machine learning methods for forecasting TB/HIV co-infection

TB/HIV coinfection poses a complex public health challenge. Accurate forecasting of future trends is essential for efficient resource allocation and intervention strategy development. This study compares classical statistical and machine learning models to predict TB/HIV coinfection cases stratified by gender and the general populations. We analyzed time series data using exponential smoothing and ARIMA to establish the baseline trend and seasonality. Subsequently, machine learning models (SVR, XGBoost, LSTM, CNN, GRU, CNN-GRU, and CNN-LSTM) were employed to capture the complex dynamics and inherent non-linearities of TB/HIV coinfection data. Performance metrics (MSE, MAE, sMAPE) and the Diebold-Mariano test were used to evaluate the model performance. Results revealed that Deep Learning models, particularly Bidirectional LSTM and CNN-LSTM, significantly outperformed classical methods. This demonstrates the effectiveness of Deep Learning for modeling TB/HIV coinfection time series and generating more accurate forecasts.

M2PSC: Multilingual sentiment analysis using improved multi-attention based Deep Learning model

Sentiment analysis, which involves determining the emotional polarity positivity, negativity, or neutrality in the source texts, is a crucial task. Multilingual sentiment analysis techniques were developed to analyze data in several languages; a notable deficiency of resources in multilingual sentiment analysis is one of the primary issues. Furthermore, the developed methods for multilingual sentiment analysis have some limitations such as data dependency, reliability, robustness, and computational complexity. To tackle these shortcomings, this research proposed a multilingual improved multi-attention Deep Learning model (M2PSC-DL), which leverages the advantages of the Bi-directional Long Short Term Memory (BiLSTM) classifier with improved attention mechanisms. The Multi-metric graph embedding technique encodes the data to provide more contextual information representation. Additionally, the combination of improved Positional Spatial Channel (SPC) attention increases the capability of the model to extract relevant features in the training process which leads to getting accurate results in sentiment analysis tasks. Furthermore, the research proposed an improved sigmoid activation for solving the vanishing gradient issues that help the model avoid gradient saturations. The validation results demonstrate that the M2PSC-DL model attains 96.26% accuracy, 96.06% precision, and 96.18% recall for the XED dataset which is far better than the traditional methods.

Exploring the Processing Paradigm of Input Data for End-to-End Deep Learning in Tool Condition Monitoring.

Tool condition monitoring technology is an indispensable part of intelligent manufacturing. Most current research focuses on complex signal processing techniques or advanced deep learning algorithms to improve prediction performance without fully leveraging the end-to-end advantages of deep learning. The challenge lies in transforming multi-sensor raw data into input data suitable for direct model feeding, all while minimizing data scale and preserving sufficient temporal interpretation of tool wear. However, there is no clear reference standard for this so far. In light of this, this paper innovatively explores the processing methods that transform raw data into input data for deep learning models, a process known as an input paradigm. This paper introduces three new input paradigms: the downsampling paradigm, the periodic paradigm, and the subsequence paradigm. Then an improved hybrid model that combines a convolutional neural network (CNN) and bidirectional long short-term memory (BiLSTM) was employed to validate the model's performance. The subsequence paradigm demonstrated considerable superiority in prediction results based on the PHM2010 dataset, as the newly generated time series maintained the integrity of the raw data. Further investigation revealed that, with 120 subsequences and the temporal indicator being the maximum value, the model's mean absolute error (MAE) and root mean square error (RMSE) were the lowest after threefold cross-validation, outperforming several classical and contemporary methods. The methods explored in this paper provide references for designing input data for deep learning models, helping to enhance the end-to-end potential of deep learning models, and promoting the industrial deployment and practical application of tool condition monitoring systems.

Forecasting energy consumption with a novel ensemble deep learning framework

Energy baseline development is a central and significant component of an energy management system. In commercial buildings where a wide variety of internal and external variables might impact the electricity usage, having a robust forecasting method is elemental to evaluate and enhance the energy performance over time. Due to extensive progress in data analytics, a number of single prediction methods have been developed recently. Nevertheless, either development of the theoretical study or application of ensemble learning methods to enhance prediction performance and stability find lacking in energy management of commercial sectors. This research proposes a novel heterogeneous ensemble model framework for short-term electricity consumption prediction under the hourly and daily granularity to bridge this research gap. The framework is composed of two levels. The first uses five stand-alone deep learning models including Recurrent Neural Network (RNN), Long Short-term Memory (LSTM), Bidirectional-LSTM (Bi-LSTM), Gated Recurrent Unit (GRU) and Convolutional Neural Network (CNN) as base models. In contrast, the second level applies the predicted outputs of the base models as feature inputs of k nearest neighbors (KNN), which is a strong model. The Grid Search is also used to search for the key optimal hyper parameters of the base models and the strong model. To demonstrate the developed model's effectiveness, four different space types of real-world educational buildings are used for verification. The extensive database has been setup for this study to collect data from Jan. 2018 to Dec. 2022. The quantitative analysis of the model was conducted in contrast with the base models and prevalent machine learning methods, including artificial neural network (ANN), KNN, support vector regression (SVR), AdaBoost, and linear regression (LR) during the testing phase. The prediction accuracy has improved substantially, with roughly a 95 % reduction in CV-RMSE compared to that of AdaBoost and SVR. Moreover, the results signify the introduced model outperforms the base models and the ML methods in all studied buildings, thereby providing a foundation for developing effective management policies.

A novel model based on a transformer for intent detection and slot filling

Building task-oriented dialogue systems has become a topic of interest in the research community and industry. The task-oriented dialogue system is a closed-domain dialogue system that can perform specific tasks for users. The natural language understanding module of a task-oriented dialogue system is crucial because it is related to a task-oriented dialogue system that provides correctional services for users. The natural language understanding module of a task-oriented dialogue system performs two tasks: intent detection and slot filling. The intent detection task can be regarded as a text classification task; a classification model is trained to predict the intention of the user from the user’s input information. The slot filling task can be regarded as a sequence analysis task; a sequence analysis model is trained to predict the details of the user’s intention. In this paper, we proposed a novel model based on a transformer encoder for intent detection and slot filling. It follows the encoder-decoder structure, including a vanilla Transformer encoder, a bidirectional LSTM encoder, a linear classification decoder for intent detection, and a conditional random field decoder for slot filling. The experimental results on two public datasets show that our proposed model outperforms the existing methods based on the Transformer and can be combined with BERT to achieve better intent detection and slot filling results.