Recurrent Neural Network Architecture Research Articles

Instrumental Variable Estimation for Causal Inference in Longitudinal Data with Time-Dependent Latent Confounders

Causal inference from longitudinal observational data is a challenging problem due to the difficulty in correctly identifying the time-dependent confounders, especially in the presence of latent time-dependent confounders. Instrumental variable (IV) is a powerful tool for addressing the latent confounders issue, but the traditional IV technique cannot deal with latent time-dependent confounders in longitudinal studies. In this work, we propose a novel Time-dependent Instrumental Factor Model (TIFM) for time-varying causal effect estimation from data with latent time-dependent confounders. At each time-step, the proposed TIFM method employs the Recurrent Neural Network (RNN) architecture to infer latent IV, and then uses the inferred latent IV factor for addressing the confounding bias caused by the latent time-dependent confounders. We provide a theoretical analysis for the proposed TIFM method regarding causal effect estimation in longitudinal data. Extensive evaluation with synthetic datasets demonstrates the effectiveness of TIFM in addressing causal effect estimation over time. We further apply TIFM to a climate dataset to showcase the potential of the proposed method in tackling real-world problems.

Beyond Word-Based Model Embeddings: Contextualized Representations for Enhanced Social Media Spam Detection

As social media platforms continue their exponential growth, so do the threats targeting their security. Detecting disguised spam messages poses an immense challenge owing to the constant evolution of tactics. This research investigates advanced artificial intelligence techniques to significantly enhance multiplatform spam classification on Twitter and YouTube. The deep neural networks we use are state-of-the-art. They are recurrent neural network architectures with long- and short-term memory cells that are powered by both static and contextualized word embeddings. Extensive comparative experiments precede rigorous hyperparameter tuning on the datasets. Results reveal a profound impact of tailored, platform-specific AI techniques in combating sophisticated and perpetually evolving threats. The key innovation lies in tailoring deep learning (DL) architectures to leverage both intrinsic platform contexts and extrinsic contextual embeddings for strengthened generalization. The results include consistent accuracy improvements of more than 10–15% in multisource datasets, unlocking actionable guidelines on optimal components of neural models, and embedding strategies for cross-platform defense systems. Contextualized embeddings like BERT and ELMo consistently outperform their noncontextualized counterparts. The standalone ELMo model with logistic regression emerges as the top performer, attaining exceptional accuracy scores of 90% on Twitter and 94% on YouTube data. This signifies the immense potential of contextualized language representations in capturing subtle semantic signals vital for identifying disguised spam. As emerging adversarial attacks exploit human vulnerabilities, advancing defense strategies through enhanced neural language understanding is imperative. We recommend that social media companies and academic researchers build on contextualized language models to strengthen social media security. This research approach demonstrates the immense potential of personalized, platform-specific DL techniques to combat the continuously evolving threats that threaten social media security.

A Recursively Recurrent Neural Network (R2N2) Architecture for Learning Iterative Algorithms

A Recursively Recurrent Neural Network (R2N2) Architecture for Learning Iterative Algorithms

Breaking Time Invariance: Assorted-Time Normalization for RNNs

Methods such as Layer Normalization (LN) and Batch Normalization have proven to be effective in improving the training of Recurrent Neural Networks (RNNs). However, existing methods normalize using only the instantaneous information at one particular time step, and the result of the normalization is a preactivation state with a time-independent distribution. This implementation fails to account for certain temporal differences inherent in the inputs and the architecture of RNNs. Since these networks share weights across time steps, it may also be desirable to account for the connections between time steps in the normalization scheme. In this paper, we propose a normalization method called Assorted-Time Normalization (ATN), which preserves information from multiple consecutive time steps and normalizes using them. This setup allows us to introduce longer time dependencies into the traditional normalization methods without introducing any new trainable parameters. We present theoretical derivations for the gradient propagation and prove the weight scaling invariance property. Our experiments applying ATN to LN demonstrate consistent improvement on various tasks, such as Adding, Copying, and Denoise Problems and Language Modeling Problems.

Predicting Melting Temperatures of Freeze Valve in Molten Salt Reactors using LSTM-Based Deep Learning Models

The realm of molten salt reactors (MSR) presents unique challenges and opportunities in the pursuit of advancing nuclear energy technology. One pivotal aspect is the development of safety mechanisms, including freeze valves, aimed at enhancing the security and operational integrity of these reactors. Freeze valves, designed to halt the flow of molten salt in emergency scenarios, play a critical role in averting potential hazardous situations. This research paper investigates the application of deep learning models for predicting melting temperatures, leveraging time-series temperature data, specifically targeting the freeze valve technology within the context of molten salt reactors. The research methodology encompasses the creation of a synthetic dataset representative of freeze valve conditions in MSR. Each dataset entry comprises a time-series temperature profile associated with the freeze valve and its corresponding melting temperature. To capture the intricate temporal relationships inherent in these profiles, a specialized Recurrent Neural Network (RNN) architecture, incorporating Long Short-Term Memory (LSTM) cells, is developed and trained. The model’s training is augmented by preprocessing techniques, including the normalization of time-series temperature data. Evaluation of the model’s predictive capabilities focuses on its performance in forecasting the melting temperatures of the freeze valve. Metrics such as mean squared error (MSE) and mean absolute error (MAE) are utilized to quantify the model’s accuracy and robustness. The study delves into the influence of architectural configurations, hyperparameter tuning, and dataset characteristics on the model’s predictions, shedding light on the factors that contribute to optimal predictive performance. The outcomes of this research highlight the potential of deep learning models in predicting the melting temperatures of freeze valves in the context of MSR. The developed model not only showcases an ability to anticipate impending safety-critical events but also underscores the significance of leveraging time-series temperature data to enhance the integrity of safety mechanisms in nuclear reactor systems. Ultimately, this research contributes to the pursuit of safer and more efficient nuclear energy solutions, offering valuable insights into predictive modeling techniques within the complex landscape of molten salt reactor technologies.

Advanced Deep Learning Model for Predicting the Academic Performances of Students in Educational Institutions

Educational institutions are increasingly focused on supporting students who may be facing academic challenges, aiming to enhance their educational outcomes through targeted interventions. Within this framework, leveraging advanced deep learning techniques to develop recommendation systems becomes essential. These systems are designed to identify students at risk of underperforming by analyzing patterns in their historical academic data, thereby facilitating personalized support strategies. This research introduces an innovative deep learning model tailored for pinpointing students in need of academic assistance. Utilizing a Gated Recurrent Neural Network (GRU) architecture, the model is rich with features such as a dense layer, max-pooling layer, and the ADAM optimization method used to optimize performance. The effectiveness of this model was tested using a comprehensive dataset containing 15,165 records of student assessments collected across several academic institutions. A comparative analysis with existing educational recommendation models, like Recurrent Neural Network (RNN), AdaBoost, and Artificial Immune Recognition System v2, highlights the superior accuracy of the proposed GRU model, which achieved an impressive overall accuracy of 99.70%. This breakthrough underscores the model’s potential in aiding educational institutions to proactively support students, thereby mitigating the risks of underachievement and dropout.

A Systematic Evaluation of Recurrent Neural Network Models for Edge Intelligence and Human Activity Recognition Applications

The Recurrent Neural Networks (RNNs) are an essential class of supervised learning algorithms. Complex tasks like speech recognition, machine translation, sentiment classification, weather prediction, etc., are now performed by well-trained RNNs. Local or cloud-based GPU machines are used to train them. However, inference is now shifting to miniature, mobile, IoT devices and even micro-controllers. Due to their colossal memory and computing requirements, mapping RNNs directly onto resource-constrained platforms is arcane and challenging. The efficacy of edge-intelligent RNNs (EI-RNNs) must satisfy both performance and memory-fitting requirements at the same time without compromising one for the other. This study’s aim was to provide an empirical evaluation and optimization of historic as well as recent RNN architectures for high-performance and low-memory footprint goals. We focused on Human Activity Recognition (HAR) tasks based on wearable sensor data for embedded healthcare applications. We evaluated and optimized six different recurrent units, namely Vanilla RNNs, Long Short-Term Memory (LSTM) units, Gated Recurrent Units (GRUs), Fast Gated Recurrent Neural Networks (FGRNNs), Fast Recurrent Neural Networks (FRNNs), and Unitary Gated Recurrent Neural Networks (UGRNNs) on eight publicly available time-series HAR datasets. We used the hold-out and cross-validation protocols for training the RNNs. We used low-rank parameterization, iterative hard thresholding, and spare retraining compression for RNNs. We found that efficient training (i.e., dataset handling and preprocessing procedures, hyperparameter tuning, and so on, and suitable compression methods (like low-rank parameterization and iterative pruning) are critical in optimizing RNNs for performance and memory efficiency. We implemented the inference of the optimized models on Raspberry Pi.

RNN-BiLSTM-CRF based amalgamated deep learning model for electricity theft detection to secure smart grids.

Electricity theft presents a substantial threat to distributed power networks, leading to non-technical losses (NTLs) that can significantly disrupt grid functionality. As power grids supply centralized electricity to connected consumers, any unauthorized consumption can harm the grids and jeopardize overall power supply quality. Detecting such fraudulent behavior becomes challenging when dealing with extensive data volumes. Smart grids provide a solution by enabling two-way electricity flow, thereby facilitating the detection, analysis, and implementation of new measures to address data flow issues. The key objective is to provide a deep learning-based amalgamated model to detect electricity theft and secure the smart grid. This research introduces an innovative approach to overcome the limitations of current electricity theft detection systems, which predominantly rely on analyzing one-dimensional (1-D) electric data. These approaches often exhibit insufficient accuracy when identifying instances of theft. To address this challenge, the article proposes an ensemble model known as the RNN-BiLSTM-CRF model. This model amalgamates the strengths of recurrent neural network (RNN) and bidirectional long short-term memory (BiLSTM) architectures. Notably, the proposed model harnesses both one-dimensional (1-D) and two-dimensional (2-D) electricity consumption data, thereby enhancing the effectiveness of the theft detection process. The experimental results showcase an impressive accuracy rate of 93.05% in detecting electricity theft, surpassing the performance of existing models in this domain.

Generalisation of Feed-Forward Neural Networks and Recurrent Neural Networks

This paper presents an in-depth analysis of Feed-Forward Neural Networks (FNNs) and Recurrent Neural Networks (RNNs), two powerful models in the field of artificial intelligence. Understanding these models and their applications is crucial for harnessing their potential. The study addresses the need to comprehend the unique characteristics and architectures of FNNs and RNNs. These models excel at processing sequential and temporal data, making them indispensable in tasks. Furthermore, the paper emphasises the importance of variables in FNNs and proposes a novel method to rank the importance of independent variables in predicting the output variable. By understanding the relationship between inputs and outputs, valuable insights can be gained into the underlying patterns and mechanisms driving the system being modelled. Additionally, the research explores the impact of initial weights on model performance. Contrary to conventional beliefs, the study provides evidence that neural networks with random weights can achieve competitive performance, particularly in situations with limited training datasets. This finding challenges the traditional notion that careful initialization is necessary for neural networks to perform well. In summary, this paper provides a comprehensive analysis of FNNs and RNNs while highlighting the importance of understanding the relationship between variables and the impact of initial weights on model performance. By shedding light on these crucial aspects, this research contributes to the advancement and effective utilisation of neural networks, paving the way for improved predictions and insights in various domains.

MDPose: Human Skeletal Motion Reconstruction Using WiFi Micro-Doppler Signatures

Motion tracking systems based on optical sensors typically suffer from poor lighting conditions, occlusion, limited coverage, and may raise privacy concerns. More recently, radiofrequency (RF) based approaches using commercial WiFi devices have emerged which offer low-cost ubiquitous sensing whilst preservin privacy. However, RF sensing systems typically output range-Doppler maps, time-frequency spectrograms, cross-range plots etc which cannot represent human motion intuitively and usually requires further processing. In this study, we propose MDPose, a novel framework for human skeletal motion reconstruction base on WiFi micro-Doppler signatures. MDPose provides an effective solution to represent human activity by reconstructing a skeleton model with 17 key points, which can assist with the interpretation of conventional RF sensing outputs in a more understandable way. Specifically, MDPose is implemented over three sequential stage to address a series of challenges: First, a denoising algorithm is employed to remove any unwanted noise that may affect feature extraction and enhance weak Doppler measurements. Secondly, a convolutional neural network (CNN)-recurrent neural network (RNN) architecture is applied to learn temporal spatial dependenc from clean micro-Doppler signatures and restore velocity information to key points under the supervision of the motion capture (Mocap) system. Finally, a pose optimisation mechanism based on learning optimisation vectors is employed to estimate the initial state of the skeleton and to limit additional errors. We hav conducted a comprehensive set of tests in a variety of environments using numerous subjects with a single receiver radar system to demonstrate the performance of MDPose, and report 29.4mm mean absolute error over all key points positions on several common daily activities, which has performance comparable to that of state-ofthe- art RF-based pose estimation systems.

Learning Experience of University Music Course Based on Emotional Computing

Identification and analysis of piano performance music are essential activities in the study of music knowledge and enjoyment. The goal of this study is to employ recurrent neural networks (RNNs) to create a multimedia identification and analysis method for piano performance music. RNNs excel at capturing the temporal relationships and dynamics seen in musical performances. The project entails gathering a large dataset of piano performance recordings that spans a variety of genres, performers, and playing techniques. The audio and video components of the performances are pre-processed in order to extract pertinent information. Long Short-Term Memory (LSTM), an RNN architecture, is used to mimic the sequential nature of the performances. The RNN is trained on the features that were retrieved in order to discover the patterns and traits connected to various piano performances. The similarity between the performance representations may be measured using similarity measures using Euclidean distance. The RNN-based system may be further developed to do tasks like score following, expressive performance analysis, and stylistic variation creation to aid in performance analysis. The system may provide insights into timing accuracy, dynamics, phrasing, and other expressive qualities of the piano performances by matching performance data with related musical scores. The proposed method RNN-LSTM provides accuracy about 99%, precision about 97% , recall about 98.9% and F1 Score of 97.6%.

Implementation of Adam Optimizer using Recurrent Neural Network (RNN) Architecture for Diabetes Classification

Implementation of Adam Optimizer using Recurrent Neural Network (RNN) Architecture for Diabetes Classification

Optimizing RNNs for EMG Signal Classification: A Novel Strategy Using Grey Wolf Optimization.

Accurate classification of electromyographic (EMG) signals is vital in biomedical applications. This study evaluates different architectures of recurrent neural networks for the classification of EMG signals associated with five movements of the right upper extremity. A Butterworth filter was implemented for signal preprocessing, followed by segmentation into 250 ms windows, with an overlap of 190 ms. The resulting dataset was divided into training, validation, and testing subsets. The Grey Wolf Optimization algorithm was applied to the gated recurrent unit (GRU), long short-term memory (LSTM) architectures, and bidirectional recurrent neural networks. In parallel, a performance comparison with support vector machines (SVMs) was performed. The results obtained in the first experimental phase revealed that all the RNN networks evaluated reached a 100% accuracy, standing above the 93% achieved by the SVM. Regarding classification speed, LSTM ranked as the fastest architecture, recording a time of 0.12 ms, followed by GRU with 0.134 ms. Bidirectional recurrent neural networks showed a response time of 0.2 ms, while SVM had the longest time at 2.7 ms. In the second experimental phase, a slight decrease in the accuracy of the RNN models was observed, standing at 98.46% for LSTM, 96.38% for GRU, and 97.63% for the bidirectional network. The findings of this study highlight the effectiveness and speed of recurrent neural networks in the EMG signal classification task.

DETECTION OF MALICIOUS ACTIVITY USING A NEURAL NETWORK FOR CONTINUOUS OPERATION

This article describes the problem of detecting malicious programs in running systems of users of mobile applications. Because users can download any application on their phone, which over time can pull up additional settings, which can store malicious routines for monitoring both personal life and their personal data, such as logins, passwords, bank data. The detection of such routines is based on dynamic analysis and is formulated as a weakly controlled problem. The article contains an analysis of information on the development of researchers who worked on detection models and methods such as: statistical and dynamic intrusion detection methods, anomaly detection model, settings classification methods, machine and deep learning methods. Machine learning, and especially deep learning, has become an extremely useful and interesting topic in cybersecurity over the past few years. In this context, the detection of malicious software has received considerable attention. The article considers the problem of detecting the activity of malicious software of mobile operating systems in the time domain by analyzing behavioral sequences of a large amount of industrial data. When malware executes on a system, its behavior consists of a series of distinct actions placed along the time axis, and there is only a subsequence of actions that lead to malicious activity. Very often, malicious software does not manifest itself immediately, and at some point in the execution, malicious activity is formed. Therefore, the main task and difficulty is to identify such a subsequence in the entire sequence of events. Due to this, it is proposed to develop a behavior model that would analyze the dynamic behavior of the program in the system during execution. For this, a sequence of API/function calls generated by the program at runtime is used as input data and a recurrent neural network (RNN) architecture is proposed to detect malicious activity. The article describes the training method of the proposed model and provides verification of its performance on a large sample of industrial data consisting of a large number of samples generated on the emulator farm. Many mobile phone vendors strive for hardware acceleration on the device to provide better support. Therefore, it can be considered that the deployment of a model based on RNM directly on the device as one of the security levels can become a viable solution. The test data of the model described in the article show sufficiently high positive results when detecting malicious activities.

Precipitation forecast using RNN variants by analyzing Optimizers and Hyperparameters for Time-series based Climatological Data

Flood is a significant problem in many regions of the world for the catastrophic damage it causes to both property and human lives; excessive precipitation being the major cause. The AI technologies, Deep Learning Neural Networks and Machine Learning algorithms attempt realistic solutions to numerous disaster management challenges. This paper works on RNN- based rainfall/ precipitation forecasting models by investigating the performances of various Recurrent Neural Network (RNN) architectures, Bidirectional RNN (BRNN), Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU) and ensemble models such as BRNN-GRU, BRNN-LSTM, LSTM-GRU, BRNN-LSTM-GRU using NASAPOWER datasets of Andhra Pradesh (AP) and Tamil Nadu (TN) in India. The different stages in the workflow of the methodology are Data collection, Data pre-processing, Data splitting, Defining hyperparameters, Model building and Performance evaluation. Experiments for identifying improved optimizers and hyperparameters for the time-series climatological data are investigated for accurate precipitation forecast. The metrics: Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Square Error (RMSE) and Root Mean Squared Logarithmic Error (RMSLE) values are used to compare the precipitation predictions of different models. The RNN variants and ensemble models, BRNN, LSTM, GRU, BRNN-GRU, BRNN-LSTM, LSTM-GRU, BRNN-LSTM-GRU produce predictions with RMSLE values of 2.448, 0.555, 0.255, 1.305, 1.383, 0.364, 1.740 for AP and 1.735, 0.663, 0.152, 0.889, 1.118, 0.379, 1.328 for TN respectively. The best performing RNN model, GRU when ensembled with the existing statistical model SARIMA produces an RMSLE value of 0.754 and 1.677 respectively for AP and TN.

Investigating Performance of ESN’s in Forecasting Financial Metrics When Compared To Traditional RNN Types

This research investigates the performance of Echo State Networks (ESN) in forecasting financial metrics and compares their effectiveness against traditional recurrent neural network (RNN) architectures like Long Short-Term Memory (LSTM) networks and Gated Recurrent Units (GRU), as well as Generalized Auto Regressive Conditional Heteroskedasticity (GARCH) models. By analyzing datasets sourced from Yahoo Finance for various financial indices, exchange-traded funds and stocks over five years, this study examines the accuracy, and structural simplicity of ESNs in predicting close prices, daily volatility, and log returns. Results indicate that ESNs, with their reservoir computing capabilities, outperform traditional RNNs by achieving lower mean absolute error (MAE) and mean squared error (MSE) overall, highlighting their potential as efficient and robust tools for financial time-series forecasting.

INTELLIGENT MODEL FOR CLASSIFYING HEMODYNAMIC PATTERNS OF BRAIN ACTIVATION TO IDENTIFY NEUROCOGNITIVE MECHANISMS OF SPATIAL-NUMERICAL ASSOCIATIONS

The study presents the results of the development and testing of deep learning neural network architectures, which demonstrate high accuracy rates in classifying neurophysiological data, in particular hemodynamic brain activation patterns obtained by functional near-infrared spectroscopy, during solving mathematical problems on spatial-numerical associations. The analyzed signal represents a multidimensional time series of oxyhemoglobin and deoxyhemoglobin dynamics. Taking the specificity of the fNIRS signal into account, a comparative analysis of 2 types of neural network architectures was carried out: (1) architectures based on recurrent neural networks: recurrent neural network with long short-term memory, recurrent neural network with long short-term memory with fully connected layers, bidirectional recurrent neural network with long short-term memory, convolutional recurrent neural network with long short-term memory; (2) architectures based on convolutional neural networks with 1D convolutions: convolutional neural network, fully convolutional neural network, residual neural network. Trained long short-term memory recurrent neural network architectures showed worse results in accuracy in comparison with 1D convolutional neural network architectures. Residual neural network (model_Resnet) demonstrated the highest accuracy rates in three experimental conditions more than 88% in detecting age-related differences in brain activation during spatial-numerical association tasks considering the individual characteristics of the respondents’ signal.

Enhancement of optical surface quality based on real-time compensation of temperature-driven thermal errors in diamond turning

In diamond turning, thermal error compensation is critical for improving machining accuracy. In the ultra-precision diamond turning of optical devices, fluctuations in room temperature cause periodic waviness on the machined surface, which degrades the optical performance. Surface finishing processes, such as polishing, are generally required to remove surface waviness. However, the materials used in the polishing process can pollute the environment. Moreover, the polishing process produces a large amount of waste, including contaminated alcohol and tissue. This study presents a new method for compensating temperature-driven thermal errors during single-point diamond turning (SPDT) using machine learning. Long short-term memory (LSTM), an artificial recurrent neural network architecture, was used to estimate the thermal error from the temperatures observed at several locations around the SPDT machine. The performance of the proposed method was evaluated by comparing its estimation accuracy with those of other machine learning methods (multiple linear regression and support vector machine). The thermal error was compensated for by controlling the tool position according to the estimated thermal error. To evaluate the results of the thermal error compensation, the intensities of the laser beam profiles reflected from two flat aluminum mirrors with diameters of 100 mm were observed at several propagation distances. The waviness formed on the uncompensated surface resulted in a dramatic increase in intensity owing to the constructive interference caused by Bragg scattering, whereas the intensity of the reflection from the compensated surface did not vary as the waviness was removed.

An Efficient Sentiment Classification Method with the Help of Neighbors and a Hybrid of RNN Models

Some texts that are challenging to recognize on their own may become more understandable in a neighborhood of related texts with similar contexts. Motivated by this intuition, a novel deep text sentiment classification (DTSC) model is proposed to improve the model’s performance by incorporating the neighborhood of related texts. Our framework uses a nonparametric approach to construct neighborhoods of related texts based on Jaccard similarities. Then, a new deep recurrent neural network architecture is proposed, comprising two distinct modules: bidirectional long short-term memory (Bi-LSTM) and gated recurrent unit (GRU). The proposed model aims to effectively capture informative features from the input text and its neighbors. The result of each module is processed through the maximum operation, which selects the most pertinent data. Finally, the extracted features are concatenated and subjected to classification to achieve accurate sentiment prediction. Previous studies have commonly employed a parametric approach to represent textual metadata. However, our approach utilizes a nonparametric approach, enabling our model to perform strongly even when the text vocabulary varies between training and testing. The proposed DTSC model has been evaluated on five real-world sentiment datasets, achieving 99.60% accuracy on the Binary_Getty (BG) dataset, 98.32% accuracy on the Binary_iStock (BIS) dataset, 96.13% accuracy on Twitter, 82.19% accuracy on the multi-view sentiment analysis (MVSA) dataset, and 87.60% accuracy on the IMDB dataset. These findings demonstrate that the proposed model outperforms established baseline techniques in terms of model evaluation criteria for text sentiment classification.

A framework for neural network based constitutive modelling of inelastic materials

Given the significant recent advances in added layer manufacturing and materials engineering, new types of materials or new material micro-structures are becoming available at a fast rate. The finite element analysis of structures or structural components requires a constitutive model that describes the behaviour of the new materials. The formulation of accurate constitutive equations is generally complex and time consuming. Hence, suitable machine learning strategies may be used to render this process obsolete and bridge the gap between experimental data and finite element analysis. In this work, a generic stress update procedure is presented that is suitable for the modelling of rate-independent, elastic or inelastic, isotropic or anisotropic material behaviour. The proposed strategy is based on a recurrent neural network architecture and must be trained on stress and strain data sequences that represent physical or numerical experiments. A training strategy based on gradient-free optimisation is presented. It is shown that piecewise linear behaviour, such as uniaxial elasto-plasticity, can be represented exactly. Further numerical examples include uniaxial damage mechanics and elasto-plasticity under plane strain conditions. An efficient criterion for the verification of thermodynamic consistency is proposed and applied to the trained stress update models. The strategy is compared to GRU or LSTM based architectures and shown to offer advantages.