Recurrent Neural Model Research Articles

Enhancing 3D planetary atmosphere simulations with a surrogate radiative transfer model

Abstract This work introduces an approach to enhancing the computational efficiency of 3D atmospheric simulations by integrating a machine-learned surrogate model into the OASIS global circulation model (GCM). Traditional GCMs, which are based on repeatedly numerically integrating physical equations governing atmospheric processes across a series of time-steps, are time-intensive, leading to compromises in spatial and temporal resolution of simulations. This research improves upon this limitation, enabling higher resolution simulations within practical timeframes. Speeding up 3D simulations holds significant implications in multiple domains. Firstly, it facilitates the integration of 3D models into exoplanet inference pipelines, allowing for robust characterisation of exoplanets from a previously unseen wealth of data anticipated from JWST and post-JWST instruments. Secondly, acceleration of 3D models will enable higher resolution atmospheric simulations of Earth and Solar System planets, enabling more detailed insights into their atmospheric physics and chemistry. Our method replaces the radiative transfer module in OASIS with a recurrent neural network-based model trained on simulation inputs and outputs. Radiative transfer is typically one of the slowest components of a GCM, thus providing the largest scope for overall model speed-up. The surrogate model was trained and tested on the specific test case of the Venusian atmosphere, to benchmark the utility of this approach in the case of non-terrestrial atmospheres. This approach yields promising results, with the surrogate-integrated GCM demonstrating above 99.0% accuracy and 147 factor GPU speed-up of the entire simulation compared to using the matched original GCM under Venus-like conditions.

A certain examination on heterogeneous systolic array (HSA) design for deep learning accelerations with low power computations

Acceleration techniques play a crucial role in enhancing the performance of modern high-speed computations, especially in Deep Learning (DL) applications where the speed is of utmost importance. One essential component in this context is the Systolic Array (SA), which effectively handles computational tasks and data processing in a rhythmic manner. Google's Tensor Processing Unit (TPU) leverages the power of SA for neural networks. The core SA's functionality and performance lies in the Computation Element (CE), which facilitates parallel data flow. In our article, we introduce a novel approach called Proposed Systolic Array (PSA), which is implemented on the CE and further enhanced with a modified Hybrid Kogge Stone adder (MHA). This design incorporates principles to expedite computations by rounding and extracting data model in SA as PSA-MHA. The PSA, utilizing a data flow model with MHA, significantly accelerates data shifts and control passes in execution cycles. We validated our approach through simulations on the Cadence Virtuoso platform using 65 nm process technology, comparing it to the General Matrix Multiplication (GMMN) benchmark. The results showed remarkable improvements in the CE, with a 30.29 % reduction in delay, a 23.07 % reduction in area, and an 11.87 % reduction in power consumption. The PSA outperformed these improvements, achieving a 46.38 % reduction in delay, a 7.58 % reduction in area, and an impressive 48.23 % decrease in Area Delay Product (ADP). To further substantiate our findings, we applied the PSA-based approach to pre-trained hybrid Convolutional and Recurrent (CNN-RNN) neural models. The PSA-based hybrid model incorporates 189 million Multiply-Accumulate (MAC) units, resulting in a weighted mean architecture value of 784.80 for the RNN component. We also explored variations in bit width, which led to delay reductions ranging from 20.17 % to 30.29 %, area variations between 13.08 % and 32.16 %, and power consumption changes spanning from 11.88 % to 20.42 %.

Dynamic data-driven multiscale modeling for predicting the degradation of a 316L stainless steel nuclear cladding material

We have developed a long short-term memory stacked ensemble (LSTM-SE) surrogate modeling approach that can provide rapid predictions of microstructural evolution and the resultant mechanical properties of American Iron and Steel Institute (AISI) 316L series stainless steel (SS316L) fuel cladding under conditions of varying temperature and radiation dose rate. To acquire training data, we developed and implemented a kinetic Monte Carlo (KMC) model to simulate precipitation kinetics of M23C6, γ’, and G phases within SS316L cladding. Experimentally reported precipitation kinetics of SS316L in literature were linked to the kinetic parameters of the simulated precipitation in our KMC model. The model was then used to simulate microstructure evolution under synthetically generated treatments of varying temperature and radiation dose rate for periods of up to 3000 h. Changes in volume fraction, number density, and particle size of precipitates were recorded, and particle area fractions were correlated using statistical methods to develop the surrogate model. Simultaneously, the mechanical properties of the simulated microstructures were evaluated using microstructure-based finite element method (FEM) analysis to determine the elastic modulus, yield stress, ultimate tensile strength, and elongation to failure of the aged microstructures. Using this approach, our surrogate model can predict precipitation behavior within 0.25 % volume fraction and mechanical properties within 6 % relative error from the values predicted by the KMC and FEM models using 50 training simulations as input. The trained recurrent neural network-based model can return estimations of precipitation kinetics and mechanical properties ∼1000 times faster than the physics-based codes. This work demonstrates, as a proof of concept, that microstructural evolution under variable conditions can be predicted using a statistics-based model informed by a practicably obtainable dataset. The potential applications of this type of modeling framework are discussed.

A Robust Recurrent Neural Networks-Based Surrogate Model for Thermal History and Melt Pool Characteristics in Directed Energy Deposition.

In directed energy deposition (DED), accurately controlling and predicting melt pool characteristics is essential for ensuring desired material qualities and geometric accuracies. This paper introduces a robust surrogate model based on recurrent neural network (RNN) architectures-Long Short-Term Memory (LSTM), Bidirectional LSTM (Bi-LSTM), and Gated Recurrent Unit (GRU). Leveraging a time series dataset from multi-physics simulations and a three-factor, three-level experimental design, the model accurately predicts melt pool peak temperatures, lengths, widths, and depths under varying conditions. RNN algorithms, particularly Bi-LSTM, demonstrate high predictive accuracy, with an R-square of 0.983 for melt pool peak temperatures. For melt pool geometry, the GRU-based model excels, achieving R-square values above 0.88 and reducing computation time by at least 29%, showcasing its accuracy and efficiency. The RNN-based surrogate model built in this research enhances understanding of melt pool dynamics and supports precise DED system setups.

Intent detection for task‐oriented conversational agents: A comparative study of recurrent neural networks and transformer models

AbstractConversational assistants (CAs) and Task‐oriented ones, in particular, are designed to interact with users in a natural language manner, assisting them in completing specific tasks or providing relevant information. These systems employ advanced natural language understanding (NLU) and dialogue management techniques to comprehend user inputs, infer their intentions, and generate appropriate responses or actions. Over time, the CAs have gradually diversified to today touch various fields such as e‐commerce, healthcare, tourism, fashion, travel, and many other sectors. NLU is fundamental in the natural language processing (NLP) field. Identifying user intents from natural language utterances is a sub‐task of NLU that is crucial for conversational systems. The diversity in user utterances makes intent detection (ID) even a challenging problem. Recently, with the emergence of Deep Neural Networks. New State of the Art (SOA) results have been achieved for different NLP tasks. Recurrent neural networks (RNNs) and Transformer architectures are two major players in those improvements. RNNs have significantly contributed to sequence modelling across various application areas. Conversely, Transformer models represent a newer architecture leveraging attention mechanisms, extensive training data sets, and computational power. This review paper begins with a detailed exploration of RNN and Transformer models. Subsequently, it conducts a comparative analysis of their performance in intent recognition for Task‐oriented (CAs). Finally, it concludes by addressing the main challenges and outlining future research directions.

Artificial intelligence-based pulmonary embolism classification: Development and validation using real-world data.

This paper presents an artificial intelligence-based classification model for the detection of pulmonary embolism in computed tomography angiography. The proposed model, developed from public data and validated on a large dataset from a tertiary hospital, uses a two-dimensional approach that integrates temporal series to classify each slice of the examination and make predictions at both slice and examination levels. The training process consists of two stages: first using a convolutional neural network InceptionResNet V2 and then a recurrent neural network long short-term memory model. This approach achieved an accuracy of 93% at the slice level and 77% at the examination level. External validation using a hospital dataset resulted in a precision of 86% for positive pulmonary embolism cases and 69% for negative pulmonary embolism cases. Notably, the model excels in excluding pulmonary embolism, achieving a precision of 73% and a recall of 82%, emphasizing its clinical value in reducing unnecessary interventions. In addition, the diverse demographic distribution in the validation dataset strengthens the model's generalizability. Overall, this model offers promising potential for accurate detection and exclusion of pulmonary embolism, potentially streamlining diagnosis and improving patient outcomes.

Charging Station Power System Data Prediction Model Based on Deep Learning

The increase growth and adoption of electric vehicles (EVs) are playing a crucial role in advanced transportation system, helping to minimizing the emissions of harmful greenhouse gases and enhancing environmental sustainability. The need for EVs to be charged immediately has gained significant attention due to the rise in EVs sales over the last years. As a result of this, need for electric car charging is important to reducing the impact of electric network and providing minimum charging fares. In order to calculate the demand for charging EVs, a novel Deep Learning (DL)-based Long-Short Term Memory (LSTM) recurrent neural network predictor model is attempted to be developed in this research study. The Modified Aquilla Optimizer Algorithm (MAOA) is used to optimizing the parameter of the new Deep LSTM (DLSTM) neural predictor models and Independent Component Analysis (ICA) is utilized to solve the input time series data while conserving its properties. In this research, a novel ICA—AOA—DLSTM neural predictor model was developed to addressed the challenges of vanishing and exploding gradient in basic recurrent neural learnings. The predictor model was tested on the EV charging dataset from Georgia Tech, Atlanta, USA, indicating its performance. During simulation, the predictor achieved a prediction accuracies of 96.24% with a mean absolute errors of 0.1083 and a RMSE errors of 3.0629 × 10^-5, outperforming previous techniques. Additionally, the mean absolute error was found to be 0.2083, with a mean square error of 3.25516 × 10^-10. These results highlight the effectiveness of the novel deep learning LSTM neural predictor for this dataset compared to existing techniques.

The Unveiling Distress: Harnessing NLP and Deep Learning to Identify Suicidal Signals in Tweets

The rise of social media platforms has provided researchers with unprecedented access to vast amounts of user-generated content, offering a unique opportunity to explore various aspects of human behavior, including mental health. This paper presents a novel approach to identifying suicidal signals in tweets using Natural Language Processing (NLP) techniques and Deep Learning algorithms. We propose a multi-step methodology that involves data collection, preprocessing, feature extraction, and classification. Leveraging state-of-the-art deep learning architectures such as recurrent neural networks (RNNs) and transformer models, our approach aims to accurately detect linguistic patterns indicative of suicidal ideation and distress. We evaluate the effectiveness of our method using a large dataset of annotated tweets and demonstrate promising results in terms of both precision and recall. Furthermore, we discuss the ethical implications and potential applications of our research in suicide prevention and mental health support systems.

Multimodal Content Analysis Using Deep Learning

Abstract: The multimodal content analysis platform combines sentiment analysis and neural style transfertechniques to process and improve various types of digital content. The sentiment analysis module utilizes natural language processing (NLP) algorithms, such as recurrent neural networks (RNNs) or transformer models like BERT, to extract emotional signals from textual, visual, and auditory inputs. Signals are classified into predefined sentiment categories, providing granular insights into the emotional context of the content. The platform employs neural style transfer algorithms, such as style transfer networks (NSTNs) or generative adversarial networks (GANs), to transfer stylistic attributes between texts. By training on a diverse range of artistic styles, the system learns to apply these styles to input text while preserving semantic meaning. This process enhances the visual representation of textual content, making it more appealing and engaging to users.

DEVELOPMENT OF VOICE ASSISTANT BASED ON NEURAL SYSTEMS

The development of a voice assistant based on neural systems is an actual line of research in the field of artificial intelligence. Existing voice assistants were analyzed and their advantages and disadvantages were identified in this paper. The following concepts were considered: natural language processing, and neural networks. Different methods of natural language processing significantly increase the quality of work of many programs, because they can interact with people at a natural level for humans without any problems. Modern voice assistants can help millions of people around the world at the same time, regardless of the time of the request. They are covering more and more spheres every year, from ordering food and taxis and making plans for the day to acting as translators. The types of dialogue systems, the voice assistants` scope of use and their types, recurrent neural networks and data processing models Seq2seq and Transformer, as well as the work of the theoretical component of voice assistants based on neural networks, were considered. Node.js libraries were examined in detail, and an own voice assistant was created and outputs of this system were given. In addition, the possibilities of integrating the voice assistant with other systems and platforms, which expands its functionality and improves interaction with the user, were examined. General theoretical information about voice assistants and their components and principles of work were analysed. An environment for writing a voice assistant was chosen and a practical implementation of writing code for a voice assistant and an example of ready-made code were shown.

CNSMolGen: A Bidirectional Recurrent Neural Network-Based Generative Model for De Novo Central Nervous System Drug Design.

Central nervous system (CNS) drugs have had a significant impact on treating a wide range of neurodegenerative and psychiatric disorders. In recent years, deep learning-based generative models have shown great potential for accelerating drug discovery and improving efficacy. However, specific applications of these techniques in CNS drug discovery have not been widely reported. In this study, we developed the CNSMolGen model, which uses a framework of bidirectional recurrent neural networks (Bi-RNNs) for de novo molecular design of CNS drugs. Results showed that the pretrained model was able to generate more than 90% of completely new molecular structures, which possessed the properties of CNS drug molecules and were synthesizable. In addition, transfer learning was performed on small data sets with specific biological activities to evaluate the potential application of the model for CNS drug optimization. Here, we used drugs against the classical CNS disease target serotonin transporter (SERT) as a fine-tuned data set and generated a focused database against the target protein. The potential biological activities of the generated molecules were verified by using the physics-based induced-fit docking study. The success of this model demonstrates its potential in CNS drug design and optimization, which provides a new impetus for future CNS drug development.

A Research Study on How AI Creates Fiction Stories

Abstract: This paper includes the study of complex process by which artificial intelligence systems create fictional stories. In this process we cover the five processes such as The pre -processing steps for using AI to create a fictional story, Plot structure and plot production process for creating fictional stories using artificial intelligence, Character growth in the process of creating a virtual story using artificial intelligence, Generating dialogue in the process of creating a virtual story using artificial intelligence, and Using AI to revisit and iterate the process of creating fictional stories. Using state-of-the-art artificial intelligence models, especially those based on deep learning architectures such as recurrent neural networks (RNNs) and transformer models such as generative pre- trained transformers (GPTs), we analyse how these systems understand and manipulate language to create consistent fictional pieces. This paper This Make up story through empirical experiments and qualitative analysis, we carefully study the impact of different parameters, including Algorithm, Education, Fine-tuning Hint, Generation, and Evaluation methods, on the quality and diversity of the generated stories. The study also explores the role of artificial intelligence as well as human collaboration in the fiction stories writing and creative process. It discuss the Future Directions of AI’S role in creating fiction stories.

A Comparative Study of Khasi Speech Recognition Systems with Recurrent Neural Network-Based Language Model

This paper offers a comparative analysis of Khasi speech recognition systems utilizing a recurrent neural network-based language model (RNN-LM). Develop different acoustic models (AMs) to evaluate the optimal performance. This paper observed that using RNN-LM performed best than traditional other models. The wave surfer performs data processing followed by collecting the recorder based continuous speech database. Moreover, a minimization of word error rate (WER) in 2.83.8% range for major speech data and 2.4-3.5% for minor speech data. Additionally, two acoustic features are used, and from the experimental results, the Mel frequency cepstral coefficient (MFCC) yielded improved performance than the perceptual linear prediction (PLP).

Sentiment Analysis of Modern Chinese Literature Based on Deep Learning

Recent advancements in deep learning have facilitated sentiment analysis of modern Chinese literature. By leveraging techniques such as recurrent neural networks (RNNs) and transformer models like BERT, researchers can effectively gauge the sentiment expressed within literary texts. These models learn intricate patterns and context-specific nuances, enabling them to discern the emotional tone of Chinese literature accurately.Sentiment analysis, a crucial task in natural language processing, plays a pivotal role in understanding human emotions and opinions expressed in textual data. In this paper, we propose a novel deep learning framework, termed BERT-LLSTM-DL, for sentiment analysis in Chinese literature. The framework integrates Bidirectional Encoder Representations from Transformers (BERT) for language representation, Long Short-Term Memory (LSTM) networks for sequential learning, and deep learning techniques for feature extraction. We evaluate the proposed model on a dataset comprising Chinese literature texts and achieve promising results in terms of accuracy, precision, recall, and F1-score.

Recurrent neural network plasticity models: Unveiling their common core through multi-task learning

Recurrent neural network models are known to be particularly suitable for data-driven constitutive modeling due to their built-in memory variables. The main challenge preventing their widespread application to engineering materials lies in their excessive need of data for training. Here, we postulate that RNN models of elasto-plastic solids feature a large common core that is shared by all materials of the same class. The common core is complemented by material-specific layers with parameters that vary from material-to-material. After training RNN models for 25 different von Mises materials, we identify the common core of a multi-task model. Furthermore, we demonstrate through ensemble transfer learning that adding a new material to the multi-task model requires datasets that are two to three orders of magnitude smaller than the datasets needed for training an RNN from scratch. In addition, to multi-task learning, we introduce probabilistic ensembles of RNN plasticity models to quantify the epistemic uncertainty. A deep drawing simulation is performed to demonstrate the superior generalization capabilities of RNNs identified via multi-task learning as compared to those obtained through single task training.

Multi-step planning with learned effects of partial action executions

In this paper, we propose a novel affordance model, which combines object, action, and effect information in the latent space of a predictive neural network architecture that is built on Conditional Neural Processes. Our model allows us to make predictions of intermediate effects expected to be obtained during action executions and make multi-step plans that include partial actions. We first compared the prediction capability of our model using an existing interaction data set and showed that it outperforms a recurrent neural network-based model in predicting the effects of lever-up actions. Next, we showed that our model can generate accurate effect predictions for other actions, such as push and grasp actions. Our system was shown to generate successful multi-step plans to bring objects to desired positions using the traditional A* search algorithm. Furthermore, we realized a continuous planning method and showed that the proposed system generated more accurate and effective plans with sequences of partial action executions compared to plans that only consider full action executions using both planning algorithms.

Gibbs sampler for Bayesian prediction of triple seasonal autoregressive processes

Researchers have extended autoregressive (AR) time-series models to adequately fit and model time-series with triple seasonality. These AR extensions can be referred to as triple seasonal AR (TSAR) models. For these TSAR time-series models, only Bayesian estimation and identification have been introduced. Therefore, in this article, we aim to extend the existing work for presenting the Bayesian prediction for TSAR models using the Gibbs sampler algorithm. In this Bayesian prediction, we first assume the TSAR errors are identically normally distributed, and we employ normal-gamma and g priors for the TSAR parameters. Based on the normally distributed TSAR errors and the specified TSAR parameters’ priors, we are able to derive the conditional predictive and posterior densities. Particularly, we show the conditional posterior densities of TSAR coefficients and variance are the multivariate normal and inverse-gamma, respectively. In addition, for the future TSAR observations, we show the conditional predictive density is the multivariate normal. Using these derived full conditional distributions, we propose the Gibbs sampler to efficiently approximate the joint predictive and posterior densities and to easily carry out multiple-step ahead predictions. We validate the accuracy of forecasting for the proposed Gibbs sampler by conducting a simulation study. Moreover, we apply the proposed Gibbs sampler to hourly electricity-load time-series datasets in some European countries, along with a comparison with the well-known long-short term memory (LSTM) recurrent neural model.

Flood Forecasting Method and Application Based on Informer Model

Flood Forecasting Method and Application Based on Informer Model

Fabric defect image generation method based on the dual-stage W-net generative adversarial network

Due to the intricate and diverse nature of textile defects, detecting them poses an exceptionally challenging task. In comparison with conventional defect detection methods, deep learning-based defect detection methods generally exhibit superior precision. However, utilizing deep learning for defect detection requires a substantial volume of training data, which can be particularly challenging to accumulate for textile flaws. To augment the fabric defect dataset and enhance fabric defect detection accuracy, we propose a fabric defect image generation method based on Pix2Pix generative adversarial network. This approach devises a novel dual-stage W-net generative adversarial network. By increasing the network depth, this model can effectively extract intricate textile image features, thereby enhancing its ability to expand information sharing capacity. The dual-stage W-net generative adversarial network allows generating desired defects on defect-free textile images. We conduct quality assessment of the generated fabric defect images resulting in peak signal-to-noise ratio and structural similarity values exceeding 30 and 0.930, respectively, and a learned perceptual image patch similarity value no greater than 0.085, demonstrating the effectiveness of fabric defect data augmentation. The effectiveness of dual-stage W-net generative adversarial network is established through multiple comparative experiments evaluating the generated images. By examining the detection performance before and after data augmentation, the results demonstrate that mean average precision improves by 6.13% and 14.57% on YOLO V5 and faster recurrent convolutional neural networks detection models, respectively.

The metamorphosis of machine translation: The rise of neural machine translation and its challenges

Machine translation refers to the process of using computers to translate source language into target language, which has undergone significant transformations since its inception, with the current mainstream neural machine translation achieving satisfactory translation performance. This paper overviews the three developmental stages of machine translation: rule-based machine translation, statistical machine translation, and neural machine translation, with a focus on neural machine translation. It introduces the key models that emerged in the development process of neural machine translation, namely the recurrent neural network encoder-decoder model, recurrent neural network search model, and Transformer, and compares their strengths and limitations. Other relevant technologies and models developed alongside neural machine translation are also discussed. Addressing the current challenges of neural machine translation, the paper delves into issues of overfitting, low-resource translation, structural optimization of Transformer models, and enhancement of neural machine translation interpretability. Finally, the paper explores the prospects of applying neural machine translation to multimodal translation.