Short-term Memory Mechanism Research Articles

Improving de novo Molecule Generation by Embedding LSTM and Attention Mechanism in CycleGAN.

The application of deep learning in the field of drug discovery brings the development and expansion of molecular generative models along with new challenges in this field. One of challenges in de novo molecular generation is how to produce new reasonable molecules with desired pharmacological, physical, and chemical properties. To improve the similarity between the generated molecule and the starting molecule, we propose a new molecule generation model by embedding Long Short-Term Memory (LSTM) and Attention mechanism in CycleGAN architecture, LA-CycleGAN. The network layer of the generator in CycleGAN is fused head and tail to improve the similarity of the generated structure. The embedded LSTM and Attention mechanism can overcome long-term dependency problems in treating the normally used SMILES input. From our quantitative evaluation, we present that LA-CycleGAN expands the chemical space of the molecules and improves the ability of structure conversion. The generated molecules are highly similar to the starting compound structures while obtaining expected molecular properties during cycle generative adversarial network learning, which comprehensively improves the performance of the generative model.

A federated learning system with enhanced feature extraction for human activity recognition

With the rapid growth of mobile devices, wearable sensor-based human activity recognition (HAR) has become one of the hottest topics in the Internet of Things. However, it is challenging for traditional approaches to achieving high recognition accuracy while protecting users’ privacy and sensitive information. To this end, we design a federated learning system for HAR (HARFLS). Based on the FederatedAveraging method, HARFLS enables each user to handle its activity recognition task safely and collectively. However, the recognition accuracy largely depends on the system’s feature extraction ability. To capture sufficient features from HAR data, we design a perceptive extraction network (PEN) as the feature extractor for each user. PEN is mainly composed of a feature network and a relation network. The feature network, based on a convolutional block, is responsible for discovering local features from the HAR data while the relation network, a combination of long short-term memory (LSTM) and attention mechanism, focuses on mining global relationships hidden in the data. Four widely used datasets, i.e., WISDM, UCI_HAR 2012, OPPORTUNITY, and PAMAP2, are used for performance evaluation. Experimental results demonstrate that PEN outperforms 14 existing HAR algorithms on these datasets in terms of the F1-score; HARFLS with PEN obtains better recognition results on the WISDM and PAMAP2 datasets, compared with 11 existing federated learning systems with various feature extraction structures.

A Semantic-Embedding Model-Driven Seq2Seq Method for Domain-Oriented Entity Linking on Resource-Restricted Devices

General entity linking systems usually leverage global coherence of all the mapped entities in the same document by using semantic embeddings and graph-based approaches. However, graph-based approaches are computationally expensive for open-domain datasets. In this paper, the authors overcome these problems by presenting an RDF embedding-based seq2seq entity linking method in specific domains. They filter candidate entities of mentions having similar meanings by using the domain information of the annotated pairs. They resolve high ambiguous pairs by using Bi-directional long short-term memory (Bi-LSTM) and attention mechanism for the entity disambiguation. To evaluate the system with baseline methods, they generate a dataset including book, music, and movie categories. They achieved 0.55 (Mi-F1), 0.586 (Ma-F1), 0.846 (Mi-F1), and 0.87 (Ma-F1) scores for high and low ambiguous datasets. They compare the method by using recent (WNED-CWEB) datasets with existing methods. Considering the domain-specificity of the proposed method, it tends to achieve competitive results while using the domain-oriented datasets.

Integrated Generative Model for Industrial Anomaly Detection via Bidirectional LSTM and Attention Mechanism

For emerging industrial Internet of Things (IIoT), intelligent anomaly detection is a key step to build smart industry. Especially, explosive time-series data pose enormous challenges to the information mining and processing for modern industry. How to identify and detect the multidimensional industrial time-series anomaly is an important issue. However, most of the existing studies fail to handle with large amounts of unlabeled data, thus generating the undesirable results. In this article, we propose a novel integrated deep generative model, which is built by generative adversarial networks based on bidirectional long short-term memory and attention mechanism (AMBi-GAN). The structure for the generator and the discriminator is the bidirectional long short-term memory with attention mechanism, which can capture time-series dependence. Reconstruction loss and generation loss test the input of sample training space and random latent space. Experimental results show that the detection performance of our proposed AMBi-GAN has the potential to improve the detection accuracy of industrial multidimensional time-series anomaly toward IIoT in the era of artificial intelligence.

Application of novel hybrid deep leaning model for cleaner production in a paper industrial wastewater treatment system

Developing monitoring system for paper industrial wastewater treatment system is an important route for wastewater reuse and recycling from wastewater, which are regarded as effective way for cleaner production. A novel hybrid deep leaning CLSTMA model, which based on sequential fusion convolutional neural network (CNN), long short term memory (LSTM) and attention mechanism (AM), was developed to monitor the water quality in a full-scale paper industrial wastewater treatment system for energy conservation and emissions reduction. The hybrid CLSTMA model for predicting water quality of paper industrial wastewater treatment system was divided into three steps: spatial information fusion by using CNN module, temporal information fusion by using LSTM module and variable weighted calculation by using AM module. Compare with other models (CNN, LSTM and CLSTM models), RMSE of CLSTMA model for the effluent chemical oxygen demand (CODeff) reduced by 23.3–31.55%, MAE of CLSTMA model reduced by 38.89–74.50%, R of CLSTMA model increased by 8.29–11.86%. For the effluent suspended solids (SSeff), compared with CNN and LSTM models, RMSE of CLSTMA model reduced by 10.26% and 9.92%, MAE of CLSTMA model reduced by 5.37% and 3.44%, R of CLSTMA model increased by 15.13% and 37.21%, respectively. While, R of CLSTMA was consistent with CLSTM model, but RMSE and MAE of CLSTMA model reduced by 16.07% and 7.49% than the CLSTM model. Simulation results demonstrate that the proposed CLSTMA model has a great potential in monitoring paper industrial wastewater treatment system for cleaner production.

Bridge weigh-in-motion through bidirectional Recurrent Neural Network with long short-term memory and attention mechanism

In bridge weigh-in-motion (BWIM), dynamic bridge response is measured during traffic and used to identify overloaded vehicles. Most past studies of BWIM use mechanics-based algorithms to estimate axle weights. This research instead investigates deep learning, specifically the recurrent neural network (RNN), toward BWIM. In order to acquire the large data volume to train a RNN network that uses bridge response to estimate axle weights, a finite element bridge model is built through the commercial software package LS-DYNA. To mimic everyday traffic scenarios, tens of thousands of randomized vehicle formations are simulated, with different combinations of vehicle types, spacings, speeds, axle weights, axle distances, etc. Dynamic response from each of the randomized traffic scenarios is recorded for training the RNN. In this paper we propose a 3-stage Bidirectional RNN toward BWIM. Long short-term memory (LSTM) and attention mechanism are embedded in the BRNN to further improve the network performance. Additional test data indicates that the BRNN network achieves high accuracy in estimating axle weights, in comparison with a conventional moving force identification (MFI) method.

Adaptive Sliding Mode Long Short-Term Memory Fuzzy Neural Control for Harmonic Suppression

In this paper, an adaptive sliding mode controller based on a long and short-term memory fuzzy neural network (ASMC-LSTMFNN) is proposed to suppress harmonics for an active power filter (APF). Firstly, a mathematical dynamic model of a single-phase shunt active power filter considering lumped uncertainties is introduced. Then, based on the design of conventional sliding mode control (SMC), a new type of long and short-term memory fuzzy neural network (LSTMFNN) is proposed to approximate the unknown function of the system. The LSTMFNN incorporates a fuzzy neural network (FNN) structure and long and short-term memory (LSTM) mechanism, excellent learning ability and approximation performance. Moreover, the parameters of the neural network are all automatically adjusted through the adaptive laws, and the Lyapunov stability theorem guarantees the current tracking performance and the stability of the closed-loop system. Finally, hardware experiments are carried out based on the dSPACE hardware platform, and the experimental results show that it has good steady-state and dynamic performance, verifying that it has better control performance and harmonic compensation ability compared with the adaptive sliding mode control based on recurrent fuzzy neural network (ASMC-RFNN).

Multi-Grained Attention Representation With ALBERT for Aspect-Level Sentiment Classification

Aspect-level sentiment classification aims to solve the problem, which is to judge the sentiment tendency of each aspect in a sentence with multiple aspects. Previous works mainly employed Long Short-Term Memory (LSTM) and Attention mechanisms to fuse information between aspects and sentences, or to improve large language models such as BERT to adapt aspect-level sentiment classification tasks. The former methods either did not integrate the interactive information of related aspects and sentences, or ignored the feature extraction of sentences. This paper proposes a novel multi-grained attention representation with ALBERT (MGAR-ALBERT). It can learn the representation that contains the relevant information of the sentence and the aspect, while integrating it into the process of sentence modeling with multi granularity, and finally get a comprehensive sentence representation. In Masked LM (MLM) task, in order to avoid the influence of aspect words being masked in the initial stage of the pre-training, the noise linear cosine decay is introduced into n-gram. We implemented a series of comparative experiments to verify the effectiveness of the method. The experimental results show that our model can achieve excellent results on Restaurant dataset with numerous number of parameters reduced, and it is not inferior to other models on Laptop dataset.

A Sentence-Level Joint Relation Classification Model Based on Reinforcement Learning.

Relation classification is an important semantic processing task in the field of natural language processing (NLP). Data sources generally adopt remote monitoring strategies to automatically generate large-scale training data, which inevitably causes label noise problems. At the same time, another challenge is that important information can appear at any place in the sentence. This paper presents a sentence-level joint relation classification model. The model has two modules: a reinforcement learning (RL) agent and a joint network model. In particular, we combine bidirectional long short-term memory (Bi-LSTM) and attention mechanism as a joint model to process the text features of sentences and classify the relation between two entities. At the same time, we introduce an attention mechanism to discover hidden information in sentences. The joint training of the two modules solves the noise problem in relation extraction, sentence-level information extraction, and relation classification. Experimental results demonstrate that the model can effectively deal with data noise and achieve better relation classification performance at the sentence level.

A Genetic Programming-Based Low-Level Instructions Robot for Realtimebattle.

RealTimeBattle is an environment in which robots controlled by programs fight each other. Programs control the simulated robots using low-level messages (e.g., turn radar, accelerate). Unlike other tools like Robocode, each of these robots can be developed using different programming languages. Our purpose is to generate, without human programming or other intervention, a robot that is highly competitive in RealTimeBattle. To that end, we implemented an Evolutionary Computation technique: Genetic Programming. The robot controllers created in the course of the experiments exhibit several different and effective combat strategies such as avoidance, sniping, encircling and shooting. To further improve their performance, we propose a function-set that includes short-term memory mechanisms, which allowed us to evolve a robot that is superior to all of the rivals used for its training. The robot was also tested in a bout with the winner of the previous “RealTimeBattle Championship”, which it won. Finally, our robot was tested in a multi-robot battle arena, with five simultaneous opponents, and obtained the best results among the contenders.

An improved aspect-category sentiment analysis model for text sentiment analysis based on RoBERTa

The aspect-category sentiment analysis can provide more and deeper information than the document-level sentiment analysis, because it aims to predict the sentiment polarities of different aspect categories in the same text. The main challenge of aspect-category sentiment analysis is that different aspect categories may present different polarities in the same text. Previous studies combine the Long Short-Term Memory (LSTM) and attention mechanism to predict the sentiment polarity of the given aspect category, but the LSTM-based methods are not really bidirectional text feature extraction methods. In this paper, we propose a multi-task aspect-category sentiment analysis model based on RoBERTa (Robustly Optimized BERT Pre-training Approach). Treating each aspect category as a subtask, we employ the RoBERTa based on deep bidirectional Transformer to extract features from both text and aspect tokens, and apply the cross-attention mechanism to guide the model to focus on the features most relevant to the given aspect category. According to the experimental results, the proposed model outperforms other models for comparison in aspect-category sentiment analysis. Furthermore, the influencing factors of our proposed model are also analyzed.

The role of explicit memory in syntactic persistence: Effects of lexical cueing and load on sentence memory and sentence production.

Speakers' memory of sentence structure can persist and modulate the syntactic choices of subsequent utterances (i.e., structural priming). Much research on structural priming posited a multifactorial account by which an implicit learning process and a process related to explicit memory jointly contribute to the priming effect. Here, we tested two predictions from that account: (1) that lexical repetition facilitates the retrieval of sentence structures from memory; (2) that priming is partly driven by a short-term explicit memory mechanism with limited resources. In two pairs of structural priming and sentence structure memory experiments, we examined the effects of structural priming and its modulation by lexical repetition as a function of cognitive load in native Dutch speakers. Cognitive load was manipulated by interspersing the prime and target trials with easy or difficult mathematical problems. Lexical repetition boosted both structural priming (Experiments 1a-2a) and memory for sentence structure (Experiments 1b-2b) and did so with a comparable magnitude. In Experiment 1, there were no load effects, but in Experiment 2, with a stronger manipulation of load, both the priming and memory effects were reduced with a larger cognitive load. The findings support an explicit memory mechanism in structural priming that is cue-dependent and attention-demanding, consistent with a multifactorial account of structural priming.

Massive MIMO CSI reconstruction using CNN‐LSTM and attention mechanism

In massive multiple-input multiple-output (MIMO) systems, the channel state information (CSI) feedback enables performance gain in frequency division duplex networks. However, with the increase in the number of antennas, the feedback overhead of CSI will also enhance. To this end, this study addresses the issue of massive MIMO CSI reconstruction using convolutional neural network (CNN), long short-term memory (LSTM) and attention mechanism, and proposes an efficient network architecture (denoted as CNN-LSTM-A). To achieve a compromise between performance and complexity, the proposed method significantly reduces the number of training parameters by utilising a single-stage network rather than a multiple-stage network. Finally, simulation results show that the authors method can reduce the feedback overhead of CSI effectively, and achieves better performance in terms of CSI compression and recovery accuracy compared with existing state-of-the-art methods.

MemoryPath: A deep reinforcement learning framework for incorporating memory component into knowledge graph reasoning

Knowledge Graph (KG) is identified as a major area in artificial intelligence, which is used for many real-world applications. The task of knowledge graph reasoning has been widely used and proven to be effective, which aims to find these reasonable paths for various relations to solve the issue of incompleteness in KGs. However, many previous works on KG reasoning, such as path-based or reinforcement learning-based methods, are too reliant on the pre-training, where the paths from the head entity and the target entity must be given to pre-train the model, which would easily lead the model to overfit on the given paths seen in the pre-training. To address this issue, we propose a novel reasoning model named MemoryPath with a deep reinforcement learning framework, which incorporates Long Short Term Memory (LSTM) and graph attention mechanism to form the memory component. The well-designed memory component can get rid of the pre-training so that the model doesn’t depend on the given target entity for training. A tailored mechanism of reinforcement learning is presented in this proposed deep reinforcement framework to optimize the training procedure, where two metrics, Mean Selection Rate (MSR) and Mean Alternative Rate (MAR), are defined to quantitatively measure the complexities of the query relations. Meanwhile, three different training mechanisms, Action Dropout, Reward Shaping and Force Forward, are proposed to optimize the training process of the proposed MemoryPath. The proposed MemoryPath is validated on two datasets from FB15K-237 and NELL-995 on different tasks including fact prediction, link prediction and success rate in finding paths. The experimental results demonstrate that the tailored mechanism of reinforcement learning make the MemoryPath achieves state-of-the-art performance comparing with the other models. Also, the qualitative analysis indicates that the MemoryPath can store the learning process and automatically find the promising paths for a reasoning task during the training, and shows the effectiveness of the memory component.

Deep Learning for Arabic Error Detection and Correction

Research on tools for automating the proofreading of Arabic text has received much attention in recent years. There is an increasing demand for applications that can detect and correct Arabic spelling and grammatical errors to improve the quality of Arabic text content and application input. Our review of previous studies indicates that few Arabic spell-checking research efforts appropriately address the detection and correction of ill-formed words that do not conform to the Arabic morphology system. Even fewer systems address the detection and correction of erroneous well-formed Arabic words that are either contextually or semantically inconsistent within the text. We introduce an approach that investigates employing deep neural network technology for error detection in Arabic text. We have developed a systematic framework for spelling and grammar error detection, as well as correction at the word level, based on a bidirectional long short-term memory mechanism and word embedding, in which a polynomial network classifier is at the top of the system. To get conclusive results, we have developed the most significant gold standard annotated corpus to date, containing 15 million fully inflected Arabic words. The data were collected from diverse text sources and genres, in which every erroneous and ill-formed word has been annotated, validated, and manually revised by Arabic specialists. This valuable asset is available for the Arabic natural language processing research community. The experimental results confirm that our proposed system significantly outperforms the performance of Microsoft Word 2013 and Open Office Ayaspell 3.4, which have been used in the literature for evaluating similar research.

Deep Bi-LSTM Networks for Sequential Recommendation

Recent years have seen a surge in approaches that combine deep learning and recommendation systems to capture user preference or item interaction evolution over time. However, the most related work only consider the sequential similarity between the items and neglects the item content feature information and the impact difference of interacted items on the next items. This paper introduces the deep bidirectional long short-term memory (LSTM) and self-attention mechanism into the sequential recommender while fusing the information of item sequences and contents. Specifically, we deal with the issues in a three-pronged attack: the improved item embedding, weight update, and the deep bidirectional LSTM preference learning. First, the user-item sequences are embedded into a low-dimensional item vector space representation via Item2vec, and the class label vectors are concatenated for each embedded item vector. Second, the embedded item vectors learn different impact weights of each item to achieve item awareness via self-attention mechanism; the embedded item vectors and corresponding weights are then fed into the bidirectional LSTM model to learn the user preference vectors. Finally, the top similar items in the preference vector space are evaluated to generate the recommendation list for users. By conducting comprehensive experiments, we demonstrate that our model outperforms the traditional recommendation algorithms on Recall@20 and Mean Reciprocal Rank (MRR@20).

Reduced Dorsal Visual Oscillatory Activity During Working Memory Maintenance in the First-Episode Schizophrenia Spectrum.

Cognitive deficits in people with schizophrenia are among the hardest to treat and strongly predict functional outcome. The ability to maintain sensory precepts in memory over a short delay is impacted early in the progression of schizophrenia and has been linked to reliable neurophysiological markers. Yet, little is known about the mechanisms of these deficits. Here, we investigated possible neurophysiological mechanisms of impaired visual short-term memory (vSTM, aka working memory maintenance) in the first-episode schizophrenia spectrum (FESz) using magnetoencephalography (MEG). Twenty-eight FESz and 25 matched controls performed a lateralized change detection task where they were cued to selectively attend and remember colors of circles presented in either the left or right peripheral visual field over a 1 s delay. Contralateral alpha suppression (CAS) during the delay period was used to assess selective attention to cued visual hemifields held in vSTM. Delay-period CAS was compared between FESz and controls and between trials presenting one vs three items per visual hemifield. CAS in dorsal visual cortex was reduced in FESz compared to controls in high-load trials, but not low-load trials. Group differences in CAS were found beginning 100 ms after the disappearance of the memory set, suggesting deficits were not due to the initial deployment of attention to the cued visual hemifield prior to stimulus presentation. CAS was not greater for high-load vs low-load trials in FESz subjects, although this effect was prominent in controls. Further, lateralized gamma (34–40 Hz) power emerged in dorsal visual cortex prior to the onset of CAS in controls but not FESz. Gamma power in this cluster differed between groups at both high and low load. CAS deficits observed in FESz were correlated with change detection accuracy, working memory function, estimated IQ, and negative symptoms. Our results implicate deficits in CAS in trials requiring broad, but not narrow, focus of attention to spatially distributed objects maintained in vSTM in FESz, possibly due to reduced ability to broadly distribute visuospatial attention (alpha) or disruption of object-location binding (gamma) during encoding/consolidation. This early pathophysiology may shed light upon mechanisms of emerging working memory deficits that are intrinsic to schizophrenia.

Multiple-Feature-Based Vehicle Supply–Demand Difference Prediction Method for Social Transportation

Big data for social transportation brings unprecedented opportunities for us to solve the transportation problems that cannot be solved by traditional methods and build the next generation of the intelligent transportation system (ITS). As one of the important functions of the ITS, supply-demand difference prediction for autonomous vehicles provides a decision basis for its control. In this article, a new learning process is proposed with Multiple feature Extraction and Fusion utilizing the combination of deep and shallow Features (MEFF) (the spatial deep features, (short and long) temporal deep features, and fuzzy shallow (semantic) features). The spatial deep features are captured with residual network and dimension reduction in spatial deep block. The fuzzy shallow (semantic) features are captured with multiattention fuzzy mechanism in the fuzzy shallow block. With the fused spatial deep features and fuzzy shallow features, the temporal deep features are captured with long short-term memory (LSTM) and attention mechanism in the temporal and prediction block to get the final prediction results. Based on two different distributions of membership attention (mean distribution and Gaussian distribution) in the fuzzy shallow block, our process MEFF has two methods, i.e., MEFF-mean method and MEFF-Gaussian method. Extensive experiments show that our methods provide more accurate and stable prediction results than the existing state-of-art-methods.

Spatial-temporal adaptive transient stability assessment for power system under missing data

Transient stability assessment (TSA) plays an important role in the design and operation of power system. With the widespread deployment of phasor measurement units (PMUs), the machine learning-based method has attracted much attention for its speed and generalization. However, the generalization will deteriorate if some features are missing due to PMU failure. In this paper, a spatial-temporal adaptive TSA method is proposed to handle the missing data issue. By developing an optimal PMU clusters searching model based on temporal feature importance, and by constructing an ensemble mechanism of long short-term memory (LSTM) for the optimal PMU clusters, the spatial-temporal information is utilized adaptively. Therefore, the aim of maintaining the robustness of TSA performance under any possible PMU failure event is achieved. The proposed approach is demonstrated on New England 39-bus power system. Compared with existing methods, the proposed method achieves state-of-art performance in both accuracy and response time under missing data conditions. In addition, the proposed method is more robust in the case of PMU failure than others.

Short-Term Plasticity at Hippocampal Mossy Fiber Synapses Is Induced by Natural Activity Patterns and Associated with Vesicle Pool Engram Formation

SummaryPost-tetanic potentiation (PTP) is an attractive candidate mechanism for hippocampus-dependent short-term memory. Although PTP has a uniquely large magnitude at hippocampal mossy fiber-CA3 pyramidal neuron synapses, it is unclear whether it can be induced by natural activity and whether its lifetime is sufficient to support short-term memory. We combined in vivo recordings from granule cells (GCs), in vitro paired recordings from mossy fiber terminals and postsynaptic CA3 neurons, and “flash and freeze” electron microscopy. PTP was induced at single synapses and showed a low induction threshold adapted to sparse GC activity in vivo. PTP was mainly generated by enlargement of the readily releasable pool of synaptic vesicles, allowing multiplicative interaction with other plasticity forms. PTP was associated with an increase in the docked vesicle pool, suggesting formation of structural “pool engrams.” Absence of presynaptic activity extended the lifetime of the potentiation, enabling prolonged information storage in the hippocampal network.