Traditional Feedforward Neural Networks Research Articles

Neural network process simulations support a distributed memory system and aid design of a novel computer adaptive digital memory test for preclinical and prodromal Alzheimer's disease.

Growing evidence supports the importance of learning as a central deficit in preclinical/prodromal Alzheimer's disease. The aims of this study were to conduct a series of neural network simulations to develop a functional understanding of a distributed, nonmodular memory system that can learn efficiently without interference. This understanding is applied to the development of a novel digital memory test. Simulations using traditional feed forward neural network architectures to learn simple logic problems are presented. The simulations demonstrate three limitations: (a) inefficiency, (b) an inability to learn problems consistently, and (c) catastrophic interference when given multiple problems. A new mirrored cascaded architecture is introduced to address these limitations, with support provided by a series of simulations. The mirrored cascaded architecture demonstrates efficient and consistent learning relative to feed forward networks but also suffers from catastrophic interference. Addition of context values to add the capability of distinguishing features as part of learning eliminates the problem of interference in the mirrored cascaded, but not the feed forward, architectures. A mirrored cascaded architecture addresses the limitations of traditional feed forward neural networks, provides support for a distributed memory system, and emphasizes the importance of context to avoid interference. These process models contributed to the design of a digital computer-adaptive word list learning test that places maximum stress on the capability to distinguish specific episodes of learning. Process simulations provide a useful method of testing models of brain function and contribute to new approaches to neuropsychological assessment. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

A novel approach to minimal reservoir computing

Reservoir computers are powerful machine learning algorithms for predicting nonlinear systems. Unlike traditional feedforward neural networks, they work on small training data sets, operate with linear optimization, and therefore require minimal computational resources. However, the traditional reservoir computer uses random matrices to define the underlying recurrent neural network and has a large number of hyperparameters that need to be optimized. Recent approaches show that randomness can be taken out by running regressions on a large library of linear and nonlinear combinations constructed from the input data and their time lags and polynomials thereof. However, for high-dimensional and nonlinear data, the number of these combinations explodes. Here, we show that a few simple changes to the traditional reservoir computer architecture further minimizing computational resources lead to significant and robust improvements in short- and long-term predictive performances compared to similar models while requiring minimal sizes of training data sets.

Correction of nonstationary tidal prediction using deep-learning neural network models in tidal estuaries and rivers

An operational tidal prediction method based on a combination of nonstationary harmonic analysis and deep-learning neural network models is described. Nonstationary harmonic analysis (NSHA), which was based on classical harmonic analysis (CHA), has been developed for tidal forecasting in tidal rivers and estuaries. However, the prediction accuracy is poor when the discharge grows very large or changes abruptly. Therefore, in this study, we aimed to combine nonstationary harmonic analysis with a deep-learning neural network model to improve tidal forecasts in tidal rivers and estuaries. The long short-term memory (LSTM) neural network model, which works well for processing long-term time series data, was chosen to correct the errors from NSHA. In addition, the traditional feed-forward neural network (FFNN) model was also applied and compared with LSTM to determine and optimize the structure of the neural network. Through experiments, the results showed that a two-layer network with a fully connected layer on top of an LSTM layer that uses discharge in addition to the previous time series as input data exhibited the best performance in predicting the errors of the NS_TIDE model. After correction by the proposed model at three stations on the West River, a branch river of the Pearl River, the root mean square error (RMSE) at 24 h by the NS_TIDE model can be reduced from approximately 0.3 m to less than 0.1 m. More specifically, the results indicated a significant improvement for extremely high-level prediction, which is crucial for water conservancy administrations. Finally, optimization approaches of the neural network to prevent overfitting and improve efficiency are also discussed.

Smart Tool Wear Monitoring of CFRP/CFRP Stack Drilling Using Autoencoders and Memory-Based Neural Networks

The drilling of carbon fiber-reinforced plastic (CFRP) materials is a key process in the aerospace industry, where ensuring high product quality is a critical issue. Low-quality of final products may be caused by the occurrence of drilling-induced defects such as delamination, which can be highly affected by the tool conditions. The abrasive carbon fibers generally produce very fast tool wear with negative effects on the hole quality. This suggests the need to develop a method able to accurately monitor the tool wear development during the drilling process in order to set up optimal tool management strategies. Nowadays, different types of sensors can be employed to acquire relevant signals associated with process variables which are useful to monitor tool wear during drilling. Moreover, the increasing computational capacity of modern computers allows the successful development of procedures based on Artificial Intelligence (AI) techniques for signal processing and decision making aimed at online tool condition monitoring. In this work, an advanced tool condition monitoring method based on the employment of autoencoders and gated recurrent unit (GRU) recurrent neural networks (RNN) is developed and implemented to estimate tool wear in the drilling of CFRP/CFRP stacks. This method exploits the automatic feature extraction capability of autoencoders to obtain relevant features from the sensor signals acquired by a multiple sensor system during the drilling process and the memory abilities of GRU to estimate tool wear based on the extracted sensor signal features. The results obtained with the proposed method are compared with other neural network approaches, such as traditional feedforward neural networks, and considerations are made on the influence that memory-based hyperparameters have on tool wear estimation performance.

Neural Network-Based Li-Ion Battery Aging Model at Accelerated C-Rate

Lithium-ion (Li-ion) batteries are widely used in electric vehicles (EVs) because of their high energy density, low self-discharge, and superior performance. Despite this, Li-ion batteries’ performance and reliability become critical as they lose their capacity with increasing charge and discharging cycles. Moreover, Li-ion batteries are subject to aging in EVs due to load variations in discharge. Monitoring the battery cycle life at various discharge rates would enable the battery management system (BMS) to implement control parameters to resolve the aging issue. In this paper, a battery lifetime degradation model is proposed at an accelerated current rate (C-rate). Furthermore, an ideal lifetime discharge rate within the standard C-rate and beyond the C-rate is proposed. The consequence of discharging at an accelerated C-rate on the cycle life of the batteries is thoroughly investigated. Moreover, the battery degradation model is investigated with a deep learning algorithm-based feed-forward neural network (FNN), and a recurrent neural network (RNN) with long short-term memory (LSTM) layer. A comparative assessment of performance of the developed models is carried out and it is shown that the LSTM-RNN battery aging model has superior performance at accelerated C-rate compared to the traditional FNN network.

A novel compound fault-tolerant method based on online sequential extreme learning machine with cycle reservoir for turbofan engine direct thrust control

Sensors are the primary information source of the aeroengine control system, their measurement accuracy is closely related to whether the engine can operate safely and efficiently. Aiming at the direct thrust control system of aeroengines, this paper proposes an intelligent prediction algorithm combining feedforward and recurrent networks and a fault-tolerant control strategy combining analytical redundancy and controller switching. First, the cycle reservoir with regular jumps is introduced into the online sequential extreme learning machine. The CR-OSELM is developed, which maps the inputs from the low-dimensional space to the high-dimensional space. Then the CR-OSELM is adopted to build the sensor measurement prediction module, and identify and reconstruct the sensor bias and drift fault. Furthermore, in allusion to the rotor speed sensor and temperature/pressure sensor faults, analytic redundancy and controller switching strategies are designed respectively, realizing the fault-tolerant control of thrust feedback. The main contribution of this paper is to come up with the CR-OSELM algorithm with the ability to save historical input information, which overcomes the limitations of traditional feedforward neural networks in identifying time series. And the compound fault-tolerant control scheme is put forward based on the fault characteristics and their impact on the direct thrust control system, which can deal with different types of sensor faults. Simulations are performed on some benchmark datasets and a turbofan engine model to investigate the time series prediction accuracy, sensor fault diagnosis, and thrust fault-tolerant control effect. The results show that the system can detect various faults rapidly, realize the analytic redundancy with high accuracy, reduce the influence of sensor faults and restore the engine performance.

SiaLog: detecting anomalies in software execution logs using the siamese network

Detecting anomalies in software logs has become a notable concern for software engineers and maintainers as they represent anomalies in software execution paths and states. This paper propose a novel anomaly detection approach based on the Siamese network on top of Recurrent Neural Networks(RNN). Accordingly, we introduce a novel training pair generation algorithm to train the Siamese network which reduces generated training significantly while maintaining the F_1 score. Additionally, we propose a hybrid model by combining the Siamese network with a traditional feedforward neural network to make end-to-end training possible, reducing engineering effort in setting up a deep-learning-based log anomaly detector. Furthermore, we provides validations of the approach on the Hadoop Distributed File System (HDFS), Blue Gene/L (BGL), and Hadoop map-reduce task log datasets. To the best of our knowledge, the proposed approach outperforms other methods on the same dataset at the F_1 scores of respectively 0.99, 0.99, and 0.94 on HDFS, BGL, and Hadoop datasets, resulting in a new state-of-the-art performance.To further evaluate the proposed method, we examine our method’s robustness to log evolutions by evaluating the model on synthetically evolved log sequences; we got the F_1 score of 0.95 on the HDFS dataset at the noise ratio of 20%. Finally, we dive deep into some of the side benefits of the Siamese network. Accordingly, we introduce an unsupervised log evolution monitoring method alongside a visualization technique that facilitates model interpretability.

An InSAR Interferogram Filtering Method Based on Multi-Level Feature Fusion CNN.

Interferogram filtering is an essential step in processing data from interferometric synthetic aperture radar (InSAR), which greatly improves the accuracy of terrain reconstruction and deformation monitoring. Most traditional interferogram filtering methods achieve noise suppression and detail preservation through morphological estimation based on the statistical properties of the interferogram in the spatial or frequency domain. However, as the interferogram’s spatial distribution is diverse and complex, traditional filtering methods struggle to adapt to different distribution and noise conditions and cannot handle detail preservation and noise suppression simultaneously. The study proposes a convolutional neural network (CNN)-based multi-level feature fusion model for interferogram filtering that differs from the traditional feedforward neural network (FNN). Adopting a multi-depth multi-path convolution strategy, the method preserves phase details and suppresses noise during interferogram filtering. In filtering experiments based on simulated data, qualitative and quantitative evaluations were used to validate the performance and generalization capabilities of the proposed method. The method’s applicability was evaluated by visual observation during filtering and unwrapping experiments on real data, and the time-series deformation acquired by time series (TS)-InSAR technique is used to evaluate the effect of interferogram filters on deformation monitoring accuracy. Compared to commonly used interferogram filtering methods, the proposed method has significant advantages in terms of performance and efficiency. The study findings suggest new directions for research on high-precision InSAR data processing and provide technical support for practical applications of InSAR.

Enhancing Real-Time Prediction of Effluent Water Quality of Wastewater Treatment Plant Based on Improved Feedforward Neural Network Coupled with Optimization Algorithm

To provide real-time prediction of wastewater treatment plant (WWTP) effluent water quality, a machine learning (ML) model was developed by combining an improved feedforward neural network (IFFNN) with an optimization algorithm. Data used as input variables of the IFFNN included hourly influent water quality parameters, influent flow rate and WWTP process monitoring and operational parameters. Additionally, input variables included historical effluent water quality parameters for future prediction. The model was demonstrated in a WWTP in Jiangsu Province, China, where prediction of effluent chemical oxygen demand (COD) and total nitrogen (TN) with large variations were tested. Relative to the traditional feedforward neural network (FFNN) model without considering historical effluent water quality parameter input, the IFFNN enhanced prediction performance by 52.3% (COD) and 72.6% (TN) based on the mean absolute percentage errors of test datasets, after its model structure was optimized with a genetic algorithm (GA). The problem of over-fitting could also be overcome through the use of the IFFNN, with the determination of coefficient increased from 0.20 to 0.76 for test datasets of effluent COD. The GA-IFFNN model, which was efficient in capturing complex non-linear relationships and extrapolation, could be a useful tool for real-time direction of regulatory changes in WWTP operations.

The New Approach Research on Singing Voice Detection Algorithm Based on Enhanced Reconstruction Residual Network

With the development of Internet technology, multimedia information resources are increasing rapidly. Faced with the massive resources in the multimedia music library, it is extremely difficult for people to find the target music that meets their needs. How to realize computer analysis and perceive users’ needs for music resources has become the goal of the future development of human-computer interaction capabilities. Content-based music information retrieval applications are mainly embodied in the automatic classification and recognition of music. Traditional feedforward neural networks are prone to lose local information when extracting singing voice features. For this reason, on the basis of fully considering the impact of information persistence in the network propagation process, this paper proposes an enhanced two-stage super-resolution reconstruction residual network which can effectively integrate the learned features of each layer while increasing the depth of the network. The first stage of reconstruction is to complete the hierarchical learning of singing voice features through dense residual units to improve the integration of information. The second stage of reconstruction is mainly to perform residual relearning on the high-frequency information of the singing voice learned in the first stage to reduce the reconstruction error. In the middle of these two stages, the model introduces feature scaling and expansion convolution to achieve the dual purpose of reducing information redundancy and increasing the receptive field of the convolution kernel. A monophonic singing voice separation based on the high-resolution neural network is proposed. Because the high-resolution network has parallel subnetworks with different resolutions, it also has original resolution representations and multiple low-resolution representations, avoiding information loss caused by serial network downsampling effects and repeating multiple feature fusions to generate new semantic representations, allowing for the learning of comprehensive, high-precision, and highly abstract features. In this article, a high-resolution neural network is utilized to model the time spectrogram in order to correctly estimate the real value of the anticipated time-amplitude spectrograms. Experiments on the dataset MIR-1K show that compared with the current leading SH-4Stack model, the method in this paper has improved SDR, SIR, and SAR indicators for measuring the separation performance, confirming the effectiveness of the algorithm in this paper.

A Deep Convolutional Neural Network With Multiscale Feature Dynamic Fusion for InSAR Phase Filtering

Interferometric phase filtering is a crucial step in the interferometric synthetic aperture radar (InSAR) data processing, which is also important for improving the accuracy of topography mapping and deformation monitoring. Most of the commonly used phase filtering methods perform windowing computations based on the statistical characteristics of a single interferogram in the spatial or frequency domain. However, the difficulty in taking into account the diversity and complexity of the phase image results in filtering methods with weak denoising, limited detail preservation, and poor generalization ability. At the same time, regardless of the spatial or frequency domain, improved phase filtering performance inevitably leads to the problem of declining effectiveness. This paper proposes a phase filtering method (MSFF-DCNN) based on the deep convolution neural network (DCNN) with Multi-scale feature dynamic fusion. Unlike the traditional feedforward neural networks (FNN), the proposed method adopts a strategy of multi-scale feature dynamic fusion that accounts for the deep and shallow features of the interferometric phase while also taking into account image detail preservation and noise suppression during phase filtering. Based on both subjective and objective evaluations, the experimental results using the simulated data prove that the proposed method has better noise suppression and detail preservation than the commonly used methods and that the filtering performance is less dependent on noise level. Experiments using the real data confirm that the proposed method has better generalization ability and can meet the precision requirements of practical applications. The method presented in this paper can provide a new approach for research in high-precision InSAR data processing technology while also offering technical support for practical InSAR applications.

A Novel Hybrid Method for KPI Anomaly Detection Based on VAE and SVDD

Key performance indicator (KPI) anomaly detection is the underlying core technology in Artificial Intelligence for IT operations (AIOps). It has an important impact on subsequent anomaly location and root cause analysis. Variational auto-encoder (VAE) is a symmetry network structure composed of encoder and decoder, which has attracted extensive attention because of its ability to capture complex KPI data features and better detection results. However, VAE is not well applied to the modeling of KPI time series data and it is often necessary to set the threshold to obtain more accurate results. In response to these problems, this paper proposes a novel hybrid method for KPI anomaly detection based on VAE and support vector data description (SVDD). This method consists of two modules: a VAE reconstructor and SVDD anomaly detector. In the VAE reconstruction module, firstly, bi-directional long short-term memory (BiLSTM) is used to replace the traditional feedforward neural network in VAE to capture the time correlation of sequences; then, batch normalization is used at the output of the encoder to prevent the disappearance of KL (Kullback–Leibler) divergence, which prevents ignoring latent variables to reconstruct data directly. Finally, exponentially weighted moving average (EWMA) is used to smooth the reconstruction error, which reduces false positives and false negatives during the detection process. In the SVDD anomaly detection module, smoothed reconstruction errors are introduced into the SVDD for training to determine the threshold of adaptively anomaly detection. Experimental results on the public dataset show that this method has a better detection effect than baseline methods.

Speech Emotion Recognition Using Deep Learning LSTM for Tamil Language

Deep Neural Networks (DNN) are more than just neural networks with several hidden units that gives better results with classification algorithm in automated voice recognition activities. Then spatial correlation was considered in traditional feedforward neural networks and which do not manage speech signal properly to it extend, so recurrent neural networks (RNNs) were implemented. Long Short-Term Memory (LSTM) systems is a unique case of RNNs for speech processing, thus considering long-term dependencies Deep Hierarchical LSTM and BiLSTM is designed with dropout layers to reduce the gradient and long-term learning error in emotional speech analysis. Thus, four different combinations of deep hierarchical learning architecture Deep Hierarchical LSTM and LSTM (DHLL), Deep Hierarchical LSTM and BiLSTM (DHLB), Deep Hierarchical BiLSTM and LSTM (DHBL) and Deep Hierarchical dual BiLSTM (DHBB) is designed with dropout layers to improve the networks. The performance test of all four model were compared in this paper and better efficiency of classification is attained with minimal dataset of Tamil Language. The experimental results show that DHLB reaches the best precision of about 84% in recognition of emotions for Tamil database, however, the DHBL gives 83% of efficiency. Other design layers also show equal performance but less than the above models DHLL & DHBB shows 81% of efficiency for lesser dataset and minimal execution and training time.

Identification modeling and prediction of ship maneuvering motion based on LSTM deep neural network

This paper proposes a novel system identification scheme to obtain a MIMO model of ship maneuvering motion, which can leverage the temporal correlation from the constructed training data to learn the underlying feasible model robust to extraneous noise. The scheme is based on long–short-term-memory (LSTM) deep neural network, which is more easily trained than traditional feedforward neural network with more complicated network structure. First, multiple datasets of simulated standard maneuvers (10°/10° and 20°/20° zigzag, 35° turning circle) of a KVLCC2 model are artificially polluted with white noise of various levels and used simultaneously to train the deep neural network model. Meanwhile, the data of 15°/15° zigzag maneuver are used to facilitate the training process to alleviate overfitting problem. Second, different datasets of modified zigzag tests are used to validate the generalization performance and robustness to noise of the trained neural network model. The training and validation results demonstrate that a mapping between the dynamics of ship motion and the computation in LSTM deep neural network is correctly identified. This mapping indicates that the complex nonlinear features of ship maneuvering motion can be learned from the measured temporal data, using standard training techniques for deep neural networks. An equivalent LSTM deep neural network model with better generalization performance and robustness is established, and its accuracy in predicting ship maneuvering motion is validated.

Optimizing Deep Learning Methods in Neural Network Architectures

Deep neural networks are a powerful tool for computer-assisted learning and have achieved significant success in numerous computer vision and image processing tasks. This paper discusses several new neural network structures that have better performance than the traditional feedforward neural network structure. A method of network structure optimization based on gradient descent and heavy-ball algorithms has been proposed. Furthermore, an approach based on the concept of sparse representation for simultaneous training and optimizing the network structure has been presented. According to CIFAR-10 and CIFAR-100 dataset classification tasks and experimental results, the optimization of ResNet and DenseNet structures using gradient descent and heavy-ball algorithms, accordingly, has been shown to result in better performance with increased depth of neural network. A neural network based on a sparse representation is shown to have the highest performance in all datasets. This strategy encourages quick data adaptation at each iteration. The results obtained can be used to design deeper neural networks with no loss of precision and computing speed.

RETRACTED: Extreme learning machine for stock price prediction

Stock market performance prediction has always been a hit research topic and is attractive due to its strong potential to generate financial profit. Being able to predict future stock price in a relatively accurate way forms a significant task of stock market analysis. Different mechanisms from fundamental analysis to statistical modeling have been deployed to study stock market performance and various factors from fundamental factors, technical factors to market sentiments are also incorporated in the stock price prediction task. However, due to the chaotic stock market performance, which is close to random walk, and the difficulty in discerning influential factors, predicting stock price faces a lot of challenges. In recent years, fast development in fields such as machine learning has offered new ways to look at this task. In this paper, we employ Extreme Learning Machine (ELM) algorithm, a recent modification of traditional feedforward neural network with single hidden layer, whose learning speed is greatly improved based on solid mathematical background and capability to circumvent problems such as local minimum is also enhanced, to construct an ELM combination model to study stock market performance and predict stock price. A comparison between the predicted output and the real data is carried out to test the feasibility of applying ELM model to stock market analysis. The result indicates that ELM model is desirable for predicting stock price variation trend while some inaccuracy exists in the prediction of peak values, which may require further model modification. Overall, by applying the machine learning model ELM to predict stock price and generating desirable outcome, this paper both contributes to offering a new way to investigate stock market performance and enlarging the field deployment of ELM model as well.

An FPGA-Based LSTM Acceleration Engine for Deep Learning Frameworks

Over the past two decades, Long Short-Term Memory (LSTM) networks have been used to solve problems that require modeling of long sequence because they can selectively remember certain patterns over a long period, thus outperforming traditional feed-forward neural networks and Recurrent Neural Network (RNN) on learning long-term dependencies. However, LSTM is characterized by feedback dependence, which limits the high parallelism of general-purpose processors such as CPU and GPU. Besides, in terms of the energy efficiency of data center applications, the high consumption of GPU and CPU computing cannot be ignored. To deal with the above problems, Field Programmable Gate Array (FPGA) is becoming an ideal alternative. FPGA has the characteristics of low power consumption and low latency, which are helpful for the acceleration and optimization of LSTM and other RNNs. This paper proposes an implementation scheme of the LSTM network acceleration engine based on FPGA and further optimizes the implementation through fixed-point arithmetic, systolic array and lookup table for nonlinear function. On this basis, for easy deployment and application, we integrate the proposed acceleration engine into Caffe, one of the most popular deep learning frameworks. Experimental results show that, compared with CPU and GPU, the FPGA-based acceleration engine can achieve performance improvement of 8.8 and 2.2 times and energy efficiency improvement of 16.9 and 9.6 times, respectively, within Caffe framework.

Beetle Antennae Search Strategy for Neural Network Model Optimization with Application to Glomerular Filtration Rate Estimation

The determination of weights is very important for neural network models. Nevertheless, the traditional feedforward neural networks usually use the method of random initial values to determine the weights of the neural networks, such a method would make the related neural network models possess unstable performance in accuracy. Therefore, in order to improve the prediction accuracy and efficiency of neural networks, a novel neural network model based on beetle antennae search (BAS) and weights and structure policy (WASP) is proposed and applied to the estimation of glomerular filtration rate (GFR). Through a series of numerical experiments, it is proved that the accuracy of the proposed neural network model for GFR estimation is higher than that of the model without BAS algorithm optimization. In addition, compared with the traditional one-step method, the method proposed in this article is more than 300% higher in the three performance indicators of MSE, MAE, and MAPE.

Efficient pricing and hedging of high-dimensional American options using deep recurrent networks

We propose a deep recurrent neural network (RNN) framework for computing prices and deltas of American options in high dimensions. Our proposed framework uses two deep RNNs, where one network learns the continuation price and the other learns the delta for each timestep. Our proposed framework yields prices and deltas for the entire spacetime, not only at a given point (e.g. t = 0). The computational cost of the proposed approach is linear in N, which improves on the quadratic time seen for feedforward networks that price American options. The computational memory cost of our method is constant in N, which is an improvement over the linear memory costs seen in feedforward networks. Our numerical simulations demonstrate these contributions and show that the proposed deep RNN framework is computationally more efficient than traditional feedforward neural network frameworks in time and memory.

A physics-constrained deep residual network for solving the sine-Gordon equation

Despite some empirical successes for solving nonlinear evolution equations using deep learning, there are several unresolved issues. First, it could not uncover the dynamical behaviors of some equations where highly nonlinear source terms are included very well. Second, the gradient exploding and vanishing problems often occur for the traditional feedforward neural networks. In this paper, we propose a new architecture that combines the deep residual neural network with some underlying physical laws. Using the sine-Gordon equation as an example, we show that the numerical result is in good agreement with the exact soliton solution. In addition, a lot of numerical experiments show that the model is robust under small perturbations to a certain extent.