Reservoir Inflow Modeling Using Temporal Neural Networks with Forgetting Factor Approach

This paper analyses the performance of different types of Deep Neural Networks to jointly estimate age and identify gender from speech, to be applied in Interactive Voice Response systems available in call centres. Deep Neural Networks are used, because they have recently demonstrated discriminative and representation capabilities in a wide range of applications, including speech processing problems based on feature extraction and selection. Networks with different sizes are analysed to obtain information on how performance depends on the network architecture and the number of free parameters. The speech corpus used for the experiments is Mozilla’s Common Voice dataset, an open and crowdsourced speech corpus. The results are really good for gender classification, independently of the type of neural network, but improve with the network size. Regarding the classification by age groups, the combination of convolutional neural networks and temporal neural networks seems to be the best option among the analysed, and again, the larger the size of the network, the better the results. The results are promising for use in IVR systems, with the best systems achieving a gender identification error of less than 2% and a classification error by age group of less than 20%.

Diel dissolved oxygen (DO) time series measured continuously using proximal sensors in situ for a temperate lake were denoised using discrete wavelet transform (DWT) with the orthogonal wavelet families of coiflet, daubechies, and symmlet with order of 10. Diel DO time series denoised were modeled using nine temporal artificial neural networks (ANNs) as a function of water level, water temperature, electrical conductivity, pH, day of year, and hour. Our results showed that time-lag recurrent network (TLRN) using denoised data emulated diel DO dynamics better than the best-performing TLRN using the original data, time-delay neural network (TDNN), and recurrent network (RNN). Daubechies basis dealt with diel DO data slightly better than the other bases given its coefficient of determination (r (2) = 87.1%), while symmlet performed slightly better than the other bases in terms of root mean square error (RMSE = 1.2ppm) and mean absolute error (MAE = 0.9ppm).

The proposal of a Corporate Governance Model called Service-Focused Operation Methodology (MOCA) was carried out, applied in Public and Private Partnerships (PPP) to improve services quality offered by the Brazilian states. This PPP model enabled several Service Center (in portuguese Central de Atendimento—CA) implementation projects supported by several multidisciplinary knowledge areas that involve projects and governments. However, this article explored an aspect of how a MOCA’s use with new technologies embedded in projects provide continuous improvements in results. In this case, for example, a demand study was applied to Planning and Control of Operations (PCO) in a use of Research and Development (RD MOCA applied in PCO; obtained from stabilized proof of concepts; providing data collection and more accurate performance information in each CA, collected directly by an ERP used. From these data, the design of service production lines was performed using the following methodologies: (1) Descriptive Statistics, (2) Temporal Series and (3) Temporal Underground Neural Networks (ANNT). A Temporal Neural Networks (ANNT) was obtained, using recursive corrections in demand balancing by attendant performance. Using these technologies, a more accurate performance forecast to estimates attendants work was achieved in order to obtain a more realistic operational planning.

Temporal Neural Networks (TNNs), inspired from the mammalian neocortex, exhibit energy-efficient online sensory processing capabilities. Recent works have proposed a microar-chitecture framework for implementing TNNs and demonstrated competitive performance on vision and time-series applications. Building on these previous works, this work proposes TNN7, a suite of nine highly optimized custom macros developed using a predictive 7nm Process Design Kit (PDK), to enhance the efficiency, modularity and flexibility of the TNN design framework. TNN prototypes for two applications are used for evaluation of TNN7. An unsupervised time-series clustering TNN delivering competitive performance can be implemented within 40 uW power and 0.05 <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{mm}^{2}$</tex> area, while a 4-layer TNN that achieves an MNIST error rate of 1% consumes only 18 mW and 24.63 <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{mm}^{2}$</tex> . On average, the proposed macros reduce power, delay, area, and energy-delay product by 14%, 16%, 28%, and 45 %, respectively. Furthermore, employing TNN7 significantly reduces the synthesis runtime of TNN designs (by more than 3x), allowing for highly-scaled TNN implementations to be realized.

Temporal Neural Networks (TNNs) are spiking neural networks that use time as a resource to represent and process information, similar to the mammalian neocortex. In contrast to compute-intensive deep neural networks that employ separate training and inference phases, TNNs are capable of extremely efficient online incremental/continual learning and are excellent candidates for building edge-native sensory processing units. This work proposes a microarchitecture framework for implementing TNNs using standard CMOS. Gate-level implementations of three key building blocks are presented: 1) multi-synapse neurons, 2) multi-neuron columns, and 3) unsupervised and supervised online learning algorithms based on Spike Timing Dependent Plasticity (STDP). The proposed microarchitecture is embodied in a set of characteristic scaling equations for assessing the gate count, area, delay and power for any TNN design. Post-synthesis results (in 45nm CMOS) for the proposed designs are presented, and their online incremental learning capability is demonstrated.

Temporal Neural Networks (TNNs), a special class of spiking neural networks, draw inspiration from the neocortex in utilizing spike-timings for information processing. Recent works proposed a microarchitecture framework and custom macro suite for designing highly energy-efficient application-specific TNNs. This paper introduces TNNGen, a pioneering effort towards the automated design of TNNs from PyTorch software models to post-layout netlists. TNNGen comprises a novel PyTorch functional simulator for TNN modeling and application exploration and a Python-based hardware generator for PyTorch-to-RTL and RTL-to-Layout conversions. Seven representative TNN designs for time-series signal clustering across diverse sensory modalities are simulated, and their post-layout hardware complexity and design process runtimes are assessed to demonstrate the effectiveness of TNNGen. We also show TNNGen’s ability to forecast silicon metrics accurately without running the hardware process flow.

A novel deterministic approach to the convergence analysis of (stochastic) temporal neural networks is presented. The link between the two is a new concept of time-average invariance (TAI) which is a property of deterministic signals but with applications to stochastic signals. With this new concept, the conventional ODE method can be extended to the case of constant learning rate. With weaker conditions, not requiring mutually independence, it is shown that a temporal neural network is /spl epsiv/-convergent to x/sup 0/, if its associated (autonomous) equations are asymptotically stable at x/sup 0/. This result is then extended to the case of perturbed TAI signals. A temporal neural network for blind signal separation is used as an example.

Biological neural networks have always motivated creation of new artificial neural networks, and in this case a new autonomous temporal neural network system. Among the more challenging problems of temporal neural networks are the design and incorporation of short and long-term memories as well as the choice of network topology and training mechanism. In general, delayed copies of network signals can form short-term memory (STM), providing a limited temporal history of events similar to FIR filters, whereas the synaptic connection strengths as well as delayed feedback loops (ER circuits) can constitute longer-term memories (LTM). This dissertation introduces a new general evolutionary temporal neural network framework (GETnet) through automatic design of arbitrary neural networks with STM and LTM. GETnet is a step towards realization of general intelligent systems that need minimum or no human intervention and can be applied to a broad range of problems. GETnet utilizes nonlinear moving average/autoregressive nodes and sub-circuits that are trained by enhanced gradient descent and evolutionary search in terms of architecture, synaptic delay, and synaptic weight spaces. The mixture of Lamarckian and Darwinian evolutionary mechanisms facilitates the Baldwin effect and speeds up the hybrid training. The ability to evolve arbitrary adaptive time-delay connections enables GETnet to find novel answers to many classification and system identification tasks expressed in the general form of desired multidimensional input and output signals. Simulations using Mackey-Glass chaotic time series and fingerprint perspiration-induced temporal variations are given to demonstrate the above stated capabilities of GETnet.

An experiment on predicting multivariate water resource time series, specifically the prediction of hydropower reservoir inflow using temporal neural networks, is presented. This paper focuses on dynamic neural networks to address the temporal relationships of the hydrological series. Three types of temporal neural network architectures with different inherent representations of temporal information are investigated. An input delayed neural network (IDNN) and a recurrent neural network (RNN) with and without input time delays are proposed for multivariate reservoir inflow forecasting. The forecast results indicate that, overall, the RNN obtained the best performance. The results also suggest that the use of input time delays significantly improves the conventional multilayer perceptron (MLP) network but does not provide any improvement in the RNN model. However, the RNN with input time delays remains slightly more effective for multivariate reservoir inflow prediction than the IDNN model. Moreover, it is found that the conventional MLP network widely used in hydrological applications is less effective at multivariate reservoir inflow forecasting than the proposed models. Furthermore, the experiment shows that employing only time-delayed recurrences can be the more effective and less costly method for multivariate water resources time series prediction.

This paper describes a new temporal rough neural network which consists of a combination of rough set and temporal concept. Temporal factors are combined with the input of neural network. That is to say, input of Neural Network is function of time, so conventional neurons are reconstructed temporal neurons. Neurons of temporal rough neural network is temporal rough neurons, they use pairs of upper and lower bounds as values for input and output and as variability for time. In some practical situations, it is preferable to develop prediction models that use ranges as values for input and/or output variables and as variability for time. A need to provide tolerance these ranges is an example of such a situation. Inability to record precise values of the variables is another situation where ranges of values must be used. In the example used in this study, a number of input values are associated with a single value of the output variable and with time. Hence, it seems appropriate to represent the input values as ranges. Temporal rough neural network can depict these situations and provide a better solution.

Comparison of the ARMA, ARIMA, and the autoregressive artificial neural network models in forecasting the monthly inflow of Dez dam reservoir

Modeling of land surface radiation budget and its components is essential to a better understanding of soil-vegetation-atmosphere interactions. Time Delay Neural Network (TDNN) and Time Lag Recurrent Network (TLRN) models were used to emulate all the hourly surface radiation components for a temperate peatland during day and night under all-sky and -surface conditions. Sensitivity analyses of full versus reduced models, daytime versus nighttime periods, and TDNN versus TLRN models were carried out using training-, cross-validation, and testing-derived metrics of root mean squared error (RMSE), mean absolute error (MAE), and coefficient of determination (R2) for each of the components. The full daytime temporal neural network models performed best based on RMSE of 2.3 W m−2 in downwelling longwave radiation to 112.2 W m−2 in upwelling shortwave radiation; R2 of 0.52 in downwelling longwave radiation to 0.88 in net shortwave radiation and net radiation; and MAE of 1.73 W m−2 in downwelling longwave radiation to 90.57 W m−2 in upwelling shortwave radiation. The best nighttime TDNN models led to RMSE values that ranged from 4.4 W m−2 in downwelling longwave radiation to 9.3 W m−2 in upwelling longwave radiation; R2 values that ranged from 0.38 in net longwave radiation to 0.60 in downwelling longwave radiation; and MAE values that ranged from 4.1 W m−2 in downwelling longwave radiation to 8.1 W m−2 in upwelling longwave radiation. Temporal neural networks used in this study appear to be a promising approach to predict nonlinear behaviors of the daytime and nighttime surface radiation components.

This paper proposes a novel method using deep spatial-temporal neural networks based on deep Convolutional Neural Network (CNN) for multimedia event detection. To sufficiently take advantage of the motion and appearance information of events from videos, our networks contain two branches: a temporal neural network and a spatial neural network. The temporal neural network captures motion information by Recurrent Neural Networks with the mutation of gated recurrent unit. The spatial neural network catches object information by using the deep CNN, to encode the CNN features as a bag of semantics with more discriminative representations. Both the temporal and spatial features are beneficial for event detection in a fully coupled way. Finally, we employ the generalized multiple kernel learning method to effectively fuse these two types of heterogeneous and complementary features for action recognition. Experiments on TRECVID MEDTest 14 dataset show that our method achieves better performance than the state of the art.

We present a method of fault detection and diagnosis in turbine engines using temporal neural networks. Temporal neural networks allow us to represent the complete engine operating range by complementing the first-principle models which are usually restricted to takeoff and cruise phases. Because faults that are manifest only in particular phases can be detected, complete coverage leads to more accurate anomaly detection and fault diagnosis systems. The time series sensor data from the engine is collected during particular aircraft flight phases such as startup, takeoff, cruise, and shutdown. We use the echo state network to develop an incipient fault detection and diagnosis system. Echo state networks have several advantages over conventional types of temporal neural networks, including accuracy and ease of training. We demonstrate the efficacy of using the echo state networks to focus on flight phases that are difficult to model. We present results of our fault detection and diagnosis method with actual propulsion engine transient flight data.

Spatiotemporal forecasting has been attracting tremendous interest in various fields, among which traffic flow prediction is a representative example. Existing methods typically deal with the complex spatial and temporal dependencies in traffic flow through graph neural networks (GNNs) and temporal neural networks (TNNs), respectively. However, these works still fall short due to: 1) deep GNNs have the over-smoothing problem that hinders the handling of higher-order spatial correlations; 2) TNNs have difficulty in extracting the temporal dependencies with different localities. To this end, this paper proposes Higher-Order Masked Graph Neural Networks (HOMGNNs) to model and predict the traffic flow data. Concretely, we design the spatial graph learning layer to adaptively characterize the dependency correlations of different orders, and the higher-order GNN (HOGNN) is further proposed to deal with these correlations. Furthermore, we define and construct a temporal graph to represent the temporal dynamics in traffic data. The masked GNN (MGNN) is further proposed to extract these dynamics based on the temporal graph. To validate the superiority of the proposed HOMGNNs, we conduct extensive experiments on METR-LA and PEMS-BAY datasets. Experimental results demonstrate the remarkable performance of our method compared with 11 state-of-the-art baselines.