Continuous-time Neural Networks Research Articles

Closed-Form Continuous-Time Neural Networks for Sliding Mode Control with Neural Gravity Compensation

Closed-Form Continuous-Time Neural Networks for Sliding Mode Control with Neural Gravity Compensation

Evolutionary End-to-End Autonomous Driving Model With Continuous-Time Neural Networks

Evolutionary End-to-End Autonomous Driving Model With Continuous-Time Neural Networks

Long-term prediction method for PM2.5 concentration using edge channel graph attention network and gating closed-form continuous-time neural networks

Fine particulate matter such as PM2.5 threatens significantly to the environment and human health, so it is essential to design a reliable long-term prediction method for PM2.5 concentrations. Existing long-term PM2.5 prediction models inadequately utilize urban spatial features, fail to consider the role of meteorological factors in PM2.5 levels, and overlook the interaction between PM2.5 concentrations in different cities. To tackle this issue, we propose two new models and integrate them. Firstly, we develop a spatial feature model (ECGAT) for extracting PM2.5 concentration among regions based on Graph Neural Networks (GNN), edge-channel mechanisms, and Graph Attention Convolution (GATConv). This model utilizes GNN to extract urban adjacency relationships and meteorological features, employs edge-channel mechanisms to recalculate weights for interactions between cities, and outputs spatial correlations through GATConv. Secondly, we propose Gating Closed-form Continuous-time Neural Networks (GCFC) as a temporal model to extract the PM2.5 concentration's temporal features. The fusion of these two models, named ECGAT-GCFC (EGCFC), enhances the model's capability to capture spatiotemporal features and improves performance in PM2.5 long-term predictions. Results from real-world data analysis show that the proposed algorithm outperforms state-of-the-art existing prediction models in predicting PM2.5 levels over long durations. Compared to baseline models, EGCFC reduces RMSE by an average of 3.39 %, decreases MAE by 4.83 %, increases R2 by 4.89 %, CSI by 3.13 %, and lowers FAR by 11.39 %. These indicate that EGCFC is an effective method for predicting trends in urban PM2.5 concentrations.

Signal Propagation: The Framework for Learning and Inference in a Forward Pass.

We propose a new learning framework, signal propagation (sigprop), for propagating a learning signal and updating neural network parameters via a forward pass, as an alternative to backpropagation (BP). In sigprop, there is only the forward path for inference and learning. So, there are no structural or computational constraints necessary for learning to take place, beyond the inference model itself, such as feedback connectivity, weight transport, or a backward pass, which exist under BP-based approaches. That is, sigprop enables global supervised learning with only a forward path. This is ideal for parallel training of layers or modules. In biology, this explains how neurons without feedback connections can still receive a global learning signal. In hardware, this provides an approach for global supervised learning without backward connectivity. Sigprop by construction has compatibility with models of learning in the brain and in hardware than BP, including alternative approaches relaxing learning constraints. We also demonstrate that sigprop is more efficient in time and memory than they are. To further explain the behavior of sigprop, we provide evidence that sigprop provides useful learning signals in context to BP. To further support relevance to biological and hardware learning, we use sigprop to train continuous time neural networks with the Hebbian updates and train spiking neural networks (SNNs) with only the voltage or with biologically and hardware-compatible surrogate functions.

Unified predictive modeling of supercontinuum spectra: Using multi-material data with Closed-Form Continuous Time Neural Networks

This paper explores the application of the Closed-Form Continuous-Time Neural Networks (CfC) model in predicting supercontinuum spectra in a unified dataset of various core materials in a planar waveguide. Through numerical simulations and dataset generation, the study constructs a robust model capable of accurately predicting spectral behavior under different conditions and material variation. Initially, the unified dataset comprises spectral data for three materials: Silicon Nitride (Si3N4), Lithium Niobate (LiNbO3), and Silicon Carbide (SiC). The CfC model demonstrates remarkable accuracy in capturing spectral nuances and exhibits a minimal Mean Squared Error (MSE) loss value of 7.2537×10−4. Subsequently, the dataset is expanded to include Tantalum Pentoxide (Ta2O5), introducing a fourth material for evaluation. The inclusion of Ta2O5 data further validates the model’s scalability and generalization capabilities with Mean Squared Error (MSE) loss value of 6.7894×10−4. The advantage of the CfC model in precisely predicting supercontinuum spectra while retaining computational efficiency is demonstrated by comparisons with conventional methods. The CfC model is a viable tool for enhancing predictive modeling in photonics research because of its scalability and generalization characteristics. This will pave the way for future developments in deep learning techniques for optical device optimization.

Cardiac abnormality detection with a tiny diagonal state space model based on sequential liquid neural processing unit

This manuscript introduces a novel method for cardiac abnormality detection by combining the Diagonal State Space Sequence (S4D) model with the Closed-form Continuous-time neural network (CfC), yielding a highly effective, robust, generalizable, and compact solution. Our proposed S4D-CfC model is evaluated on 12- and single-lead electrocardiogram data from over 20 000 patients. The system exhibits validation results with strong average F1 score and average area under the receiver operating characteristic curve values of 0.88% and 98%, respectively. To demonstrate the tiny machine learning of our 242 KB size model, we deployed the system on relatively resource-constrained hardware to evaluate its training performance on-the-edge. Such on-device fine-tuning can enhance personalized solutions in this context, allowing the system to learn each patient’s data features. A comparison with a structured 2D convolutional long short-term memory CfC model demonstrates the S4D-CfC model’s superior performance. The proposed model’s size can be significantly reduced to 25 KB, maintaining reasonable performance on 2.5 s data, 75% shorter than the original 10 s data, making it suitable for resource-constrained hardware and minimizing latency. In summary, the S4D-CfC model represents a groundbreaking advancement in cardiac abnormality detection, offering robustness, generalization, and practicality with the potential for efficient deployment on limited-resource platforms, revolutionizing healthcare technology.

AC-CfC: An attention-based convolutional closed-form continuous-time neural network for raw multi-channel EEG-based emotion recognition

Emotion recognition based on electroencephalogram (EEG) is a critical task in the field of affective brain-computer interfaces. However, due to the non-stationarity and individual variability of EEG, hand-designed features cannot adequately capture the nonlinear and high-dimensional properties of raw EEG. Currently, spatial–temporal models have been verified to effectively capture the spatial–temporal information for EEG. However, these models are characterized by complex structures, large parameter counts, and the need for extensive training data. To overcome these disadvantages, this paper propose an end-to-end spatial–temporal model for emotion recognition based on raw EEG, called attention-based convolutional closed-form continuous-time neural network (AC-CfC). This model employs a channel attention mechanism to weight and encode EEG, capturing channel dependencies. Subsequently, one-dimensional convolutional neural networks and closed-form continuous-time neural networks are used to extract deep spatial–temporal features. Additionally, an adaptive loss-controlling mechanism is designed to enhance the model's decision-making ability between classes that are easily confused. To verify the effectiveness of the proposed model, experiments are conducted on the DEAP and DREAMER datasets. The average accuracies of the proposed model reach to 94.76% and 93.01% for valence and arousal in subject-independent experiments on the DEAP dataset, with an improvement of 11.8% and 8.73% respectively compared with the state-of-art model ACRNN. On the DREAMER dataset, the average accuracies reach to 81.83%, 81.22%, and 80.63% for valence, arousal and dominance, with an improvement of 2.55%, 6.64%, and 6.98% respectively over ACRNN. These results show that our proposed model exhibits better performance than some state-of-art models in the same category.

Efficient Edge-AI Models for Robust ECG Abnormality Detection on Resource-Constrained Hardware

This study introduces two models, ConvLSTM2D-liquid time-constant network (CLTC) and ConvLSTM2D-closed-form continuous-time neural network (CCfC), designed for abnormality identification using electrocardiogram (ECG) data. Trained on the Telehealth Network of Minas Gerais (TNMG) subset dataset, both models were evaluated for their performance, generalizability capacity, and resilience. They demonstrated comparable results in terms of F1 scores and AUROC values. The CCfC model achieved slightly higher accuracy, while the CLTC model showed better handling of empty channels. Remarkably, the models were successfully deployed on a resource-constrained microcontroller, proving their suitability for edge device applications. Generalization capabilities were confirmed through the evaluation on the China Physiological Signal Challenge 2018 (CPSC) dataset. The models’ efficient resource utilization, occupying 70.6% of memory and 9.4% of flash memory, makes them promising candidates for real-world healthcare applications. Overall, this research advances abnormality identification in ECG data, contributing to the progress of AI in healthcare.Graphical

H∞ state estimation of continuous-time neural networks with uncertainties.

[Formula: see text] state estimation is addressed for continuous-time neural networks in the paper. The norm-bounded uncertainties are considered in communication neural networks. For the considered neural networks with uncertainties, a reduced-order [Formula: see text] state estimator is designed, which makes that the error dynamics is exponentially stable and has weighted [Formula: see text] performance index by Lyapunov function method. Moreover, it is also given the devised method of the reduced-order [Formula: see text] state estimator. Then, considering that sampling the output y(t) of the neural network at every moment will result in waste of excess resources, the event-triggered sampling strategy is used to solve the oversampling problem. In addition, a devised method is also given for the event-triggered reduced-order [Formula: see text] state estimator. Finally, by the well-known Tunnel Diode Circuit example, it shows that a lower order state estimator can be designed under the premise of maintaining the same weighted [Formula: see text] performance index, and using the event-triggered sampling method can reduce the computational and time costs and save communication resources.

State identification for a class of uncertain switched systems by differential neural networks

ABSTRACT This paper presents a non-parametric identification scheme for a class of uncertain switched nonlinear systems based on continuous-time neural networks. This scheme is based on a continuous neural network identifier. This adaptive identifier guaranteed the convergence of the identification errors to a small vicinity of the origin. The convergence of the identification error was determined by the Lyapunov theory supported by a practical stability variation for switched systems. The same stability analysis generated the learning laws that adjust the identifier structure. The upper bound of the convergence region was characterized in terms of uncertainties and noises affecting the switched system. A second finite-time convergence learning law was also developed to describe an alternative way of forcing the identification error’s stability. The study presented in this paper described a formal technique for analysing the application of adaptive identifiers based on continuous neural networks for uncertain switched systems. The identifier was tested for two basic problems: a simple mechanical system and a switched representation of the human gait model. In both cases, accurate results for the identification problem were achieved.

Non-Euclidean Contraction Analysis of Continuous-Time Neural Networks

Non-Euclidean Contraction Analysis of Continuous-Time Neural Networks

Combined wind speed forecasting model based on secondary decomposition and quantile regression closed-form continuous-time neural network

ABSTRACT Reliable interval forecasts can quantify the potential risk of wind speeds, which can help people make better use of wind energy. In this study, a new probabilistic prediction model, called Quantile Regression Closed-Form Continuous-time Neural Network (QRCfC), is proposed by combining quantile regression and closed-form continuous-time neural network. A new combined model combining QRCfC, secondary decomposition, multi-objective optimization, and dynamic weight combination strategy is proposed, which makes full use of the advantages of each single model to obtain accurate deterministic prediction and reliable probabilistic interval prediction of wind speed. First, the original wind speed series is divided into several subseries using a secondary decomposition technique built on variational mode decomposition and singular spectrum analysis. Then, four base models are used to predict these subseries separately. After that, the predicted values of the four base models are input into QRCfC for training, where the hyperparameters of QRCfC are dynamically adjusted by a multi-objective ant lion optimization algorithm. Finally, to verify the effectiveness of the proposed models, experiments are conducted using data sets from three wind farms in Gansu, China. Simulation results show that the proposed model achieves excellent prediction performance in both deterministic and probabilistic prediction. For example, compared with the unoptimized quantile regression time convolution network, quantile regression long-short time memory, quantile regression gated recurrent unit, and quantile regression neural network, the proposed model reduces the MAPE by 21.89% and the CRPS by 11.14% for 1-step prediction at site 1.

Two analog neural models with the controllability on number of assets for sparse portfolio design

The aim of this research is to use continuous time neural networks for solving the sparse portfolio problem, where the objective function includes a non-differentiable ℓ1-norm regularization term. To address the non-differentiable issue, we introduce two novel continuous time neural models that are based on the Lagrange programming neural network (LPNN) framework. The proposed neural models have the ability to control the number of the assets in the resulting portfolio and adjust the weighting between risk and return. The first model, called LPNN-Approximation, handles the non-differentiable ℓ1-norm term by utilizing a differentiable approximation. The second model merges the locally competitive algorithm (LCA) concept with the LPNN framework to address the non-differentiable term. It is called LPNN-LCA. For the LPNN-Approximation, we prove that the state of the network globally converges to the optimal solution of the sparse portfolio problem. Meanwhile, for the LPNN-LCA, we prove that all the equilibrium points of its dynamics correspond to the optimal solution of the sparse portfolio problem and are asymptotically stable. In addition, the realizations of the two models are discussed. Specifically, a thorough circuit realization for the thresholding elements of the LPNN-LCA model is presented. The effectiveness of the proposed approaches is verified by a number of numerical experiments, which show that the two proposed models outperform two state-of-the-art analog models.

Model predictive control of nonlinear processes using neural ordinary differential equation models

Neural Ordinary Differential Equation (NODE) is a recently proposed family of deep learning models that can perform a continuous approximation of a linear/nonlinear dynamic system using time-series data by integrating the neural network model with classical ordinary differential equation solvers. Modeling nonlinear dynamic processes using data has historically been a critical challenge in the field of chemical engineering research. With the development of computer science and neural network technology, recurrent neural networks (RNN) have become a popular black-box approach to accomplish this task and have been utilized to design model predictive control (MPC) systems. However, as a discrete-time approximation model, RNN requires strictly uniform step sequence data to operate, which makes it less robust to an irregular sampling scenario, such as missing data points during operation due to sensor failure or other types of random errors. This paper aims to develop a NODE model and construct an MPC based on this novel continuous-time neural network model. In this context, closed-loop stability is established and robustness to noise is addressed using a variety of data analysis techniques. An example of a chemical process is utilized to evaluate the performance of NODE-based MPC. Furthermore, the performance of the NODE-based MPC under Gaussian and non-Gaussian noise is investigated, and the subsampling method is found to be effective in building models for MPC that are suitable for handling the presence of non-Gaussian noise in the data.

Quasi-Synchronization of Timescale-Type Delayed Neural Networks With Parameter Mismatches via Impulsive Control

Most synchronization criteria are scale-free on time evolution, whose main research objects are discrete-time/continuous-time systems. Unlike these theoretical results, in order to develop impulsive control schemes for discrete-time and continuous-time neural networks (NNs) in a unified framework. This article investigates impulsive control design of timescale-type NNs (TNNs) with parameter mismatches and time-varying delays (TVDs). First, several timescale impulsive differential inequalities are demonstrated by the timescale theory, which offer new inequality techniques for the investigation of timescale-type impulsive systems. Next, some criteria are proved for discrete-time NNs and TNNs by utilizing impulsive control theory, timescale inequality techniques, and the average impulsive interval method. Unlike the published works, this article gives some impulsive control schemes to ensure quasi-synchronization (QS) even if there exist TVDs in TNNs. In the end, four simulation examples are offered to demonstrate the validness of the obtained theoretical results.

Stability Criteria of Delayed Memristor-Based Neural Networks via Continuous-Time Model and Interval Matrix Approach

Modeling and stability analysis of memristor-based neural networks (MNNs) are the premise of designs and applications. Different from most previous research, delayed MNNs are described by continuous differential equations with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2^{2n^{2}+n}$</tex-math> </inline-formula> variables where the memductances of memristors are continuously dependent on the fluxes. Both system delays and input delays are considered, and the delays and their derivatives may vary in intervals whose lower bounds are not restricted to be zero. The systems are further reduced to continuous-time neural networks (NNs) with interval matrix uncertainties, and a unified method is developed to solve the stability of delayed NNs and MNNs. Stability criteria are obtained for delayed MNNs by augmented Lyapunov functionals, Wirtinger-based integral inequality, reciprocally convex approach, and linear matrix inequalities. In the end, two numerical simulations are used to demonstrate the validity of our theorems.

Impulsive Stabilization of Nonautonomous Timescale-Type Neural Networks With Constant and Unbounded Time-Varying Delays

This article focuses on the impulsive stabilization of nonautonomous timescale-type neural networks (NTNNs) with constant and unbounded time-varying delays. By choosing two discontinuous piecewise linear functions and employing the convex combination method, several algebraic criteria are demonstrated to achieve globally asymptotic stabilization of NTNNs with constant and unbounded time-varying delays. First, the asymptotic stability theorem for timescale-type systems is constructed by utilizing the timescale theory. Based on this asymptotic stability theorem, an impulsive control scheme is designed to guarantee global asymptotic stabilization of NTNNs with constant delay. Second, we propose several impulsive control schemes to achieve globally asymptotic stabilization of NTNNs with unbounded delay by virtue of the comparison strategy. Globally asymptotic stabilization criteria for NTNNs include the theoretical results of continuous-time neural networks (NNs), their discrete-time forms and NNs on continuous-discrete hybrid time scales. Especially, impulsive control for globally asymptotic stabilization of NTNNs with proportional delay is demonstrated without any variable transformation. Finally, the effectiveness of our proposed theoretical results is verified by four numerical examples.

Closed-form continuous-time neural networks

Continuous-time neural networks are a class of machine learning systems that can tackle representation learning on spatiotemporal decision-making tasks. These models are typically represented by continuous differential equations. However, their expressive power when they are deployed on computers is bottlenecked by numerical differential equation solvers. This limitation has notably slowed down the scaling and understanding of numerous natural physical phenomena such as the dynamics of nervous systems. Ideally, we would circumvent this bottleneck by solving the given dynamical system in closed form. This is known to be intractable in general. Here, we show that it is possible to closely approximate the interaction between neurons and synapses—the building blocks of natural and artificial neural networks—constructed by liquid time-constant networks efficiently in closed form. To this end, we compute a tightly bounded approximation of the solution of an integral appearing in liquid time-constant dynamics that has had no known closed-form solution so far. This closed-form solution impacts the design of continuous-time and continuous-depth neural models. For instance, since time appears explicitly in closed form, the formulation relaxes the need for complex numerical solvers. Consequently, we obtain models that are between one and five orders of magnitude faster in training and inference compared with differential equation-based counterparts. More importantly, in contrast to ordinary differential equation-based continuous networks, closed-form networks can scale remarkably well compared with other deep learning instances. Lastly, as these models are derived from liquid networks, they show good performance in time-series modelling compared with advanced recurrent neural network models.

Accelerated convergent zeroing neurodynamics models for solving multi-linear systems with [formula omitted]-tensors

As a special type of neurodynamic methodology dedicated to finding zeros of equations, zeroing neurodynamics has shown a powerful ability in solving challenging online time-varying problems. Multi-linear systems, on the other hand, are a type of tensor equations with a wide range of applications. In this paper, a zeroing neurodynamics approach for solving multi-linear systems with M-tensors is proposed along with three specific accelerated finite-time convergent zeroing neurodynamics models, which breaks the limitations of a recently presented neural network approach termed continuous time neural network. Activation functions available in constructing zeroing neurodynamics models for solving multi-linear systems with M-tensors are also investigated in the rest of the paper. Theoretical analyses prove that the proposed zeroing neurodynamics approach is stable in the sense of Lyapunov stability theory and the proposed models converge to the theoretical solution in finite time. Finally, computer simulations are provided to substantiate the efficacy and superiority of the proposed zeroing neurodynamics models for solving multi-linear systems with M-tensors.

Almost automorphic solutions for Clifford-valued neutral-type fuzzy cellular neural networks with leakage delays on time scales

We propose a class of Clifford-valued neutral-type fuzzy cellular neural network models with leakage delays on time scales, give the definition and basic properties of Clifford-valued almost automorphic functions on time scales, and prove a composition theorem of Clifford-valued almost automorphic functions on time scales, and on this basis, we study the existence and global exponential stability of almost automorphic solutions of this class of neural networks. The research in this paper can unify the research of real-valued neural networks, complex-valued neural networks and quaternion-valued neural networks, as well as the research of continuous-time neural networks and discrete-time neural networks. Finally, an example illustrates the feasibility of the results obtained.