Oscillatory Neural Networks Research Articles

Energy-Performance Assessment of Oscillatory Neural Networks Based on VO2 Devices for Future Edge AI Computing.

Oscillatory neural network (ONN) is an emerging neuromorphic architecture composed of oscillators that implement neurons and are coupled by synapses. ONNs exhibit rich dynamics and associative properties, which can be used to solve problems in the analog domain according to the paradigm let physics compute. For example, compact oscillators made of VO2 material are good candidates for building low-power ONN architectures dedicated to AI applications at the edge, like pattern recognition. However, little is known about the ONN scalability and its performance when implemented in hardware. Before deploying ONN, it is necessary to assess its computation time, energy consumption, performance, and accuracy for a given application. Here, we consider a VO2-oscillator as an ONN building block and perform circuit-level simulations to evaluate the ONN performances at the architecture level. Notably, we investigate how the ONN computation time, energy, and memory capacity scale with the number of oscillators. It appears that the ONN energy grows linearly when scaling up the network, making it suitable for large-scale integration at the edge. Furthermore, we investigate the design knobs for minimizing the ONN energy. Assisted by technology computer-aided design (TCAD) simulations, we report on scaling down the dimensions of VO2 devices in crossbar (CB) geometry to decrease the oscillator voltage and energy. We benchmark ONN versus state-of-the-art architectures and observe that the ONN paradigm is a competitive energy-efficient solution for scaled VO2 devices oscillating above 100 MHz. Finally, we present how ONN can efficiently detect edges in images captured on low-power edge devices and compare the results with Sobel and Canny edge detectors.

Data Clustering by Chaotic Oscillatory NeuralNetworks with Dipole Synaptic Connections

Data Clustering by Chaotic Oscillatory NeuralNetworks with Dipole Synaptic Connections

Binary system modes of matrix-coupled multidimensional oscillators

The standard Kuramoto model has been instrumental in explaining synchronization and desynchronization, two emergent phenomena often observed in biological, neuronal, and physical systems. While the Kuramoto model has turned out effective with one-dimensional oscillators, real-world systems often involve high-dimensional interacting units, such as biological swarms, necessitating a model of multidimensional oscillators. However, existing high-dimensional generalizations of the Kuramoto model commonly rely on a scalar-valued coupling strength, which limits their ability to capture the full complexity of high-dimensional interactions. This work introduces a matrix, A, to couple the interconnected components of the oscillators in a d-dimensional space, leading to a matrix-coupled multidimensional Kuramoto model that approximates a prototypical swarm dynamics by its first-order Fourier harmonics. Moreover, the matrix A introduces an inter-dimensional higher-order interaction that partly accounts for the emergence of 2 d system modes in a d-dimensional population, where each dimension can either be synchronized or desynchronized, represented by a set of almost binary order parameters. The binary system modes capture characteristic swarm behaviors such as fish milling or polarized schooling. Additionally, our findings provides a theoretical analogy to cerebral activity, where the resting state and the activated state coexist unihemispherically. It also suggests a new possibility for information storage in oscillatory neural networks.

Designing a Fully Current-Controlled Memristors-Based Oscillator

One of the promising applications of locally-active memristors (LAMs) is to construct oscillators for oscillatory neural networks. By using two current-controlled (CC) LAMs, a fully CC LAM-based oscillator is designed in this paper. The oscillator principle originates from the small-signal inductive and capacitive impedance characteristics of two different CC LAMs, and thus extra reactance element is not required in the circuit. Based on bifurcation theory and small-signal analysis method, conditions of the equilibrium point instability are quantitatively derived. Theoretical analysis indicates that the circuit oscillation is dependent on three critical parameters. Then, according to the conditions of the equilibrium point instability, parameters design methods of the two LAMs are proposed, including the static and dynamic parameters. A simple NbOx CC LAM model is taken as an example to conduct detailed simulation analysis. The simulation results verify the feasibility of the proposed circuit and analysis methods. Finally, the effects of the LAM model parameters on the oscillator performance are investigated, which is helpful for optimal design of the oscillator.

Vertically Stackable Ovonic Threshold Switch Oscillator Using Atomic Layer Deposited Ge0.6Se0.4 Film for High-Density Artificial Neural Networks.

Nanodevice oscillators (nano-oscillators) have received considerable attention to implement in neuromorphic computing as hardware because they can significantly improve the device integration density and energy efficiency compared to complementary metal oxide semiconductor circuit-based oscillators. This work demonstrates vertically stackable nano-oscillators using an ovonic threshold switch (OTS) for high-density neuromorphic hardware. A vertically stackable Ge0.6Se0.4 OTS-oscillator (VOTS-OSC) is fabricated with a vertical crossbar array structure by growing Ge0.6Se0.4 film conformally on a contact hole structure using atomic layer deposition. The VOTS-OSC can be vertically integrated onto peripheral circuits without causing thermal damage because the fabrication temperature is <400 °C. The fabricated device exhibits oscillation characteristics, which can serve as leaky integrate-and-fire neurons in spiking neural networks (SNNs) and coupled oscillators in oscillatory neural networks (ONNs). For practical applications, pattern recognition and vertex coloring are demonstrated with SNNs and ONNs, respectively, using semiempirical simulations. This structure increases the oscillator integration density significantly, enabling complex tasks with a large number of oscillators. Moreover, it can enhance the computational speed of neural networks due to its rapid switching speed.

Design of oscillatory neural networks by machine learning.

We demonstrate the utility of machine learning algorithms for the design of oscillatory neural networks (ONNs). After constructing a circuit model of the oscillators in a machine-learning-enabled simulator and performing Backpropagation through time (BPTT) for determining the coupling resistances between the ring oscillators, we demonstrate the design of associative memories and multi-layered ONN classifiers. The machine-learning-designed ONNs show superior performance compared to other design methods (such as Hebbian learning), and they also enable significant simplifications in the circuit topology. We also demonstrate the design of multi-layered ONNs that show superior performance compared to single-layer ones. We argue that machine learning can be a valuable tool to unlock the true computing potential of ONNs hardware.

A Nanoscale Bistable Resistor for an Oscillatory Neural Network.

Coupled oscillators construct an oscillatory neural network (ONN) by mimicking the interactions among neurons in the human brain. This work demonstrates a fully CMOS-based oscillator consisting of a bistable resistor (biristor), which shares a structure identical with that of a metal-oxide-semiconductor field-effect transistor, except for the use of a gate electrode. The biristor-based oscillator (birillator) generates oscillating voltage signals in the form of spikes due to a single transistor latch phenomenon. When two birillators are connected with a coupling capacitor, they become synchronized with a phase difference of 180°. These coupled oscillation characteristics are experimentally investigated for an ONN. As practical applications of the ONN with coupled birillators, edge detection and vertex coloring are conducted by encoding information into phase differences between them. The proposed fully CMOS-based birillators are advantageous for low power consumption, high CMOS compatibility, and a compact footprint area.

Predictor-Based Neural Attitude Control of A Quadrotor With Disturbances

An attitude control issue is concerned for a quadrotor with external disturbances in this paper. For unknown system dynamics, predictor-based neural networks (NNs) are introduced, where prediction errors, angular velocities, are constructed, instead of tracking errors, for updating NNs' weights. This replacement reduces the occurrence of high-frequency oscillations in NNs' approximation. With this improved NNs, a predictorbased NN disturbance observer is then developed for compensation for external disturbances and NNs' approximation errors, and a normalization learning technique is employed for reduction of the number of learning parameters. A predictor-based neural attitude control strategy is proposed for a quadrotor with external disturbances. Furthermore, measurement noise are taken into account in our predictor-based neural attitude control strategy. The Lyapunov-based stability analysis shows that all closed-loop signals in the designed attitude system are semiglobally bounded. A numerical simulation and a hardware-in-loop experiment as well as outdoor flight verify the effectiveness of the proposed anti-disturbance attitude control strategy.

Experimental demonstration of coupled differential oscillator networks for versatile applications.

Oscillatory neural networks (ONNs) exhibit a high potential for energy-efficient computing. In ONNs, neurons are implemented with oscillators and synapses with resistive and/or capacitive coupling between pairs of oscillators. Computing is carried out on the basis of the rich, complex, non-linear synchronization dynamics of a system of coupled oscillators. The exploited synchronization phenomena in ONNs are an example of fully parallel collective computing. A fast system's convergence to stable states, which correspond to the desired processed information, enables an energy-efficient solution if small area and low-power oscillators are used, specifically when they are built on the basis of the hysteresis exhibited by phase-transition materials such as VO2. In recent years, there have been numerous studies on ONNs using VO2. Most of them report simulation results. Although in some cases experimental results are also shown, they do not implement the design techniques that other works on electrical simulations report that allow to improve the behavior of the ONNs. Experimental validation of these approaches is necessary. Therefore, in this study, we describe an ONN realized in a commercial CMOS technology in which the oscillators are built using a circuit that we have developed to emulate the VO2 device. The purpose is to be able to study in-depth the synchronization dynamics of relaxation oscillators similar to those that can be performed with VO2 devices. The fabricated circuit is very flexible. It allows programming the synapses to implement different ONNs, calibrating the frequency of the oscillators, or controlling their initialization. It uses differential oscillators and resistive synapses, equivalent to the use of memristors. In this article, the designed and fabricated circuits are described in detail, and experimental results are shown. Specifically, its satisfactory operation as an associative memory is demonstrated. The experiments carried out allow us to conclude that the ONN must be operated according to the type of computational task to be solved, and guidelines are extracted in this regard.

Learning algorithms for oscillatory neural networks as associative memory for pattern recognition

Alternative paradigms to the von Neumann computing scheme are currently arousing huge interest. Oscillatory neural networks (ONNs) using emerging phase-change materials like VO2 constitute an energy-efficient, massively parallel, brain-inspired, in-memory computing approach. The encoding of information in the phase pattern of frequency-locked, weakly coupled oscillators makes it possible to exploit their rich non-linear dynamics and their synchronization phenomena for computing. A single fully connected ONN layer can implement an auto-associative memory comparable to that of a Hopfield network, hence Hebbian learning rule is the most widely adopted method for configuring ONNs for such applications, despite its well-known limitations. An extensive amount of literature is available about learning in Hopfield networks, with information regarding many different learning algorithms that perform better than the Hebbian rule. However, not all of these algorithms are useful for ONN training due to the constraints imposed by their physical implementation. This paper evaluates different learning methods with respect to their suitability for ONNs. It proposes a new approach, which is compared against previous works. The proposed method has been shown to produce competitive results in terms of pattern recognition accuracy with reduced precision in synaptic weights, and to be suitable for online learning.

A phenomenological model of whole brain dynamics using a network of neural oscillators with power-coupling

We present a general, trainable oscillatory neural network as a large-scale model of brain dynamics. The model has a cascade of two stages - an oscillatory stage and a complex-valued feedforward stage - for modelling the relationship between structural connectivity and functional connectivity from neuroimaging data under resting brain conditions. Earlier works of large-scale brain dynamics that used Hopf oscillators used linear coupling of oscillators. A distinctive feature of the proposed model employs a novel form of coupling known as power coupling. Oscillatory networks based on power coupling can accurately model arbitrary multi-dimensional signals. Training the lateral connections in the oscillator layer is done by a modified form of Hebbian learning, whereas a variation of the complex backpropagation algorithm does training in the second stage. The proposed model can not only model the empirical functional connectivity with remarkable accuracy (correlation coefficient between simulated and empirical functional connectivity- 0.99) but also identify default mode network regions. In addition, we also inspected how structural loss in the brain can cause significant aberration in simulated functional connectivity and functional connectivity dynamics; and how it can be restored with optimized model parameters by an in silico perturbational study.

How fast can vanadium dioxide neuron-mimicking devices oscillate? Physical mechanisms limiting the frequency of vanadium dioxide oscillators

The frequency of vanadium dioxide (VO2) oscillators is a fundamental figure of merit for the realization of neuromorphic circuits called oscillatory neural networks (ONNs) since the high frequency of oscillators ensures low-power consuming, real-time computing ONNs. In this study, we perform electrothermal 3D technology computer-aided design (TCAD) simulations of a VO2 relaxation oscillator. We find that there exists an upper limit to its operating frequency, where such a limit is not predicted from a purely circuital model of the VO2 oscillator. We investigate the intrinsic physical mechanisms that give rise to this upper limit. Our TCAD simulations show that it arises a dependence on the frequency of the points of the curve current versus voltage across the VO2 device corresponding to the insulator-to-metal transition (IMT) and metal-to-insulator transition (MIT) during oscillation, below some threshold values of . This implies that the condition for the self-oscillatory regime may be satisfied by a given load-line in the low-frequency range but no longer at higher frequencies, with consequent suppression of oscillations. We note that this variation of the IMT/MIT points below some threshold values of is due to a combination of different factors: intermediate resistive states achievable by VO2 channel and the interplay between frequency and heat transfer rate. Although the upper limit on the frequency that we extract is linked to the specific VO2 device we simulate, our findings apply qualitatively to any VO2 oscillator. Overall, our study elucidates the link between electrical and thermal behavior in VO2 devices that sets a constraint on the upper values of the operating frequency of any VO2 oscillator.

A mixed-signal oscillatory neural network for scalable analog computations in phase domain

Digital electronics based on von Neumann’s architecture is reaching its limits to solve large-scale problems essentially due to the memory fetching. Instead, recent efforts to bring the memory near the computation have enabled highly parallel computations at low energy costs. Oscillatory neural network (ONN) is one example of in-memory analog computing paradigm consisting of coupled oscillating neurons. When implemented in hardware, ONNs naturally perform gradient descent of an energy landscape which makes them particularly suited for solving optimization problems. Although the ONN computational capability and its link with the Ising model are known for decades, implementing a large-scale ONN remains difficult. Beyond the oscillators’ variations, there are still design challenges such as having compact, programmable synapses and a modular architecture for solving large problem instances. In this paper, we propose a mixed-signal architecture named Saturated Kuramoto ONN (SKONN) that leverages both analog and digital domains for efficient ONN hardware implementation. SKONN computes in the analog phase domain while propagating the information digitally to facilitate scaling up the ONN size. SKONN’s separation between computation and propagation enhances the robustness and enables a feed-forward phase propagation that is showcased for the first time. Moreover, the SKONN architecture leads to unique binarizing dynamics that are particularly suitable for solving NP-hard combinatorial optimization problems such as finding the weighted Max-cut of a graph. We find that SKONN’s accuracy is as good as the Goemans–Williamson 0.878-approximation algorithm for Max-cut; whereas SKONN’s computation time only grows logarithmically. We report on Weighted Max-cut experiments using a 9-neuron SKONN proof-of-concept on a printed circuit board (PCB). Finally, we present a low-power 16-neuron SKONN integrated circuit and illustrate SKONN’s feed-forward ability while computing the XOR function.

Superconducting-Oscillatory Neural Network With Pixel Error Detection for Image Recognition

Brain-inspired oscillatory neural networks (ONNs) utilize coupled oscillators to emulate biological neuronal dynamics. ONNs are naturally suitable for image/pattern recognition. Many hardware implementations of ONNs have been explored recently by using analog or digital CMOS systems, which can be limited by Moore's law. In this work, we used one of the advanced superconducting technologies for coupled oscillator networks to further improve the energy efficiency and processing speed. Inductively coupled ring oscillators using rapid single flux quantum (RSFQ) technology were designed and simulated. These Josephson junction (JJ) based oscillators can operate as fast as tens of GHz but only consume a energy of several aJ per operation. Excluding cryo-cooling factors, the estimated power consumption of our RSFQ oscillator is hundreds of nW at a frequency of 12.5 GHz, which is multiple times smaller than a typical CMOS ring oscillator. Furthermore, a system for pattern recognition with error detection was designed based on these oscillators. Synchronization dynamics in a pair of coupled oscillators were used to identify whether a test pixel matches a reference pixel. A network comprised of 16 pairs of oscillators were wired together, along with synchronization detectors and a fluxon integrator to demonstrate pattern recognition function. This work was performed using standard JJ models and demonstrated through WRspice and JoSIM software simulations. The circuits and systems proposed in this work aim to provide insights into the use of superconducting technology for implementing ONNs and may serve as basic building blocks for the design of more complex oscillatory computational networks in image processing and beyond. Single oscillator modeling and systematic network function exploration will be performed in future work.

Development of a hybrid neural network model for mine detection by using ultrawideband radar data

The object of the study is the architecture of a hybrid neural network for mine recognition using ultra-wideband radar data. The work solves the problem of filtering reflected signals with interference and recognizing mines detected by ultra-wideband (UWB) radar. A hybrid neural network model in combination with the Adam learning algorithm is proposed. Filtering of reflected signals from mines is carried out using an MLP (multilayer perceptron) filter, which selects low-amplitude parts of signals that carry information about a hidden mine from the entire reflected signal. Mine recognition is carried out by a Hilbert block and an oscillatory neural network, which are included in the structure of a hybrid neural network. The peculiarity of the obtained results, which allowed to solve the investigated problem, is the transformation of the signal frequency by the Hilbert block and the recognition of mines by the oscillatory neural network in the resonant mode. The three-layer MLP filter effectively filters out the unwanted component in the total signal reflected from the subsurface object, as the MSE (Mean Squared Error) of the MLP filter is 1·10-5. If the frequency of the Hilbert signal is equal to the natural frequency of oscillations of neurons then the recognition of signals with a small amplitude from subsurface objects is carried out by an oscillatory neural network based on the resonant amplitude, which is indicated by a small value of cross-entropy. The proposed model of a hybrid neural network provides amplification of useful signals due to resonance and has higher performance compared to existing models of artificial neural networks. The practical significance of the obtained results lies in their application in the field of automated neural network technologies for detection and recognition of subsurface objects of various nature based on reflected radar signals with an amplitude at the noise level

Oscillatory neural network learning for pattern recognition: an on-chip learning perspective and implementation.

In the human brain, learning is continuous, while currently in AI, learning algorithms are pre-trained, making the model non-evolutive and predetermined. However, even in AI models, environment and input data change over time. Thus, there is a need to study continual learning algorithms. In particular, there is a need to investigate how to implement such continual learning algorithms on-chip. In this work, we focus on Oscillatory Neural Networks (ONNs), a neuromorphic computing paradigm performing auto-associative memory tasks, like Hopfield Neural Networks (HNNs). We study the adaptability of the HNN unsupervised learning rules to on-chip learning with ONN. In addition, we propose a first solution to implement unsupervised on-chip learning using a digital ONN design. We show that the architecture enables efficient ONN on-chip learning with Hebbian and Storkey learning rules in hundreds of microseconds for networks with up to 35 fully-connected digital oscillators.

Training energy-based single-layer Hopfield and oscillatory networks with unsupervised and supervised algorithms for image classification

This paper investigates how to solve image classification with Hopfield neural networks (HNNs) and oscillatory neural networks (ONNs). This is a first attempt to apply ONNs for image classification. State-of-the-art image classification networks are multi-layer models trained with supervised gradient back-propagation, which provide high-fidelity results but require high energy consumption and computational resources to be implemented. On the contrary, HNN and ONN networks are single-layer, requiring less computational resources, however, they necessitate some adaptation as they are not directly applicable for image classification. ONN is a novel brain-inspired computing paradigm that performs low-power computation and is attractive for edge artificial intelligence applications, such as image classification. In this paper, we perform image classification with HNN and ONN by exploiting their auto-associative memory (AAM) properties. We evaluate precision of HNN and ONN trained with state-of-the-art unsupervised learning algorithms. Additionally, we adapt the supervised equilibrium propagation (EP) algorithm to single-layer AAM architectures, proposing the AAM-EP. We test and validate HNN and ONN classification on images of handwritten digits using a simplified MNIST set. We find that using unsupervised learning, HNN reaches 65.2%, and ONN 59.1% precision. Moreover, we show that AAM-EP can increase HNN and ONN precision up to 67.04% for HNN and 62.6% for ONN. While intrinsically HNN and ONN are not meant for classification tasks, to the best of our knowledge, these are the best-reported precisions of HNN and ONN performing classification of images of handwritten digits.

Enhancement of temperature-modulated NbO2-based relaxation oscillator via interfacial and bulk treatments

This work demonstrates oscillation frequency modulation in a NbO2-based relaxation oscillator device, in which the oscillation frequency increases with operating temperature and source voltage, and decreases with load resistance. An annealing-induced oxygen diffusion at 373 K was carried out to optimize the stoichiometry of the bulk NbO2 to achieve consistent oscillation frequency shift with device temperature. The device exhibits stable self-sustained oscillation in which the frequency can be modulated between 2 and 33 MHz, and a wider operating voltage range can be obtained. An additional surface treatment step was employed during fabrication to reduce the surface roughness of the bottom electrode and to remove surface contaminants that affect the interfacial properties of the device. The device frequency tunability coupled with high oscillating frequency and high endurance capability of more than 1.5 × 108 cycles indicates that the Pt/NbO2/Pt device is particularly suitable for applications in an oscillatory neural network.

Simulation and implementation of two-layer oscillatory neural networks for image edge detection: bidirectional and feedforward architectures

The growing number of edge devices in everyday life generates a considerable amount of data that current AI algorithms, like artificial neural networks, cannot handle inside edge devices with limited bandwidth, memory, and energy available. Neuromorphic computing, with low-power oscillatory neural networks (ONNs), is an alternative and attractive solution to solve complex problems at the edge. However, ONN is currently limited with its fully-connected recurrent architecture to solve auto-associative memory problems. In this work, we use an alternative two-layer bidirectional ONN architecture. We introduce a two-layer feedforward ONN architecture to perform image edge detection, using the ONN to replace convolutional filters to scan the image. Using an HNN Matlab emulator and digital ONN design simulations, we report efficient image edge detection from both architectures using various size filters (3 × 3, 5 × 5, and 7 × 7) on black and white images. In contrast, the feedforward architectures can also perform image edge detection on gray scale images. With the digital ONN design, we also assess latency performances and obtain that the bidirectional architecture with a 3 × 3 filter size can perform image edge detection in real-time (camera flow from 25 to 30 images per second) on images with up to 128 × 128 pixels while the feedforward architecture with same 3 × 3 filter size can deal with 170 × 170 pixels, due to its faster computation.

Effect of Device Mismatches in Differential Oscillatory Neural Networks

Analog implementation of Oscillatory Neural Networks (ONNs) has the potential to implement fast and ultra-low-power computing capabilities. One of the drawbacks of analog implementation is component mismatches which cause desynchronization and instability in ONNs. Emerging devices like memristors and VO2are particularly prone to variations. In this paper, we study the effect of component mismatches on the performance of differential ONNs (DONNs). Mismatches were considered in two main blocks: differential oscillatory neurons and synaptic circuits. To measure DONN tolerance to mismatches in each block, performance was evaluated with mismatches being present separately in each block. Memristor-bridge circuits with four memristors were used as the synaptic circuits. The differential oscillatory neurons were based on VO2-devices. The simulation results showed that DONN performance was more vulnerable to mismatches in the components of the differential oscillatory neurons than to mismatches in the synaptic circuits. DONNs were found to tolerate up to 20% mismatches in the memristance of the synaptic circuits. However, mismatches in the differential oscillatory neurons resulted in non-uniformity of the natural frequencies, causing desynchronization and instability. Simulations showed that 0.5% relative standard deviation (RSD) in natural frequencies can reduce DONN performance dramatically. In addition, sensitivity analyses showed that the high threshold voltage of VO2-devices is the most sensitive parameter for frequency non-uniformity and desynchronization.