Liquid State Machine Research Articles

Accurate identification of salt domes using deep learning techniques: Transformers, generative artificial intelligence and liquid state machines

AbstractAcross various global regions abundant in oil and natural gas reserves, the presence of substantial sub‐surface salt deposits holds significant relevance. Accurate identification of salt domes becomes crucial for enterprises engaged in oil and gas exploration. Our research introduces a precise method for the automatic detection of salt domes, leveraging advanced deep learning architectures such as U‐net, transformers, artificial intelligence generative models and liquid state machines. In comparison with state‐of‐the‐art techniques, our model demonstrates superior performance, achieving a stable and validated intersection over the union metric, indicating high accuracy and robustness. Furthermore, the Dice similarity coefficient attaining underscores the model's proficiency in closely aligning with ground truth across diverse scenarios. This evaluation, conducted on 1000 seismic images, reveals that our proposed architecture is not only comparable but often surpasses existing segmentation models in effectiveness and reliability.

A Brain-Inspired Decision-Making method for upper limb exoskeleton based on Multi-Brain-Region structure and multimodal information fusion

For the challenges faced by upper limb exoskeleton in meeting human motion intention, this article proposes a novel brain-inspired decision-making model based on multi-brain-region information transmission mechanism and multimodal information fusion method to provide accurate prediction of motion trajectory. Firstly, a multimodal information perception and fusion method is proposed. Based on surface electromyographic (sEMG) signals and angle signals collected by sensors. Then a random forest algorithm is applied to screen the key features of sEMG signal for trajectory prediction. A multimodal information hierarchical fusion method based on dual-period mechanism is designed to solve the problem of fusion defect of information with different frequency. Finally, a multi-brain-region decision-making model based on hybrid hierarchical liquid state machine (HHLSM) is established to predict motion trajectory for the exoskeleton to meet human motion intention. Experiments show that the established model has a high computational efficiency and improves the effectiveness of exoskeleton assistance. The brain-inspired decision-making model based on HHLSM established in this paper is of great significance for further research on brain-inspired control and improving the human-robot interaction effect of upper limb exoskeleton.

Ultralow Power In-Sensor Neuronal Computing with Oscillatory Retinal Neurons for Frequency-Multiplexed, Parallel Machine Vision.

In-sensor and near-sensor computing architectures enable multiply accumulate operations to be carried out directly at the point of sensing. In-sensor architectures offer dramatic power and speed improvements over traditional von Neumann architectures by eliminating multiple analog-to-digital conversions, data storage, and data movement operations. Current in-sensor processing approaches rely on tunable sensors or additional weighting elements to perform linear functions such as multiply accumulate operations as the sensor acquires data. This work implements in-sensor computing with an oscillatory retinal neuron device that converts incident optical signals into voltage oscillations. A computing scheme is introduced based on the frequency shift of coupled oscillators that enables parallel, frequency multiplexed, nonlinear operations on the inputs. An experimentally implemented 3 × 3 focal plane array of coupled neurons shows that functions approximating edge detection, thresholding, and segmentation occur in parallel. An example of inference on handwritten digits from the MNIST database is also experimentally demonstrated with a 3 × 3 array of coupled neurons feeding into a single hidden layer neural network, approximating a liquid-state machine. Finally, the equivalent energy consumption to carry out image processing operations, including peripherals such as the Fourier transform circuits, is projected to be <20 fJ/OP, possibly reaching as low as 15 aJ/OP.

Memristor Based Liquid State Machine With Method for In-Situ Training

Memristor Based Liquid State Machine With Method for In-Situ Training

Cerebellum-Inspired Model-Free Tracking Control and Visual Servoing of a Rigid-Flexible Hybrid Robotic Endoscope With RCM Constraints

This article designs a model-free control scheme based on a cerebellum-inspired network model for tracking control and visual servoing of a surgical rigid-flexible hybrid robotic endoscope. The hybrid robot is composed of a rigid robot manipulator and a flexible endoscope as the end-effector. Inspired by the cerebellum associated with human motion control, we proposed a cerebellum-inspired model-free control scheme for the visual servoing and tracking control of the robotic endoscope with RCM (remote center of motion) constraints, which does not need to know the kinematic model of the robot. This scheme combines liquid state machines (LSM) and zeroing neural network (ZNN), where the LSM is able to generate control signals and the ZNN is used as the training signal for the LSM. Simulations and physical experiments demonstrate that the proposed cerebellum-inspired control scheme is effective to handle the tracking task under RCM constraints as well as the visual servoing task.

Neural Synchrony-Based State Representation in Liquid State Machines, an Exploratory Study

Neural Synchrony-Based State Representation in Liquid State Machines, an Exploratory Study

Characterization of the neuronal and network dynamics of liquid state machines

Reservoir computing (RC) is a relatively new machine-learning framework that uses an abstract neural network model, called reservoir. The reservoir forms a complex system with high dimensionality, nonlinearity, and intrinsic memory effect due to recurrent connections among individual neurons. RC manifests a best-in-class performance in processing information generated by complex dynamical systems, yet little is known about its microscopic/macroscopic dynamics underlying the computational capability. Here, we characterize the neuronal and network dynamics of liquid state machines (LSMs) using numerical simulations and Modified National Institute of Standards and Technology (MNIST) database classification tasks. The computational performance of LSMs largely depends on a dynamic range of neuronal avalanches whereby the avalanche patterns are determined by the neuron and network models. A larger dynamic range leads to higher performance—the MNIST classification accuracy is highest when the avalanche sizes follow a slowly decaying power-law distribution with an exponent of ∼1.5, followed by the power-law statistics with a larger exponent and the mixture of power-law/log-normal distributions. Network-theoretic analysis suggests that the formation of large functional clusters and the promotion of dynamic transitions between large and small clusters may contribute to the scale-invariant nature. This study provides new insight into our understanding of the computational principles of RC concerning the actions of the individual neurons and the system-level collective behavior.

Adaptive structure evolution and biologically plausible synaptic plasticity for recurrent spiking neural networks

The architecture design and multi-scale learning principles of the human brain that evolved over hundreds of millions of years are crucial to realizing human-like intelligence. Spiking neural network based Liquid State Machine (LSM) serves as a suitable architecture to study brain-inspired intelligence because of its brain-inspired structure and the potential for integrating multiple biological principles. Existing researches on LSM focus on different certain perspectives, including high-dimensional encoding or optimization of the liquid layer, network architecture search, and application to hardware devices. There is still a lack of in-depth inspiration from the learning and structural evolution mechanism of the brain. Considering these limitations, this paper presents a novel LSM learning model that integrates adaptive structural evolution and multi-scale biological learning rules. For structural evolution, an adaptive evolvable LSM model is developed to optimize the neural architecture design of liquid layer with separation property. For brain-inspired learning of LSM, we propose a dopamine-modulated Bienenstock-Cooper-Munros (DA-BCM) method that incorporates global long-term dopamine regulation and local trace-based BCM synaptic plasticity. Comparative experimental results on different decision-making tasks show that introducing structural evolution of the liquid layer, and the DA-BCM regulation of the liquid layer and the readout layer could improve the decision-making ability of LSM and flexibly adapt to rule reversal. This work is committed to exploring how evolution can help to design more appropriate network architectures and how multi-scale neuroplasticity principles coordinated to enable the optimization and learning of LSMs for relatively complex decision-making tasks.

Surrogate-Assisted Cooperative Co-evolutionary Reservoir Architecture Search for Liquid State Machines

Liquid state machines (LSMs) are biologically more plausible than feedforward spiking neural networks for brain-inspired computing and neuromorphic engineering. However, optimizing and training complex recurrent network architectures in the reservoir of LSMs remains challenging. Most existing algorithms aim to adjust the synaptic strength only, without fundamentally modifying the reservoir architecture of LSMs. Recently, it has become popular to simultaneously optimize the architecture and parameters of the reservoir in LSMs. However, most existing architecture representation schemes are too restricted to discover more powerful architectures of LSMs. To address the above issue, this paper proposes a generative liquid state machine, whose reservoir architecture is evolved using a cooperative co-evolutionary algorithm whose weights are tuned by synaptic plasticity rules. To reduce the computation time for evolving the reservoir, random forest is adopted to assist the cooperative co-evolutionary algorithm, together with a data parallelism strategy. The proposed algorithm is assessed on three sequence classification benchmarks and our experimental results show that the proposed algorithm outperforms the state-of-the-art on the benchmark problems. Meanwhile, our analysis shows that the data parallelism strategy is effective in speeding up the evaluation process.

Indoor Scene Recognition: An Attention-Based Approach Using Feature Selection-Based Transfer Learning and Deep Liquid State Machine

Scene understanding is one of the most challenging areas of research in the fields of robotics and computer vision. Recognising indoor scenes is one of the research applications in the category of scene understanding that has gained attention in recent years. Recent developments in deep learning and transfer learning approaches have attracted huge attention in addressing this challenging area. In our work, we have proposed a fine-tuned deep transfer learning approach using DenseNet201 for feature extraction and a deep Liquid State Machine model as the classifier in order to develop a model for recognising and understanding indoor scenes. We have included fuzzy colour stacking techniques, colour-based segmentation, and an adaptive World Cup optimisation algorithm to improve the performance of our deep model. Our proposed model would dedicatedly assist the visually impaired and blind to navigate in the indoor environment and completely integrate into their day-to-day activities. Our proposed work was implemented on the NYU depth dataset and attained an accuracy of 96% for classifying the indoor scenes.

Brain-inspired multimodal hybrid neural network for robot place recognition.

Place recognition is an essential spatial intelligence capability for robots to understand and navigate the world. However, recognizing places in natural environments remains a challenging task for robots because of resource limitations and changing environments. In contrast, humans and animals can robustly and efficiently recognize hundreds of thousands of places in different conditions. Here, we report a brain-inspired general place recognition system, dubbed NeuroGPR, that enables robots to recognize places by mimicking the neural mechanism of multimodal sensing, encoding, and computing through a continuum of space and time. Our system consists of a multimodal hybrid neural network (MHNN) that encodes and integrates multimodal cues from both conventional and neuromorphic sensors. Specifically, to encode different sensory cues, we built various neural networks of spatial view cells, place cells, head direction cells, and time cells. To integrate these cues, we designed a multiscale liquid state machine that can process and fuse multimodal information effectively and asynchronously using diverse neuronal dynamics and bioinspired inhibitory circuits. We deployed the MHNN on Tianjic, a hybrid neuromorphic chip, and integrated it into a quadruped robot. Our results show that NeuroGPR achieves better performance compared with conventional and existing biologically inspired approaches, exhibiting robustness to diverse environmental uncertainty, including perceptual aliasing, motion blur, light, or weather changes. Running NeuroGPR as an overall multi-neural network workload on Tianjic showcases its advantages with 10.5 times lower latency and 43.6% lower power consumption than the commonly used mobile robot processor Jetson Xavier NX.

PLSM: A Parallelized Liquid State Machine for Unintentional Action Detection

Reservoir Computing (RC) offers a viable option to deploy AI algorithms on low-end embedded system platforms. Liquid State Machine (LSM) is a bio-inspired RC model that mimics the cortical microcircuits and uses spiking neural networks (SNN) that can be directly realized on neuromorphic hardware. In this paper, we present a novel Parallelized LSM (PLSM) architecture that incorporates spatio-temporal read-out layer and semantic constraints on model output. To the best of our knowledge, such a formulation has been done for the first time in literature, and it offers a computationally lighter alternative to traditional deep-learning models. Additionally, we also present a comprehensive algorithm for the implementation of parallelizable SNNs and LSMs that are GPU-compatible. We implement the PLSM model to classify unintentional/accidental video clips using the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Oops</i> dataset. From the experimental results on detecting unintentional action in a video, it can be observed that our proposed model outperforms a self-supervised model and a fully supervised traditional deep learning model. All the implemented codes can be found in our repository <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/sadimanna/Parallelized_LSM_for_Unintentional_Action_Recognition</uri> .

Extended liquid state machines for speech recognition.

A liquid state machine (LSM) is a biologically plausible model of a cortical microcircuit. It exists of a random, sparse reservoir of recurrently connected spiking neurons with fixed synapses and a trainable readout layer. The LSM exhibits low training complexity and enables backpropagation-free learning in a powerful, yet simple computing paradigm. In this work, the liquid state machine is enhanced by a set of bio-inspired extensions to create the extended liquid state machine (ELSM), which is evaluated on a set of speech data sets. Firstly, we ensure excitatory/inhibitory (E/I) balance to enable the LSM to operate in edge-of-chaos regime. Secondly, spike-frequency adaptation (SFA) is introduced in the LSM to improve the memory capabilities. Lastly, neuronal heterogeneity, by means of a differentiation in time constants, is introduced to extract a richer dynamical LSM response. By including E/I balance, SFA, and neuronal heterogeneity, we show that the ELSM consistently improves upon the LSM while retaining the benefits of the straightforward LSM structure and training procedure. The proposed extensions led up to an 5.2% increase in accuracy while decreasing the number of spikes in the ELSM up to 20.2% on benchmark speech data sets. On some benchmarks, the ELSM can even attain similar performances as the current state-of-the-art in spiking neural networks. Furthermore, we illustrate that the ELSM input-liquid and recurrent synaptic weights can be reduced to 4-bit resolution without any significant loss in classification performance. We thus show that the ELSM is a powerful, biologically plausible and hardware-friendly spiking neural network model that can attain near state-of-the-art accuracy on speech recognition benchmarks for spiking neural networks.

Lipreading Using Liquid State Machine with STDP-Tuning

Lipreading refers to the task of decoding the text content of a speaker based on visual information about the movement of the speaker’s lips. With the development of deep learning in recent years, lipreading has attracted extensive research. However, the deep learning method requires a lot of computing resources, which is not conducive to the migration of the system to edge devices. Inspired by the work of Spiking Neural Networks (SNNs) in recognizing human actions and gestures, we propose a lipreading system based on SNNs. Specifically, we construct the front-end feature extractor of the system using Liquid State Machine (LSM). On the other hand, a heuristic algorithm is used to select appropriate parameters for the classifier in the backend. On small-scale lipreading datasets, our recognition accuracy achieves good results. We claim that our network performs better in terms of accuracy and ratio of learned parameters compared to other networks, and has superior advantages in terms of network complexity and training cost. On the AVLetters dataset, our model achieves a 5% improvement in accuracy over traditional methods and a 90% reduction in parameters over the state-of-the-art.

Small-world spiking neural network with anti-interference ability based on speech recognition under interference

The external interference can hamper the normal function of neuromorphic hardware under complex noise environment. Therefore, the study of brain-like models with anti-interference ability is a crucial issue. A bio-brain has the advantage of self-adaptability. The existing brain-like models are lack of bio-interpretability. Drawing from the advantage of bio-brain, the purpose of this paper is to investigate the anti-interference ability of brain-like models with bio-interpretability. In this study, we construct a spiking neural network with a small-world topology (SWSNN) as a brain-like model, in which Izhikevich neuron models and synaptic plasticity models with time-delay are used to represent the nodes and edges of the network, respectively. Then, the anti-interference ability of our SWSNN against different external noise is investigated, and its mechanism is explored. Further, by taking the speech recognition task as the case study, we verify the anti-interference ability of the SWSNN. Our simulation results indicate that: (i) The anti-interference ability of SWSNN is better than that of SNNs with other topologies. (ii) The discussion of neural information processing of the SWSNN under interference implies that the dynamic regulation of synaptic plasticity is an intrinsic factor of the anti-interference, and the topology is a factor that affects the anti-interference at the level of performance. (iii) Taking the speech recognition task as a case study, the SWSNN under different external noises still have almost the same high speech recognition accuracy, compared with the SWSNN without interference, and SWSNN yields better anti-interference ability than SNNs with other topologies in terms of the speech recognition accuracy. In addition, the anti-interference ability of our SWSNN framework is superior to that of existing liquid state machine (LSM) framework. Our simulation results lay a preliminary foundation for the neuromorphic hardware with robustness under complex noise environment.

Spike-train level supervised learning algorithm based on bidirectional modification for liquid state machines

Spike-train level supervised learning algorithm based on bidirectional modification for liquid state machines

Optimizing the Neural Structure and Hyperparameters of Liquid State Machines Based on Evolutionary Membrane Algorithm

As one of the important artificial intelligence fields, brain-like computing attempts to give machines a higher intelligence level by studying and simulating the cognitive principles of the human brain. A spiking neural network (SNN) is one of the research directions of brain-like computing, characterized by better biogenesis and stronger computing power than the traditional neural network. A liquid state machine (LSM) is a neural computing model with a recurrent network structure based on SNN. In this paper, a learning algorithm based on an evolutionary membrane algorithm is proposed to optimize the neural structure and hyperparameters of an LSM. First, the object of the proposed algorithm is designed according to the neural structure and hyperparameters of the LSM. Second, the reaction rules of the proposed algorithm are employed to discover the best neural structure and hyperparameters of the LSM. Third, the membrane structure is that the skin membrane contains several elementary membranes to speed up the search of the proposed algorithm. In the simulation experiment, effectiveness verification is carried out on the MNIST and KTH datasets. In terms of the MNIST datasets, the best test results of the proposed algorithm with 500, 1000 and 2000 spiking neurons are 86.8%, 90.6% and 90.8%, respectively. The best test results of the proposed algorithm on KTH with 500, 1000 and 2000 spiking neurons are 82.9%, 85.3% and 86.3%, respectively. The simulation results show that the proposed algorithm has a more competitive advantage than other experimental algorithms.

LSMCore: A 69k-Synapse/mm2 Single-Core Digital Neuromorphic Processor for Liquid State Machine

Neuromorphic processors have gained momentum recently due to their high energy efficiency in artificial intelligence applications compared to DNN accelerators. Most neuromorphic processors are executing SNNs (Spiking Neural Networks). Liquid State Machine (LSM), as the spiking version of reservoir computing, shows advantages and great potential in image classification, speech recognition, language translation, etc.. Comparing with other SNN models, LSM has the characteristics of easy to train and low resource utilization, which is suitable for low-power and resource-constrained edge computing scenarios. In this paper, we propose a novel design of a neuromorphic processor, LSMCore, aiming at LSM acceleration. LSMCore supports both training and inference of LSM. It consists of 256 input neurons, 1024 liquid neurons, and 1.31M synapses. Besides, multiple optimization techniques, including weight quantization for reducing storage, zero-skipping for decreasing dynamic sparsity, and mini-batch training are adopted in this processor. The experimental results show that the frequency of LSMCore achieves 400 MHz, the power is 4.9W and the area is 18.49 mm² with a 40nm library. Comparing with the baseline, LSMCore achieves up to <inline-formula> <tex-math notation="LaTeX">$80.7\times $ </tex-math></inline-formula>(<inline-formula> <tex-math notation="LaTeX">$49.6\times $ </tex-math></inline-formula>), <inline-formula> <tex-math notation="LaTeX">$91.3\times $ </tex-math></inline-formula>(<inline-formula> <tex-math notation="LaTeX">$56.3\times $ </tex-math></inline-formula>), and <inline-formula> <tex-math notation="LaTeX">$83.1\times $ </tex-math></inline-formula>(<inline-formula> <tex-math notation="LaTeX">$56.8\times $ </tex-math></inline-formula>) speedup on MNIST, N-MNIST, and Free Spoken Digital Dataset (FSDD) respectively for training (inference), while the accuracy of LSMCore on these three datasets are 96.8&#x0025;, 97.6&#x0025;, and 90&#x0025; respectively.

A Brain-Inspired Approach for Collision-Free Movement Planning in the Small Operational Space.

In a small operational space, e.g., mesoscale or microscale, we need to control movements carefully because of fragile objects. This article proposes a novel structure based on spiking neural networks to imitate the joint function of multiple brain regions in visual guiding in the small operational space and offers two channels to achieve collision-free movements. For the state sensation, we simulate the primary visual cortex to directly extract features from multiple input images and the high-level visual cortex to obtain the object distance, which is indirectly measurable, in the Cartesian coordinates. Our approach emulates the prefrontal cortex from two aspects: multiple liquid state machines to predict distances of the next several steps based on the preceding trajectory and a block-based excitation-inhibition feedforward network to plan movements considering the target and prediction. Responding to "too close" states needs rich temporal information, and we leverage a cerebellar network for the subconscious reaction. From the viewpoint of the inner pathway, they also form two channels. One channel starts from state extraction to attraction movement planning, both in the camera coordinates, behaving visual-servo control. The other is the collision-avoidance channel, which calculates distances, predicts trajectories, and reacts to the repulsion, all in the Cartesian coordinates. We provide appropriate supervised signals for coarse training and apply reinforcement learning to modify synapses in accordance with reality. Simulation and experiment results validate the proposed method.

Effect of biologically-motivated energy constraints on liquid state machine dynamics and classification performance

Abstract Equipping edge devices with intelligent behavior opens up new possibilities for automating the decision making in extreme size, weight, and power-constrained application domains. To this end, several recent lines of research are aimed at the design of artificial intelligence hardware accelerators that have significantly reduced footprint and power demands compared to conventional CPU/GPU systems. However, despite some key advancements, the majority of work in this area assumes that there is an unlimited supply of energy available for computation, which is not realistic in the case of battery-powered and energy harvesting devices. In this paper, we address this gap by exploring the computational effects of energy constraints on a popular class of brain-inspired spiking neural networks–liquid state machines (LSMs). Energy constraints were applied by limiting the spiking activity in subsets of LSM neurons. We tested our designs on two biosignal processing tasks: epileptic seizure detection and biometric gait identification. For both tasks, we show that energy constraints can significantly improve classification accuracy. This demonstrates that in the design of neuromorphic systems, reducing energy and increasing performance are not always competing goals.