A neural architecture search based framework for liquid state machine design

Abstract

Liquid state machines (LSMs), also known as the recurrent version of spiking neural networks, have garnered significant research interest owing to their high computational power, biological plausibility, simple structure, and low training complexity. By exploring the design space in network architectures and parameters, recent works have demonstrated the great potential for improving the accuracy of LSM models with low complexity. However, these works are based on manually defined network architectures or predefined parameters, which may ignore the potential optimization of the architectures and parameters of LSMs. In this study, we propose a neural architecture search-based framework to explore the architecture and parameter design space for the automatic dataset-oriented LSM models. To manage the exponentially increasing design space, we adopt a three-step search for LSMs, including dynamic multiple-liquid architecture search in multiple layers, variations in the number of neurons in each liquid, and parameter search such as percentage connectivity and excitatory neuron ratio within each liquid. In addition, we propose the use of a simulated annealing algorithm to implement three-step heuristic search. Two datasets, including the image dataset of NMNIST and speech dataset of FSDD, were used to test the effectiveness of the proposed framework. Simulation results demonstrated that our framework can produce the dataset-oriented optimal LSM models with high accuracy and low complexity. The best classification accuracy on the two datasets with only 1000 spiking neurons was observed to be 92.5% and 84%. Meanwhile, the network connections of discovered optimal multiple-liquid LSM models for the two datasets, on average, were reduced by 56.3% and 60.2% separately compared with a single LSM. Furthermore, the total number of neurons in the optimal multiple-liquid LSM models on the two datasets was reduced by 20% with an accuracy loss of only 0.5%.