Virtual musculoskeletal arm and robotic arm driven by a biomimetic model of sensorimotor cortex with reinforcement learning

Abstract

Neocortical mechanisms of learning sensorimotor control involve a complex series of interactions at multiple levels, from synaptic mechanisms to network connectomics. We developed a model of sensory and motor cortex consisting of several hundred spiking model-neurons. A biomimetic model (BMM) was trained using spike-timing dependent reinforcement learning to drive a simple kinematic two-joint virtual arm in a motor task requiring convergence on a single target. After learning, networks demonstrated retention of behaviorally-relevant memories by utilizing proprioceptive information to perform reach-to-target from multiple starting positions. We utilized the output of this model to drive mirroring motion of a robotic arm. In order to improve the biological realism of the motor control system, we replaced the simple virtual arm model with a realistic virtual musculoskeletal arm which was interposed between the BMM and the robot arm. The virtual musculoskeletal arm received input from the BMM signaling neural excitation for each muscle. It then fed back realistic proprioceptive information, including muscle fiber length and joint angles, which were employed in the reinforcement learning process. The limb position information was also used to control the robotic arm, leading to more realistic movements. This work explores the use of reinforcement learning in a spiking model of sensorimotor cortex and how this is affected by the bidirectional interaction with the kinematics and dynamic constraints of a realistic musculoskeletal arm model. It also paves the way towards a full closed-loop biomimetic brain-effector system that can be incorporated in a neural decoder for prosthetic control, and used for developing biomimetic learning algorithms for controlling real-time devices. Additionally, utilizing biomimetic neuronal modeling in brain-machine interfaces offers the possibility for finer control of prosthetics, and the ability to better understand the brain.