A Reinforcement Learning Approach to Find Optimal Propulsion Strategy for Microrobots Swimming at Low Reynolds Number

Abstract

The development of artificial microscopic robots, like synthetic microswimmers, is one of the state-of-the-art research topics due to their promising biomedical applications. The movement of microswimmers is affected by stringent constraints because of the low Reynolds number of the surrounding environment. Researchers have been working on enhancing microrobots’ propulsion tactics and figuring out new approaches to find optimal propulsion strategies. In this research, we employ a Reinforcement Learning (RL) algorithm, specifically Q-Learning, to train linear-shaped microrobots, comprised of spheres and rods, by introducing an innovative and pioneer coding approach, termed “Basic Coding”, which is utilized to specify states and actions within the RL framework. Basic Coding is a powerful general method that can be employed for different agents in any discrete RL environment. We show how to apply Basic Coding to various microrobots with different geometrical configurations, like a triangular one. Our smart microswimmers, with different numbers of spheres, acquire the knowledge of the optimal propulsion cycle to accomplish large net mechanical displacement without relying on any pre-existing locomotion expertise. The three-sphere linear microrobot recovers the cycle Najafi and Golestanian suggested. The N-sphere microrobots with higher degrees of freedom can find the optimal propulsion cycle within a reasonable number of learning steps and low computational cost utilizing our RL and Basic Coding approach, while the learning step number significantly increases using other methods like Brute-force search. For example, we show this number for the 5-sphere microrobot is 1.19E03 and 8.97E10 steps using our methodology and Brute-force, respectively. Furthermore, our intelligent microrobots can successfully and adaptively find new optimal strategies in indeterministic environments in the presence of uncertainty. Moreover, the effects of learning parameters on our RL agents are investigated in this work.