Mechanical rotation at low Reynolds number via reinforcement learning

Abstract

There is growing interest in the development of artificial microscopic machines that can perform complex maneuvers like swimming microorganisms for potential biomedical applications. At the microscopic scales, the dominance of viscous over inertial forces imposes stringent constraints on locomotion. In the absence of inertia, Purcell first proposed an elegant way to generate net translation using kinematically irreversible motions [E. M. Purcell, “Life at low Reynolds number,” Am. J. Phys. 45, 3–11 (1977)]. In addition to net translation, a more recent prototype known as Purcell's “rotator” has been proposed in Dreyfus et al. [“Purcell's “rotator”: Mechanical rotation at low Reynolds number,” Eur. Phys. J. B 47, 161–164 (2005)] as a mechanical implementation of net rotation at low Reynolds numbers. These ingenious designs rely on knowledge of the surrounding environment and the physics of locomotion within the environment, which may be incomplete or unclear in more complex scenarios. More recently, reinforcement learning has been used as an alternative approach to enable a machine to learn effective locomotory gaits for net translation based on its interaction with the surroundings. In this work, we demonstrate the use of reinforcement learning to generate net mechanical rotation at low Reynolds numbers without requiring prior knowledge of locomotion. For a three-sphere configuration, the reinforcement learning recovers the strategy proposed by Dreyfus et al. As the number of spheres increases, multiple effective rotational strategies emerge from the learning process. However, given sufficiently long learning processes, all machines considered in this work converge to a single type of rotational policies that consist of traveling waves of actuation, suggesting its optimality of the strategy in generating net rotation at low Reynolds numbers.