Performing the Kick During Walking for RoboCup 3D Soccer Simulation League Using Reinforcement Learning Algorithm

Abstract

Nowadays, humanoid soccer serves as a benchmark for artificial intelligence and robotic problems. The factors such as the kicking speed and the number of kicks by robot soccer players are the most significant aims that the participating teams are pursued in the RoboCup 3D Soccer Simulation League. The proposed method presents a kicking strategy during walking for humanoid soccer robots. Achieving an accurate and powerful kicking while robots are moving requires a dynamic optimization of the speed and motion parameters of the robot. In this paper, a curved motion path has been designed based on the robot position relative to the ball and the goal. Ultimately, the robot will be able to kick at the goal by walking along this curve path. The speed and angle of the walking robot are set towards the ball with regard to the robots curved motion path. After the final step of the robot, the accurate and effective adjustment of these two parameters ensures that the robot is located in the ideal position to perform the perfect kick. Due to the noise and walking condition of the robot, it is essential that the speed and angle of motion to be measured more accurately. For this purpose, we use a reinforcement learning model to adjust the robots step size and so does achieve the optimal value of two abovementioned parameters. Using reinforcement learning, robot would learn to pursue an optimal policy to correctly kick towards designated points. Therefore, the proposed method is a model-free and based on dynamic programming. The experiments reveal that the proposed method has significantly improved the team overall performance and robots ability to kick. Our proposed method has been 9.32% successful on average and outperformed the UTAustinVilla agent in terms of goal-scoring time in a non-opponent simulator.