Fault-Tolerant Control of Skid Steering Vehicles Based on Meta-Reinforcement Learning with Situation Embedding

Huatong Dai,Pengzhan Chen,Hui Yang

doi:10.3390/act11030072

Huatong Dai, Pengzhan Chen + Show 1 more

Open Access

https://doi.org/10.3390/act11030072

Copy DOI

Journal: Actuators	Publication Date: Feb 25, 2022
Citations: 2	License type: CC BY 4.0

Affiliation: East China Jiaotong University

Abstract

Meta-reinforcement learning (meta-RL), used in the fault-tolerant control (FTC) problem, learns a meta-trained model from a set of fault situations that have a high-level similarity. However, in the real world, skid-steering vehicles might experience different types of fault situations. The use of a single initial meta-trained model limits the ability to learn different types of fault situations that do not possess a strong similarity. In this paper, we propose a novel FTC method to mitigate this limitation, by meta-training multiple initial meta-trained models and selecting the most suitable model to adapt to the fault situation. The proposed FTC method is based on the meta deep deterministic policy gradient (meta-DDPG) algorithm, which includes an offline stage and an online stage. In the offline stage, we first train multiple meta-trained models corresponding to different types of fault situations, and then a situation embedding model is trained with the state-transition data generated from meta-trained models. In the online stage, the most suitable meta-trained model is selected to adapt to the current fault situation. The simulation results demonstrate that the proposed FTC method allows skid-steering vehicles to adapt to different types of fault situations stably, while requiring significantly fewer fine-tuning steps than the baseline.

Highlights

IntroductionDue to their simple mechanical structure and flexible control, skid-steering distributed drive vehicles have been widely applied in various scenarios, including wheeled robots [1,2], agricultural vehicles [3], military vehicles [4,5], and so on
The main purpose of this study is to develop an fault-tolerant control (FTC) method based on Meta-reinforcement learning (meta-RL) which allows skid-steering vehicles to adapt to different types of fault situations
The main contributions of this study are as follows: (1) We develop a meta-DDPGSEbased FTC method for skid steering vehicles, which achieves high performance on different types of fault situations; (2) a situation embedding model is introduced into the conventional meta-RL-based FTC framework, in order to select the most suitable meta-trained model for the current fault situation; and (3) selecting the most suitable meta-trained model based on online data set allows the agent to quickly adapt the policy to different types of fault situations

Summary

Introduction

Due to their simple mechanical structure and flexible control, skid-steering distributed drive vehicles have been widely applied in various scenarios, including wheeled robots [1,2], agricultural vehicles [3], military vehicles [4,5], and so on. A skidsteering vehicle has four independent driving wheels, forming a redundant actuator system, which delivers remarkable maneuverability and more options for FTC methods [6]. With the development of meta-RL algorithms, some recent studies applied meta-RL algorithms to the FTC problem of actuator faults, providing new insights into the FTC mechanism of skid-steering vehicles [7,8]. Meta-RL learns an initial meta-trained model from a set of fault situations that have high-level similarity. The single initial meta-trained model in conventional meta-RL algorithms limits the ability to learn fault situations that do not have strong similarity to the learned faults, leading to low generalizability to novel fault situations [10,11]

Objectives

Results

Conclusion