Condition-Based Maintenance Optimization Under Large Action Space with Deep Reinforcement Learning Method

Abstract

Effective maintenance is essential in keeping industrial systems running and avoiding failure. Condition-based maintenance (CBM) leverages the current degradation condition of the studied object to optimize future maintenance interventions. CBM optimization problems are complex for multi-component systems, facing the issue of the curse of dimensionality brought by the increase in the number of components. Reinforcement learning provides a promising perspective to overcome the issue. In this paper, we studied CBM optimization for a multi-component system in which the components degrade subject to the gamma process independently. We considered multiple maintenance choices for individual components, leading to a large combinatorial action space. In this case, traditional deep reinforcement learning algorithms like DQN may struggle to face the inefficiency of exploration. Instead, we propose exploiting Branching Dueling Q-network (BDQ), which incorporates the action branching architecture into DQN to drastically decrease the number of estimated actions. We trained a learning agent to minimize the expected cost for a long time horizon by taking maintenance actions according to the observed exact degrading signal for each component. We compared the policy learned by the agent with some other pre-defined static policies. The numerical results demonstrate the effectiveness of the learning algorithm and its potential for application in systems with more complex structures.