Multi-granularity coverage criteria for deep reinforcement learning systems

Abstract

Deep reinforcement learning (DRL) systems are progressively being deployed in safety- and security-critical domains, such as self-driving cars and unmanned aerial vehicles, raising concerns about their trustworthiness. DRL, a integration of deep learning (DL) and reinforcement learning (RL) principles, represents a machine learning technique. Existing DL coverage criteria primarily focus on neuron coverage, overlooking the unique features of RL, thus falling short in assessing the test adequacy of DRL systems.This paper introduces the first set of coverage criteria designed to systematically measure the elements of DRL systems exercised by test inputs. DRL elements, including states, actions, and policies, are leveraged to define coverage criteria that consider multi-levels and multi-granularities. Furthermore, these coverage criteria undergo optimization through the application of combinatorial coverage principles. State coverage is employed as feedback to guide test case selection for DRL systems. Empirical studies have been conducted to assess the performance of the proposed coverage criteria on five well-known DRL environments. The experiments demonstrate the effectiveness of these coverage criteria in detecting unexpected behaviors, highlighting that the proposed test case selection guided by state coverage serves as an effective strategy.