Interpreting Multilayer Perceptrons Using 3-Valued Activation Function

Abstract

Multilayer perceptrons (MLPs) have been successfully applied to solving many problems, but in most cases, they are used as black boxes which are not interpretable. That is, even if an MLP can provide correct answers, we cannot understand the reasons why it makes these decisions. In this study, we try to interpret a single hidden layer MLP by discretizing the hidden neuron outputs into 3 values (e.g. -1, 0, and 1). The 3 values correspond to false, unknown, and true, respectively. The basic process is 1) train an MLP first, 2) discretize the hidden neurons, 3) retrain the output layer of the MLP, 4) add more hidden neurons if needed, and 5) induce a decision tree based on the hidden neuron outputs. Experiments on several public datasets show that the proposed method is feasible for acquiring interpretable knowledge.