Knowledge distilling based model compression and feature learning in fault diagnosis

Abstract

Recently, there has been interest in developing diagnosis methods that combine model-based and data-driven diagnosis. In both approaches, selecting the relevant measurements or extracting important features from historical data is a key determiner of the success of the algorithm. Recently, deep learning methods have been effective in automating the feature selection process. Autoencoders have been shown to be an effective neural network configuration for extracting features from complex data, however, they may also learn irrelevant features. In addition, end-to-end classification neural networks have also been used for diagnosis, but like autoencoders, this method may also learn unimportant features thus making the diagnostic inference scheme inefficient. To rapidly extract significant fault features, this paper employs end-to-end networks and develops a new feature extraction method based on importance analysis and knowledge distilling. First, a set of cumbersome neural network models are trained to predict faults and some of their internal values are defined as features. Then an occlusion-based importance analysis method is developed to select the most relevant input variables and learned features. Finally, a simple student neural network model is designed based on the previous analysis results and an improved knowledge distilling method is proposed to train the student model. Because of the way the cumbersome networks are trained, only fault features are learned, with the importance analysis further pruning the relevant feature set. These features can be rapidly generated by the student model. We discuss the algorithms, and then apply our method to two typical dynamic systems, a communication system and a 10-tank system employed to demonstrate the proposed approach.