Is it worth generating rules from neural network ensembles?

Abstract

Although many authors generated comprehensible models from individual networks, much less work has been done in the explanation of ensembles. DIMLP is a special neural network model from which rules are generated at the level of a single network and also at the level of an ensemble of networks. We applied ensembles of 25 DIMLP networks to several datasets of the public domain and a classification problem related to post-translational modifications of proteins. For the classification problems of the public domain, the average predictive accuracy of rulesets extracted from ensembles of neural networks was significantly better than the average predictive accuracy of rulesets generated from ensembles of decision trees. By varying the architectures of DIMLP networks we found that the average predictive accuracy of rules, as well as their complexity were quite stable. The comparison to other rule extraction techniques applied to neural networks showed that rules generated from DIMLP ensembles gave very good results. In the last problem related to bioinformatics, the best result obtained by ensembles of DIMLP networks was also significantly better than the best result obtained by ensembles of decision trees. Thus, although neural networks take much longer to train than decision trees and also rules are generated at a greater computational cost (however, still polynomial), at least for several classification problems it was worth using neural network ensembles, as extracted rules were more accurate, on average. The DIMLP software is available for PC-Linux under http://us.expasy.org/people/Guido.Bologna.html.