Abstract
A natural way to determine the knowledge embedded within connectionist models is to generate symbolic rules. Nevertheless, extracting rules from Multi Layer Perceptrons (MLPs) is NP-hard. With the advent of social networks, techniques applied to Sentiment Analysis show a growing interest, but rule extraction from connectionist models in this context has been rarely performed because of the very high dimensionality of the input space. To fill the gap we present a case study on rule extraction from ensembles of Neural Networks and Support Vector Machines (SVMs), the purpose being the characterization of the complexity of the rules on two particular Sentiment Analysis problems. Our rule extraction method is based on a special Multi Layer Perceptron architecture for which axis-parallel hyperplanes are precisely located. Two datasets representing movie reviews are transformed into Bag-of-Words vectors and learned by ensembles of neural networks and SVMs. Generated rules from ensembles of MLPs are less accurate and less complex than those extracted from SVMs. Moreover, a clear trade-off appears between rules’ accuracy, complexity and covering. For instance, if rules are too complex, less complex rules can be re-extracted by sacrificing to some extent their accuracy. Finally, rules can be viewed as feature detectors in which very often only one word must be present and a longer list of words must be absent.
Highlights
The inherent black-box nature of neural networks makes it difficult to trust
Even if we did not use Convolutional Neural Networks (CNN), our main contribution is that we performed a rule extraction study in sentiment analysis that has been rarely carried out with shallow architectures
We characterized the complexity of the knowledge embedded within models similar to Support Vector Machines (SVMs) and ensemble of Multi Layer Perceptrons (MLPs)
Summary
Over the last two decades various works showed that it is possible to determine the knowledge stored into neural networks by extracting symbolic rules. Andrews et al introduced a taxonomy to characterize all rule extraction techniques [1]. With the advent of social networks, a growing interest aiming at determining emotional reactions of individuals from text has been observed. Sentiment Analysis techniques try to gauge people feelings with respect to new products, artistic events, or marketing/political campaigns. Sentiment analysis techniques are grouped into three main categories [2]: knowledge-based techniques; statistical methods; and hybrid approaches. Knowledge-based techniques analyze text by detecting unambiguous words such as happy, sad, afraid, and bored [Ort90]. Statistical methods use several components from machine learning such as latent semantic analysis [3], Support Vector
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have