Adopting attention-mechanisms for Neural Logic Rule Layers

Abstract

Abstract In previous works we discovered that rule-based systems severely suffer in performance when increasing the number of rules. In order to increase the amount of possible boolean relations while keeping the number of rules fixed, we employ ideas from well known Spatial Transformer Systems and Self-Attention Networks: here, our learned rules are not static but are dynamically adjusted to fit the input data by training a separate rule-prediction system, which is predicting parameter matrices used in Neural Logic Rule Layers. We show, that these networks, termed Adaptive Neural Logic Rule Layers, outperform their static counterpart both in terms of final performance, as well as training stability and excitability during early stages of training.