Next-Generation Neural Networks: Capsule Networks With Routing-by-Agreement for Text Classification

Nikolai A K Steur,Friedhelm Schwenker

doi:10.1109/access.2021.3110911

Nikolai A K Steur, Friedhelm Schwenker

Open Access

https://doi.org/10.1109/access.2021.3110911

Copy DOI

Abstract

These days, neural networks constantly prove their high capacity for nearly every application case and are considered as key technology for learning systems. However, neural networks need to continuously evolve for managing new arising challenges like increasing task complexity, explainability of decision making processes, expanded problem domains, providing resilient and robust systems etc. One possible enhancement of traditional neural networks constitutes the innovative Capsule Network (CapsNet) technology, which combines the expressiveness of distributed entity representations with an intelligent and interpretable signal propagation, named as routing-by-agreement. Since CapsNets represent a relatively young acquirement, further research is essential for gaining profound knowledge about CapsNet theory and best practices for diverse application areas. This paper wants to contribute to the progress of CapsNets for the task of text classification. For this purpose, various research questions about this technology get formulated and experimentally answered with the aid of six selected datasets. In addition, this paper serves as a possible starting point for researchers as well as for practitioners to deal with CapsNets in the text domain, by supplying a survey about its theory, text classification basics and the combination of both areas. The analysis results empirically prove the robustness of CapsNets with routing-by-agreement for a wide spectrum of net architectures, datasets and text classification tasks. Hence, CapsNets can be viewed as a next-generation neural network technology, which offers high potential as text classification method and should be topic of future research.

Highlights

S INCE technological advances in computer systems enabled the computation of heavy calculations in relatively small time through massively parallel systems, like mulitprocessor computers or Graphical Processing Units (GPU)s, the way was prepared for the neural network technology in many application areas
It can be said that Capsule Network (CapsNet) combine the powerful methods of distributed entity representations and intelligent routing procedures with the aim to supersede conventional neural networks with single-output neurons and static signal propagation
Prominent flaws are retaining noise through static signal propagation and the lack of explainabilty caused by non-transparency in routing and chaotic representation of entities distributed over entire layers

Summary

INTRODUCTION

S INCE technological advances in computer systems enabled the computation of heavy calculations in relatively small time through massively parallel systems, like mulitprocessor computers or Graphical Processing Units (GPU)s, the way was prepared for the neural network technology in many application areas. The idea behind capsules already arose one decade ago [2], the breakthrough of CapsNets took about six further years and the introducing of an intelligent routing between adjacent capsule layers [3]. Because of this long journey for showing the potential of CapsNets, they can be still characterized as a relatively new acquirement in the field of neural networks. The structure of this paper is defined as follows: Section II Fundamentals begins with the presentation of CapsNet theory and the covering of basic knowledge about text classification.

FUNDAMENTALS

RELATED WORK

TRAINING SETUP

FEATURE EXTRACTION ABLATION STUDY

Findings

CONCLUSION