Emergence of Network Motifs in Deep Neural Networks.

Matteo Zambra,Amos Maritan,Alberto Testolin

doi:10.3390/e22020204

Matteo Zambra, Amos Maritan + Show 1 more

Open Access

https://doi.org/10.3390/e22020204

Copy DOI

Journal: Entropy	Publication Date: Feb 11, 2020
Citations: 6	License type: CC BY 4.0

Affiliation: University of Padua, INFN Sezione di Padova

Abstract

Network science can offer fundamental insights into the structural and functional properties of complex systems. For example, it is widely known that neuronal circuits tend to organize into basic functional topological modules, called network motifs. In this article, we show that network science tools can be successfully applied also to the study of artificial neural networks operating according to self-organizing (learning) principles. In particular, we study the emergence of network motifs in multi-layer perceptrons, whose initial connectivity is defined as a stack of fully-connected, bipartite graphs. Simulations show that the final network topology is shaped by learning dynamics, but can be strongly biased by choosing appropriate weight initialization schemes. Overall, our results suggest that non-trivial initialization strategies can make learning more effective by promoting the development of useful network motifs, which are often surprisingly consistent with those observed in general transduction networks.

Highlights

The topological structure of complex networks can be characterized by a series of well-known features, such as the small-world and scale-free properties, the presence of cliques and cycles, modularity, and so on, which are instead missing in random networks [1,2,3,4,5]
The orthogonal initialization scheme allowed convergence in fewer epochs, while the Xavier scheme resulted in the fastest convergence. These findings suggest that initialization plays a crucial role in shaping learning dynamics: one possible explanation could be that the orthogonal and Xavier schemes impress a sharper fingerprint to the initial significance landscape of network motifs, as we will discuss below
Individual units by themselves do not accomplish any relevant function, because it is the coordinated arrangement of groups of units that allows for the emergence of system-level, macroscopic properties [28]

Summary

Introduction

The topological structure of complex networks can be characterized by a series of well-known features, such as the small-world and scale-free properties, the presence of cliques and cycles, modularity, and so on, which are instead missing in random networks [1,2,3,4,5]. One might hope to “understand the dynamics of the entire network based on the dynamics of the individual building blocks” (see Chapter 3 in Reference [9]) In this respect, we can regard network motifs as basic structural modules which bear (in a topological sense) meaningful insights about the holistic behavior of the system as a whole. We can regard network motifs as basic structural modules which bear (in a topological sense) meaningful insights about the holistic behavior of the system as a whole We apply this perspective to the study of multi-layer (deep) neural networks, which are one of the most popular frameworks used in modern artificial intelligence applications [10,11]. To quote Reference [16], “the theoretical principles governing how even simple artificial neural networks extract semantic knowledge from

Methods

Results

Conclusion