Abstract
The Kullback–Leibler (KL) divergence is a fundamental measure of information geometry that is used in a variety of contexts in artificial intelligence. We show that, when system dynamics are given by distributed nonlinear systems, this measure can be decomposed as a function of two information-theoretic measures, transfer entropy and stochastic interaction. More specifically, these measures are applicable when selecting a candidate model for a distributed system, where individual subsystems are coupled via latent variables and observed through a filter. We represent this model as a directed acyclic graph (DAG) that characterises the unidirectional coupling between subsystems. Standard approaches to structure learning are not applicable in this framework due to the hidden variables; however, we can exploit the properties of certain dynamical systems to formulate exact methods based on differential topology. We approach the problem by using reconstruction theorems to derive an analytical expression for the KL divergence of a candidate DAG from the observed dataset. Using this result, we present a scoring function based on transfer entropy to be used as a subroutine in a structure learning algorithm. We then demonstrate its use in recovering the structure of coupled Lorenz and Rössler systems.
Highlights
Distributed information processing systems are commonly studied in complex systems and machine learning research
Entropy 2018, 20, 51 multidisciplinary study in fields such as ecology [4], neuroscience [5,6], multi-agent systems [7,8,9], and various others that focus on artificial and biological networks [10]. We represent such a spatially distributed system as a probabilistic graphical model termed a synchronous graph dynamical system (GDS) [11,12], whose structure is given by a directed acyclic graph (DAG)
We have presented a principled method to compute the KL divergence for model selection in distributed dynamical systems based on concepts from differential topology
Summary
Distributed information processing systems are commonly studied in complex systems and machine learning research. Entropy 2018, 20, 51 multidisciplinary study in fields such as ecology [4], neuroscience [5,6], multi-agent systems [7,8,9], and various others that focus on artificial and biological networks [10] We represent such a spatially distributed system as a probabilistic graphical model termed a synchronous graph dynamical system (GDS) [11,12], whose structure is given by a directed acyclic graph (DAG). We draw on state space reconstruction methods from differential topology to reformulate the KL divergence in terms of computable distributions Using this expression, we show that the maximum transfer entropy graph is the most likely to have generated the data.
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
