Kernel-based topographic map formation achieved with an information-theoretic approach

Abstract

A new information-theoretic learning algorithm is introduced for kernel-based topographic map formation. The kernels are allowed to overlap and move freely in the input space, and to have differing kernel ranges. We start with Linsker's infomax principle and observe that it cannot be readily extended to our case, exactly due to the presence of kernels. We then consider Bell and Sejnowski's generalization of Linsker's infomax principle, which suggests differential entropy maximization, and add a second component to be optimized, namely, mutual information minimization between the kernel outputs, in order to take into account the kernel overlap, and thus the topographic map's output redundancy. The result is joint entropy maximization of the kernel outputs, which we adopt as our learning criterion. We derive a learning algorithm and verify its performance both for a synthetic example, for which the optimal result can be derived analytically, and for a classic real-world example.