Unsupervised Generative Learning with Handwritten Digits

Abstract

Representations play an important role in learning of artificial and bi- ological systems that can be attributed to identification of characteristic patterns in the sensory data. In this work we attempted to approach the question of the origin of general concepts from the perspective of purely unsupervised learning that does not use prior knowledge of concepts to acquire the ability to recognize common patterns in a learning process resembling learning of biological systems in the natural environment. Generative models trained in an unsupervised process with minimization of generative error with a dataset of images of handwritten digits produced structured sparse latent representations that were shown to be correlated with characteristic patterns such as types of digits. Based on the iden- tified density structure, a proposed method of iterative empirical learning pro- duced confident recognition of most types of digits over a small number of learn- ing iterations with minimal learning data. The results demonstrated the possibil- ity of successful incorporation of unsupervised structure in informative represen- tations of generative models for successful empirical learning and conceptual modeling of the sensory environments.