Discriminative locally document embedding: Learning a smooth affine map by approximation of the probabilistic generative structure of subspace

Abstract

Document embedding is a technology that captures informative representations from high-dimensional observations by some structure-preserving maps over corpus and has been intensively explored in machine learning. Recently, some manifold-inspired embedding methods become a hot topic, mainly due to their ability in capturing discriminative embedding. However, the existing methods capture the embeddings based on the geometrical information of nearest neighbors without considering the intrinsic documents-generating structure on a subspace, thus leads to a limitation to uncover intrinsic semantic information. In this paper, we propose a semi-supervised local-invariant method, called Discriminative Locally Document Embedding (Disc-LDE), aiming to build a smooth affine map for document embedding by preserving documents-generating structure on a subspace. Disc-LDE models the documents-generating structure as a pseudo-document by a generative probabilistic model of subspace, where the subspace is acquired by a transductive learning of multi-agent random walk on neighborhood graph, and regularizes the training of Auto-Encoders (AEs) to jointly recover the input document and its pseudo-document. Under a general regularized function learning framework, the regularized training can impact the parameterized encoder network become smooth to variations along the documents-generating structure of the local field on manifold. The experimental results on three widely-used corpora demonstrate Disc-LDE could efficient capture the intrinsic semantic structure to improve the clustering and classification performance to the state-of-the-arts methods.