Effects of loss function and data sparsity on smooth manifold extraction with deep model

Abstract

Deep model is a useful tool that can extract smooth manifold from data. As a computationally intensive method, however, model parameters and data sparsity are common factors that are likely to cause uncertainty and consequently affect the result. Thus, it is a challenge to design an effective deep model for smooth manifold extraction without prior knowledge to the target datasets, which is usually sparse in many real-world applications. In this paper, we proposed a deep model based on Brenier theorem to extract smooth manifold. Beyond the model design, several experiments on both simulated and real-world datasets were conducted to explore three scientific questions: 1) How to design or select an appropriate loss function for a deep neural network to better fit the data, which will be helpful to extract smooth manifold? 2) How deep model’s sensitivity to data sparsity affect the smoothness of manifold extraction? And 3) What are the relative importance of these factors in improving the smoothness of manifold extraction? Results showed that our model outperformed the state-of-the-art models in many metrics, such as manifold smoothness and computational error. The experiments demonstrated that the less sensitivity a model is to data sparsity, the smoother manifold it can extract. Our finding indicated that the smoothness of a manifold extracted by deep models is significantly dependent on the layout of loss functions, which means that simply stack hidden layers in deep models is a less effective way to improve the smoothness of manifold extraction. In addition, our exploration also suggested that the selection of loss functions carries more weights in improving manifold smoothness than investing the effort of overcoming data sparsity.