Statistical insights into deep neural network learning in subspace classification

Abstract

Deep learning has benefited almost every aspect of modern big data applications. Yet its statistical properties still remain largely unexplored. It is commonly believed nowadays that deep neural networks (DNNs) benefit from representational learning. To gain some statistical insights into this, we design a simple simulation setting where we generate data from some latent subspace structure with each subspace regarded as a cluster. We empirically demonstrate that the performance of DNN is very similar to that of the two‐step procedure of clustering followed by classification (unsupervised plus supervised). This motivates us to ask: Does DNN indeed mimic the two‐step procedure statistically? That is, do bottom layers in DNN try to cluster first and then top layers classify within each cluster? To answer this question, we conduct a series of simulation studies, and to our surprise, none of the hidden layers in DNN conduct successful clustering. In some sense, our results provide an important complement to the common belief of representational learning, suggesting that at least in some model settings, although the performance of DNN is comparable with that of the ideal two‐step procedure knowing the true latent cluster information a priori, it does not really do clustering in any of its layers. We also provide some statistical insights and heuristic arguments to support our empirical discoveries and further demonstrate the revealed phenomenon on the real data application of traffic sign recognition.