On the Computational Power of Convolution Pooling: A Theoretical Approach for Deep Learning

Abstract

Convolutional neural networks (CNNs) have been widely used for image analysis and recognition. For example, LeNet-5 is a 7-layer convectional neural network, which can attain more than 99% test accuracy for classification of handwritten digits. CNNs repeats convolution and pooling operations alternately. However, the computational capability of such operations is not clear. We are curious to know a class of problems that can be solved by CNNs. As a formal approach for this task, we introduce a theoretical parallel computational model of CNNs that we call the convolution-pooling machine. It captures the essence of convolution and pooling operations, and application of non-linear activation functions performed in CNNs. In this paper, we assume the convolution-pooling machine operating on 1-dimensional arrays for simplicity, and focus on the problem of classification of inputs by the distance of two feature points. More specifically, we will design a convolution-pooling machine solving the problem D <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</sub> (k≥1), a problem to determine if the distance of the two 1's is at most k or not. For designing the convolution-pooling machine solving the problem D <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</sub> , we generate a mixed-integer linear programming problem (MILP) with constraints and objective functions. We have solved the generated linear programming problem for each D <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</sub> (1≤k≤128) by Gurobi optimizer, a commercial MILP solver. We succeeded in finding a solution for all D <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</sub> (1 ≤ k ≤ 128) and designing the convolution-pooling machine for solving them. This fact indicates that convolution and pooling operations in CNNs may have the computational capability of classification by the distance of feature points.