The convolutional neural network (CNN) is an excellent method for image recognition. However, there are some problems in the construction process of CNN model, such as network structure setting depends on experience knowledge, network parameters selection is difficult, and there is a lack of relevance between network model and training data. To overcome the shortcomings of CNN model construction theory, in this paper, we develop a new construction approach, named adaptive deep CNN network model based on data-driven. In our method, we first set up the initial CNN model in a simple way, and the initial model only contains one feature map in the convolution layer and pooling layer. And then, the network is adaptively constructed by using the idea of learning parameters and expanding network. In network expansion, the convergence rate of CNN model is used as evaluation index of global expansion, and some global branches are added to the network model. After global expansion, the CNN is controlled to local expansion according to the recognition rate of cross validation samples. The local network learning is stopped until the recognition rate reaches the expected value. Finally, the adaptive incremental learning of network structure is realized by expanding some new branches for new samples. Experimental results on two benchmark face databases, CMU-PIE face database and MIT-CBCL face database, demonstrate the effectiveness of the proposed method.