Robot dynamic model is widely applied to control, collision detection and motion planning. Accurate dynamic model can achieve better performance for the above applications. Traditional dynamic models have several limitations, such as the complex hypotheses for friction model and the requirement of additional joint torque sensors. This article constructs a convolution neural network (CNN) based semi-parametric dynamic (SPD) model by only using the motor encoder signals and motor currents. The SPD model not only contains the physically feasible parameters but also compensates the dynamic model by CNN. The parametric and non-parametric parts constitute the SPD model. A lightweight CNN is proposed to simultaneously ensure the accuracy and computational efficiency. To effectively train the CNN model, a dataset generation method, which expands the excitation trajectory and only uses a continuous trajectory to record data, is proposed. The CNN-based SPD model is verified on a 6-DoF laboratory-developed industrial robot only with the proprioceptive sensors. Compared with the traditional rigid body dynamics (RBD) model, the average error of the CNN-based SPD model is reduced by 9.23% in terms of the experimental results. Meanwhile, the proposed CNN-based method achieves better performance than other supervised methods.