With the deepening and improvement of the contemporary English educating reform, the lookup on the satisfactory English training has attracted greater and extra attention. The key to enhance the English training is to enhance good teaching, and English teaching model is the key measure to enhance good schooling and teaching. Based on a single neural network, it can solely describe the randomness and irregularity of English education quality and cannot describe the whole exchange traits of English education model, which makes the impact deviation of teaching model larger. Based on the in-depth learning of the contemporary state of affairs and traits of English education model, blended with the traits of neural network, this paper constructs an English teaching model primarily based on multimodal neural network algorithm. The experimental results show that the convergence speed of multimodal neural network model is 76% higher than that of single network model, the sum of squares of average error is 79%, and the average evaluation accuracy is 13.99% and 6.42% higher than that of convolution neural network model and radial basis function neural network model, respectively. It is demonstrated that the multimodal neural network model does not accelerate the convergence speed of the network or improve the prediction accuracy of the model and can quickly realize the ability of global optimization. It shows the effectiveness and accuracy of using multimodal neural network algorithm to model English teaching quality and provides a feasible solution for teaching quality model.