Multi-focus image fusion combines the focus parts of multiple images in the same scene to generate a fully focused image. Aiming at the problem of multi-focus image fusion, this paper optimizes the network design based on the existed image fusion network. The optimized network uses two groups of dilated convolutions with different numbers and sizes to extract shallow features, and then integrates the two groups of features through 1 × 1 convolution. Extracting deep information from feature map by dense block and the dense block are composed of four layers of dense connection convolution. In addition, the fused image is processed by sub-pixel convolution to obtain a super-resolution fused image, and a perceptual loss function is introduced to optimize the network. The experimental results show that the optimized network is superior to the existing fusion methods in objective and subjective evaluation. It has a certain application value in the field of image fusion.