To realize the secure transmission of double images, this paper proposes a double image encryption algorithm based on convolutional neural network (CNN) and dynamic adaptive diffusion. This scheme is different from the existing double image encryption technology. According to the characteristics of digital image, we design a dual-channel (digital channel / optical channel) encryption method, which not only ensures the security of double image, but also improves the encryption efficiency and reduces the possibility of being attacked. First, a chaotic map is used to control the initial values of the 5D conservative chaotic system to enhance the security of the key. Secondary, in order to effectively resist known-plaintext attack and chosen-plaintext attack, we employ a chaotic sequence as convolution kernel of convolution neural network to generate plaintext related chaotic pointer to control the scrambling operation of two images. On this basis, a novel image fusion method is designed, which splits and fuses two images into two different parts according to the amount of information contained. In addition, a dual-channel image encryption scheme, optical encryption channel and digital encryption channel, is designed for the two parts after fusion. The former has better parallelism and higher encryption efficiency, while the latter has higher computational complexity and better encryption reliability. Especially in the digital encryption channel, a new dynamic adaptive diffusion method is designed, which is more flexible and secure than the existing encryption algorithm. Finally, numerical simulation and experimental analysis verify the feasibility and effectiveness of the scheme.