Deep learning technology has been extensively explored in pattern recognition and image processing areas. A multi-mode medical image fusion with deep learning will be proposed, according to the characters of multi-modal medical image, medical diagnostic technology and practical implementation, according to the practical needs for medical diagnosis. It cannot be only made up for the deficiencies of MRI, CT and SPECT image fusion, but also can be implemented in different types of multi-modal medical image fusion problems in batch processing mode, and can be effectively overcome the limitation of only one-page processing. The proposed method can greatly improve the fusion effect, image detail clarity and time efficiency. The experiments on multi-modal medical images are implemented to analyze performance, algorithm stability and so on. The experimental results prove the superiority of our proposed method in terms of visual quality and a variety of quantitative evaluation criteria.