UEFSD: Unsupervised medical images fusion based on exclusive features and saliency detection for SPECT-MRI images of glioma

Abstract

Background and ObjectiveTo address the key feature extraction problem in glioma image fusion due to blurred edge and texture features, with certain exclusive features entangled with noise, we propose an unsupervised fusion network based on exclusive feature unentanglement and saliency detection (UEFSD). Single-photon emission computed tomography (SPECT) and magnetic resonance imaging (MRI) are taken as examples. MethodsFirst, the SPECT image in RGB space is converted to YCbCr space. Then, Y-channel SPECT and MRI images are fed into the UEFSD-based network and the attribute and object encoders are respectively used to separate the attribute vector of exclusive information for each modality and object feature mapping based on this attribute vector. Subsequently, the attribute vector and object feature maps are fused through the fusion decoder to obtain the fused image. In the training phase, first, the image is reconstructed using the attribute vector and object feature maps of the other modalities. The feature displaying the largest difference between the source and reconstructed images is identified as the exclusive feature that cannot be obtained via reconstruction. We trained the encoders and decoders in an unsupervised manner using exclusive feature loss as a high-level constraint and introducing a saliency mask as a low-level constraint. The trained attribute encoders and object encoders feed information to the fusion decoder to generate fused images. ResultUEFSD-based network was trained on Harvard Medical School's public dataset and tested by fusing MRI-T2 glioma images with their SPECT-T1 and SPECT-Tc counterparts. The experimental results demonstrate that UEFSD performs optimally in both subjective and objective indices when compared with nine other existing networks. ConclusionThe proposed UEFSD-based network can obtain clear edge and texture glioma features, which can effectively improve the rich and natural colour information of medical images and has the potential for computer-aided diagnosis system application.