A Computational Framework Towards Medical Image Explanation

Abstract

In this paper, a unified computational framework towards medical image explanation is proposed to promote the ability of computers on understanding and interpreting medical images. Four complementary modules are included, such as the construction of Medical Image-Text Joint Embedding (MITE) based on large-scale medical images and related texts; a Medical Image Semantic Association (MISA) mechanism based on the MITE multimodal knowledge representation; a Hierarchical Medical Image Caption (HMIC) module that is visually understandable to radiologists; and a language-independent medical imaging report generation prototype system by integrating the HMIC and transfer learning method. As an initial study of automatic medical image explanation, preliminary experiments were carried out to verify the feasibility of the proposed framework, including the extraction of large scale medical image-text pairs, semantic concept detection from medical images, and automatic medical imaging reports generation. However, there is still a great challenge to produce medical image interpretations clinically usable, and further research is needed to empower machines explaining medical images like a human being.