Combining visual and textual features for medical image modality classification with ℓp−norm multiple kernel learning

Abstract

Abstract Automatic modality classification of medical images is an important tool for medical image retrieval. In this paper, we combine visual and textual information for modality classification. The visual features used are SIFT feature, LBP feature, Gabor texture feature and Tamura texture feature. And the textual feature is a tf–idf feature vector drawn from image description text. We combine these features by l p - norm multiple kernel learning ( l p - norm MKL), and use One-vs-All approach for this multi-class problem. l p - norm MKL is explored with different norm value ( p ≥ 1 ). These MKL based methods are compared with several other feature combination methods and evaluated on the dataset of modality classification task in ImageCLEFmed 2010. The experimental results indicate that multiple kernel learning is a promising approach to combine visual and textual features for modality classification, and outperforms other simple kernel combination methods and the traditional early fusion method.