Indic handwritten script identification using offline-online multi-modal deep network

Abstract

In this paper, we propose a novel approach of word-level Indic script identification using only character-level data in training stage. Our method uses a multi-modal deep network which takes both offline and online modality of the data as input in order to explore the information from both the modalities jointly for script identification task. We take handwritten data in either modality as input and the opposite modality is generated through intermodality conversion. Thereafter, we feed this offline-online modality pair to our network. Hence, along with the advantage of utilizing information from both the modalities, the proposed framework can work for both offline and online script identification which alleviates the need for designing two separate script identification modules for individual modality. We also propose a novel conditional multi-modal fusion scheme to combine the information from offline and online modality which takes into account the original modality of the data being fed to our network and thus it combines adaptively. An exhaustive experimental study has been done on a data set including English(Roman) and 6 other official Indic scripts. Our proposed scheme outperforms traditional classifiers along with handcrafted features and deep learning based methods. Experiment results show that using only character level training data can achieve competitive performance against traditional training using word level data.