Low-shot transfer with attention for highly imbalanced cursive character recognition

Abstract

Recognition of ancient Korean–Chinese cursive character (Hanja) is a challenging problem mainly because of large number of classes, damaged cursive characters, various hand-writing styles, and similar confusable characters. They also suffer from lack of training data and class imbalance issues. To address these problems, we propose a unified Regularized Low-shot Attention Transfer with Imbalance τ-Normalizing (RELATIN) framework. This handles the problem with instance-poor classes using a novel low-shot regularizer that encourages the norm of the weight vectors for classes with few samples to be aligned to those of many-shot classes. To overcome the class imbalance problem, we incorporate a decoupled classifier to rectify the decision boundaries via classifier weight-scaling into the proposed low-shot regularizer framework. To address the limited training data issue, the proposed framework performs Jensen–Shannon divergence based data augmentation and incorporate an attention module that aligns the most attentive features of the pretrained network to a target network. We verify the proposed RELATIN framework using highly-imbalanced ancient cursive handwritten character datasets. The results suggest that (i) the extreme class imbalance has a detrimental effect on classification performance; (ii) the proposed low-shot regularizer aligns the norm of the classifier in favor of classes with few samples; (iii) weight-scaling of decoupled classifier for addressing class imbalance appeared to be dominant in all the other baseline conditions; (iv) further addition of the attention module attempts to select more representative features maps from base pretrained model; (v) the proposed (RELATIN) framework results in superior representations to address extreme class imbalance issue.