DyGAT: Dynamic stroke classification of online handwritten documents and sketches

Abstract

Online handwriting is widely used in human-machine interface, education, office automation, and so on. Stroke classification for online handwritten documents and sketches aims to divide strokes into several semantic categories and is a necessary step for document recognition and understanding. Previous methods are essentially static in that they have to wait for the user to finish the whole sketch before making prediction. However, in practice, the more user-friendly way is to make real-time prediction as the user is writing. In this paper, we introduce Dynamic Graph ATtention network (DyGAT) to solve the dynamic stroke classification problem. The core of our method is to formalize a document/sketch into a multi-feature graph, in which nodes represent strokes, edges represent the relationships between strokes, and multiple nodes are applied to one stroke to control the information flow. The proposed method is general and is applicable to online handwritten data of many types. We conduct experiments on popular public datasets to perform sketch semantic segmentation, document layout analysis and diagram recognition, and experimental results show competitive performance. Particularly, the proposed method achieves stroke classification accuracies which are only slightly lower than those of static classification.