Developing a web application for Chinese calligraphy learners using convolutional neural network and scale invariant feature transform

Abstract

Chinese Calligraphy is an ancient art form with practical benefits such as improving elders' cognition and helping diagnose Parkinson's disease. Learning Chinese calligraphy requires a lot of time and effort. This is due to the different styles and the vast number of Chinese characters used daily. Chinese character and style recognition are multi-class classification problems. In multi-class classification problems, the accuracy decreases as the number of classes increases. Previous research on Chinese calligraphy was limited to either style recognition or character recognition. More research that combines style and character recognition is needed. Furthermore, the models in previous research were limited to recognizing small sets of 300 characters. These small sets cover only 12% of the Chinese characters used daily. The small size is a severe limitation for creating applications for Chinese calligraphy learning. Thus, new research is needed on AI models that combine style and character recognition to recognize a much larger dataset, while maintaining reasonable accuracy rates. Besides, there are no applications that use such AI models to help learners of Chinese calligraphy compare their works with those created by skillful calligraphers. This research proposed enhancing Chinese calligraphy recognitions by combining the VGG (Visual Geometry Group)-net architecture, data augmentation techniques, and dropout in the ANN (artificial neural network) hidden layers. The research produced two AI models that can recognize 962 Chinese characters in five calligraphic styles. We proved that using dropout increased the accuracy convergence during training epochs. In the proposed models, style recognition achieved up to 88.3% accuracy, while character recognition achieved 95.6% accuracy. Finally, we built an online learning application by combining the proposed models with Scale Invariant Feature Transform (SIFT). The application could compare the learner's work with skillful calligraphers' and provide a similarity score to help Chinese calligraphy learners track their progress. Five university students tested the application. They provided positive feedback and some suggestions for improvement.