Online Handwritten Recognition Research Articles

Increasing Offline Handwritten Chinese Character Recognition Using Separated Pre-Training Models: A Computer Vision Approach

Offline handwritten Chinese character recognition involves the application of computer vision techniques to recognize individual handwritten Chinese characters. This technology has significantly advanced the research in online handwriting recognition. Despite its widespread application across various fields, offline recognition faces numerous challenges. These challenges include the diversity of glyphs resulting from different writers’ styles and habits, the vast number of Chinese character labels, and the presence of morphological similarities among characters. To address these challenges, an optimization method based on a separated pre-training model was proposed. The method aims to enhance the accuracy and robustness of recognizing similar character images by exploring potential correlations among them. In experiments, the HWDB and Chinese Calligraphy Styles by Calligraphers datasets were employed, utilizing precision, recall, and the Macro-F1 value as evaluation metrics. We employ a convolutional self-encoder model characterized by high recognition accuracy and robust performance. The experimental results demonstrated that the separated pre-training models improved the performance of the convolutional auto-encoder model, particularly in handling error-prone characters, resulting in an approximate 6% increase in precision.

A self-attention-based deep architecture for online handwriting recognition

A self-attention-based deep architecture for online handwriting recognition

SpaceGTN: A Time-Agnostic Graph Transformer Network for Handwritten Diagram Recognition and Segmentation

Online handwriting recognition is pivotal in domains like note-taking, education, healthcare, and office tasks. Existing diagram recognition algorithms mainly rely on the temporal information of strokes, resulting in a decline in recognition performance when dealing with notes that have been modified or have no temporal information. The current datasets are drawn based on templates and cannot reflect the real free-drawing situation. To address these challenges, we present SpaceGTN, a time-agnostic Graph Transformer Network, leveraging spatial integration and removing the need for temporal data. Extensive experiments on multiple datasets have demonstrated that our method consistently outperforms existing methods and achieves state-of-the-art performance. We also propose a pipeline that seamlessly connects offline and online handwritten diagrams. By integrating a stroke restoration technique with SpaceGTN, it enables intelligent editing of previously uneditable offline diagrams at the stroke level. In addition, we have also launched the first online handwritten diagram dataset, OHSD, which is collected using a free-drawing method and comes with modification annotations.

A new dataset for mongolian online handwritten recognition

This paper introduces a new traditional Mongolian word-level online handwriting dataset, MOLHW. The dataset consists of handwritten Mongolian words, including 164,631 samples written by 200 writers and covering 40,605 Mongolian common words. These words were selected from a large Mongolian corpus. The coordinate points of words were collected by volunteers, who wrote the corresponding words on the dedicated application for their mobile phones. Latin transliteration of Mongolian was used to annotate the coordinates of each word. At the same time, the writer’s identification number and mobile phone screen information were recorded in the dataset. Using this dataset, we propose an encoder–decoder Mongolian online handwriting recognition model with a deep bidirectional gated recurrent unit and attention mechanism as the baseline evaluation model. Under this model, the optimal performance of the word error rate (WER) on the test set was 24.281%. Furthermore, we present the experimental results of different Mongolian online handwriting recognition models. The experimental results show that compared with other models, the model based on Transformer could learn the corresponding character sequences from the coordinate data of the dataset more effectively, with a 16.969% WER on the test set. The dataset is now freely available to researchers worldwide. The dataset can be applied to handwritten text recognition as well as handwritten text generation, handwriting identification, and signature recognition.

A survey on different feature extraction methods for writer identification and verification

Identifying and verifying a person based on scanned images of their handwriting is a needful biometric application in historical document analysis, behavioural biometrics study, forensic science, access control, graphology, and copyrights management. Writer identification and verification are still challenging in offline and online handwriting recognition. Since the performances of handwriting biometric identification and verification systems depend on both the quality and types of chosen features, this is one of the most critical phases. This article represents a literature survey on offline and online biometric features used in different scripts for writer verification and identification techniques. Several previous efficient works on online and offline writer authentication methods for biometrics using cutting-edge hand-craft features in different levels of handwriting analysis like documents, paragraphs, words, and characters are analysed systematically to date for the first time in detail.

Leveraging deep feature learning for wearable sensors based handwritten character recognition

Despite rapid advancements in technology, handwritten characters still hold significant roles in various fields, including education, communication, biometric signature verification, and health care. These applications often require digitization of the handwritten characters and associated hand movements to facilitate effective analysis and interpretation of the underlying task. Offline and online handwriting recognition are crucial steps involving the digitization of handwritten characters. Most of these existing systems actively use image processing techniques that are highly sensitive to environmental lighting conditions. Surface Electromyography signals (sEMG), being invariant to lighting conditions, are used in online handwriting recognition to facilitate the automatic transcription of handwritten characters. In this article, we have leveraged deep representation learning to build an efficient and robust sEMG-based Handwritten Character Recognition (HCR) pipeline. A Stacked sparse denoising autoencoder network is applied to obtain an effective deep feature representation. These rich low dimensional features obtained are further introduced into basic classifiers, producing state-of-the-art accuracy for the task. Additional experiments were performed to analyze the effect of the fusion of complementary sensing modules (Accelerometer and Gyroscope) on the performance of sEMG based HCR pipeline. Extensive evaluations were performed to ensure the validity of the obtained results. For the experimentation, new datasets consisting of sEMG, Accelerometer, and Gyroscope signals corresponding to 26 handwritten lower English alphabets were collected from 15 subjects. Our proposed pipeline can be used to build real-time Human–Computer Interaction(HCI) applications for smart classrooms facilitating digitization of handwritten notes and clinical applications involving handwriting analysis tasks.

Handwriting Recognition Based on 3D Accelerometer Data by Deep Learning

Online handwriting recognition has been the subject of research for many years. Despite that, a limited number of practical applications are currently available. The widespread use of devices such as smartphones, smartwatches, and tablets has not been enough to convince the user to use pen-based interfaces. This implies that more research on the pen interface and recognition methods is still necessary. This paper proposes a handwritten character recognition system based on 3D accelerometer signal processing using Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM). First, a user wearing an MYO armband on the forearm writes a multi-stroke freestyle character on a touchpad by using the finger or a pen. Next, the 3D accelerometer signals generated during the writing process are fed into a CNN, LSTM, or CNN-LSTM network for recognition. The convolutional backbone obtains spatial features in order to feed an LSTM that extracts short-term temporal information. The system was evaluated on a proprietary dataset of 3D accelerometer data collected from multiple users with an armband device, corresponding to handwritten English lowercase letters (a–z) and digits (0–9) in a freestyle. The results show that the proposed system overcomes other systems from the state of the art by 0.53%.

Writer adaptation for E2E Arabic online handwriting recognition via adversarial multi task learning

The importance of online handwriting recognition has been rapidly increasing over recent years due to the rapid technological advances in handheld devices and communication software with handwriting interfaces. Deep learning end-to-end (E2E) models have provided high recognition rates as part of online handwriting recognition systems. However, attaining even higher performance levels requires supplementing these models with adaptation techniques that cater to individual penmanship. This study proposes a writer adaptation technique for Arabic online handwriting recognition systems that employs adversarial Multi-Task Learning (MTL). Adversarial training and MTL modify the deep-features distribution of the Writer Dependent (WD) model, leading its output to closely resemble that of the Writer Independent (WI) model. The design of the proposed method entails two tasks: label classification (primary task) and model features discrimination (secondary task). Our method was designed to jointly optimize both sub-networks. The proposed technique was tested against the E2E Connectionist Temporal Classification (CTC) based model, a combination of both Convolutional Neural Networks (CNNs) and Bidirectional Long Short-term Memory (BiLSTM). The proposed models were trained and evaluated against two large datasets (the Online-KHATT and CHAW). In supervised adaptation, it achieved an absolute Character Error Rate (CER) of up to 1.83% and an absolute Word Error Rate (WER) reduction of 11.71% over the WI model. Additionally, supervised adaptation achieved an absolute CER of up to 0.84% and an absolute WER reduction of 6.77% over the fine-tuned model. In unsupervised adaptation, the proposed method achieved an absolute CER of up to 0.5% absolute and an absolute WER reduction of 1.74% absolute (WER) reduction over the WI. Our experimental results indicate that our proposed supervised writer adaptation can achieve significant improvements in recognition accuracy compared with the baseline models: WI and fine-tuned models.

Transformer-based Models for Arabic Online Handwriting Recognition

Transformer neural networks have increasingly be-come the neural network design of choice, having recently been shown to outperform state-of-the-art end-to-end (E2E) recurrent neural networks (RNNs). Transformers utilize a self-attention mechanism to relate input frames and extract more expressive sequence representations. Transformers also provide parallelism computation and the ability to capture long dependencies in contexts over RNNs. This work introduces a transformer-based model for the online handwriting recognition (OnHWR) task. As the transformer follows encoder-decoder architecture, we investigated the self-attention encoder (SAE) with two different decoders: a self-attention decoder (SAD) and a connectionist temporal classification (CTC) decoder. The proposed models can recognize complete sentences without the need to integrate with external language modules. We tested our proposed mod-els against two Arabic online handwriting datasets: Online-KHATT and CHAW. On evaluation, SAE-SAD architecture per-formed better than SAE-CTC architecture. The SAE-SAD model achieved a 5% character error rate (CER) and an 18%word error rate (WER) against the CHAW dataset, and a 22% CER and a 56% WER against the Online-KHATT dataset. The SAE-SAD model showed significant improvements over existing models of the Arabic OnHWR.

Multi-lingual character handwriting framework based on an integrated deep learning based sequence-to-sequence attention model

Online signals are rich in dynamic features such as trajectory chronology, velocity, pressure and pen up/down movements. Their offline counterparts consist of a set of pixels. Thus, online handwriting recognition accuracy is generally better than offline. In this paper, we propose an original framework for recovering temporal order and pen velocity from offline multi-lingual handwriting. Our framework is based on an integrated sequence-to-sequence attention model. The proposed system involves extracting a hidden representation from an image using a convolutional neural network (CNN) and a bidirectional gated recurrent unit (BGRU), and decoding the encoded vectors to generate dynamic information using a BGRU with temporal attention. We validate our framework using an online recognition system applied to a benchmark Latin, Arabic and Indian On/Off dual-handwriting character database. The performance of the proposed multi-lingual system is demonstrated through a low error rate of point coordinates and high accuracy system rate.

Online handwriting Arabic recognition system using k-nearest neighbors classifier and DCT features

With advances in machine learning techniques, handwriting recognition systems have gained a great deal of importance. Lately, the increasing popularity of handheld computers, digital notebooks, and smartphones give the field of online handwriting recognition more interest. In this paper, we propose an enhanced method for the recognition of Arabic handwriting words using a directions-based segmentation technique and discrete cosine transform (DCT) coefficients as structural features. The main contribution of this research was combining a total of 18 structural features which were extracted by DCT coefficients and using the k-nearest neighbors (KNN) classifier to classify the segmented characters based on the extracted features. A dataset is used to validate the proposed method consisting of 2500 words in total. The obtained average 99.10% accuracy in recognition of handwritten characters shows that the proposed approach, through its multiple phases, is efficient in separating, distinguishing, and classifying Arabic handwritten characters using the KNN classifier. The availability of an online dataset of Arabic handwriting words is the main issue in this field. However, the dataset used will be available for research via the website.

DeepOnKHATT: An End-to-End Arabic Online Handwriting Recognition System

The importance of online handwriting recognition technology has steadily increased in recent years. This importance stems from the rapid increase in the number of handheld devices with digital pens and styluses. In addition, a large number of currently available communication software have been designed to support handwriting boards. During the past decade, most innovation in online handwriting recognition technology was geared towards supporting languages using the Latin alphabet. There has been a lack of sufficient development of Arabic online handwriting recognition (AOHR) systems, especially ones that perform recognition at the sentence level. In this paper, we present DeepOnKHATT, an end-to-end AOHR based on bidirectional long short-term memory and the connectionist temporal classification (BLSTM-CTC). DeepOnKHATT is capable of performing recognition at the sentence level in real-time. We evaluated our system utilizing two open access databases: CHAW and Online-KHATT. Our model achieved a 4.08% character error rate (CER) and a 14.65% word error rate (WER) against the CHAW dataset and a 12.24% CER and a 28.35% WER against the Online-KHATT dataset. A Comparison of the functionality of our proposed model to that of other existing systems showed that DeepOnKHATT had outperformed these systems.

Online handwritten Gurmukhi word recognition using fine-tuned Deep Convolutional Neural Network on offline features

The recognition of online handwriting is a vital application of pattern recognition, which involves the extraction of spatial and temporal information of handwritten patterns, and understanding the handwritten text while writing on the digital surface. Although, online handwriting recognition is a mature but exciting and fast developing field of pattern recognition, the same is not true for many of the Indic scripts. Gurmukhi is one of such popular scripts of India, and online handwriting recognition issues for larger units as words or sentences largely remained unexplored for this script till date. The existing study and first ever attempt for online handwritten Gurmukhi word recognition has relied upon the widely used hidden Markov model. This existing study evaluated against and performed very well in their chosen metrics. But, the available online handwritten Gurmukhi word recognition system could not obtain more than 90% recognition accuracy in data dependent environment too. The present study provided benchmark results for online handwritten Gurmukhi word recognition using deep learning architecture convolutional neural network, and obtained above 97% recognition accuracy in data dependent mode of handwriting. The previous Gurmukhi word recognition system followed the stroke based class labeling approach, whereas the present study has followed the word based class labeling approach. Present Online handwritten Gurmukhi word recognition results are quite satisfactory. Moreover, the proposed architecture can be used to improve the benchmark results of online handwriting recognition of several major Indian scripts. Experimental results demonstrated that the deep learning system achieved great results in Gurmukhi script and outperforms existing results in the literature.

Offline Handwritten Text Recognition Using Deep Learning: A Review

The area of offline handwritten text recognition(OHTR) has been widely researched in the last decades, but it stills an important research problem. The OHTR system has an objective to transform a document image into text data. Compared with online handwriting recognition, the dynamic information about the writing trajectories in OHTR is not available. Many advancements have been proposed in the literature, most notably the application of deep learning methods to OHTR. In this paper, we introduced how this problem has been handled in the past few decades, analyze the latest advancements and the potential directions for future research in this field.

Post-processing algorithms for the formation of online handwritten Gurmukhi character/akshara

Formation of Gurmukhi character/akshara from the recognized strokes in online handwriting recognition systems is a challenging task. In this paper, the task of character and akshara formation in an unconstrained environment have been addressed. After the recognition of online handwritten strokes the Gurmukhi akshara is formed using a hybrid approach. Two classifiers, namely, Support Vector Machine (SVM) and Recurrent Neural Network (RNN) have been experimented in this study. The classifier, yielded the maximum cross-validation accuracy has been utilized for stroke recognition. A total of 52,500 word samples have been collected from 175 writers in order to train the classifiers. Three post processing algorithms have been proposed in this article for improving the character and akshara recognition accuracy. The proposed methodology when tested on a dataset of 21,500 aksharas, written by 50 new writers, achieved average the accuracy rate of 97.1% and 87.1% for base character and akshara recognition, respectively.

CTRL –CapTuRedLight: a novel feature descriptor for online Assamese numeral recognition

Online handwriting recognition (OHR) has gained major research interest not just due to the enormous technological advancement in recent years, but also the easy availability of the various electronic devices. This digital revolution is opening up a new dimension in every passing day to the regional and low resource languages with these languages get noticed by the researchers. In this paper, we have targeted a low resource language, Assamese, which is mainly spoken in the eastern region of India. We have proposed a novel and efficient feature vector for recognition of online handwritten Assamese numeral images. Our feature vector has been conceptualized based on the properties of light rays emerging out from a point source. Here we consider that there are multiple hypothetical light emerging sources in a sample numeral image. The amount of light fenced by the image is quantified and considered as a feature. The idea of using point light source to estimate the shape of online handwritten numerals is completely new and efficient. Impressive recognition accuracy is obtained on application of the feature vector on a standard online handwritten Assamese numeral database and it outnumbers some popular and standard feature descriptors, available in the literature. The source code of this work can be found in the following github link: https://github.com/ghoshsoulib/CTRL-Assamese-Digit-Recognition .

The OnHW Dataset

This paper presents a handwriting recognition (HWR) system that deals with online character recognition in real-time. Our sensor-enhanced ballpoint pen delivers sensor data streams from triaxial acceleration, gyroscope, magnetometer and force signals at 100 Hz. As most existing datasets do not meet the requirements of online handwriting recognition and as they have been collected using specific equipment under constrained conditions, we propose a novel online handwriting dataset acquired from 119 writers consisting of 31,275 uppercase and lowercase English alphabet character recordings (52 classes) as part of the UbiComp 2020 Time Series Classification Challenge. Our novel OnHW-chars dataset allows for the evaluations of uppercase, lowercase and combined classification tasks, on both writer-dependent (WD) and writer-independent (WI) classes and we show that properly tuned machine learning pipelines as well as deep learning classifiers (such as CNNs, LSTMs, and BiLSTMs) yield accuracies up to 90 % for the WD task and 83 % for the WI task for uppercase characters. Our baseline implementations together with the rich and publicly available OnHW dataset serve as a baseline for future research in that area.

Online handwriting recognition systems for Indic and non-Indic scripts: a review

Handwriting recognition is one of the challenging tasks in the area of pattern recognition and machine learning. Handwriting recognition has two flavors, namely, Offline Handwriting Recognition and Online Handwriting Recognition. Though, saturation level has been achieved in machine printed (Offline) character recognition. Presently, due to dramatical development in IT sector, touch-based devices are available in the market with efficient processing capabilities. With this revolution, research in the area of handwriting recognition has become more popular in real-time (Online) mode. In this paper, a comprehensive review has been reported for online handwriting recognition of non-Indic and Indic scripts. The six non-Indic-scripts and eight Indic script namely, Arabic, Chinese, Japanese, Persian, Roman, Thai, and, Assamese, Bangla, Devanagari, Gurmukhi, Kannada, Malayalam, Tamil, Telugu, respectively have been considered in this article. This study comprises introduction of online handwriting recognition process, various challenges, motivations, feature extraction, and classification methodologies, used for recognizing the various scripting languages. Moreover, an effort has been made to provide the list of publicly available online handwritten dataset for various scripting languages. This study also provides the recognition and beneficial assistance to the novice researchers in field of handwriting recognition by providing a nut shell studies of various feature extraction strategies and classification techniques, used for the recognition of both Indic and non-Indic scripts.

A self controlled RDP approach for feature extraction in online handwriting recognition using deep learning

The identification of accurate features is the initial task for benchmarked handwriting recognition. For handwriting recognition, the objective of feature computation is to find those characteristics of a handwritten stroke that depict the class of a stroke and make it separable from the rest of the stroke classes. The present study proposes a feature extraction technique for online handwritten strokes based on a self controlled Ramer-Douglas-Peucker (RDP) algorithm. This novel approach prepares a smaller length feature vector for different shaped online handwritten strokes without preprocessing and without any control parameter to RDP. Thus, it also overcomes the shortcomings of the traditional chain code based feature extraction approach that requires preprocessing of data, and the original RDP algorithm that requires a control parameter as an input to RDP. We further propose a deep learning network of 1-dimensional convolutional neural networks (Conv1Ds) for recognition, which trains in few minutes due to the smaller dimension of the convolution combined with smaller length feature vectors. The proposed approach can be applied to different scripts and different writing styles. The key aim of the present study is to provide a script independent feature extraction technique that is well suited for smaller devices. It improves the recognition over the best reported accuracy in the literature which was achieved using hidden Markov models with directional features, from 87.67% to 95.61% on a Gurmukhi dataset. For Unipen online handwriting datasets the results are at par with the literature.

An Unconstrained and Effective Approach of Script Identification for Online Bilingual Handwritten Text

A number of monolingual online handwriting recognition (OHR) systems were proposed by researchers for languages like English, Chinese, Japanese, Hindi, Punjabi, Tamil, Bangla, Arabic and many more. But very few researchers have worked on bilingual or multilingual OHR systems, as the challenging part of these systems is the identification of scripts during recognition of intermixed words. In this paper, a bilingual OHR system is proposed for the mixed text containing words, written by using two scripts Gurmukhi (for Punjabi language) and Roman (for English language). For converting handwritten intermixed text to digital text, two steps have been implemented: first, to identify the script of segmented handwritten word and second, to run the respective script recognition engine. Multilayered perceptron neural network classifier along with directional code feature has been implemented to identify the script of segmented handwritten word. Considerable results have been observed for the script identification process.