Segmentation and recognition of text written in 3D using Leap motion interface

Abstract

In this paper, we present a word extraction and recognition methodology from online cursive handwritten text-lines recorded by Leap motion controller The online text, drawn by 3D gesture in air, is distinct from usual online pen-based strokes. The 3D gestures are recorded in air, hence they produce often non-uniform text style and jitter-effect while writing. Also, due to the constraint of writing in air, the pause of stroke-flow between words is missing. Instead all words and lines are connected by a continuous stroke. In this paper, we have used a simple but effective heuristic to segment words written in air. Here, we propose a segmentation methodology of continuous 3D strokes into text-lines and words. Separation of text lines is achieved by heuristically finding the large gap-information between end and start-positions of successive text lines. Word segmentation is characterized in our system as a two class problem. In the next phase, we have used Hidden Markov Model-based approach to recognize these segmented words. Our experimental validation with a large dataset consisting with 320 sentences reveals that the proposed heuristic based word segmentation algorithm performs with accuracy as high as 80.3%c and an accuracy of 77.6% has been recorded by HMM-based word recognition when these segmented words are fed to HMM. The results show that the framework is efficient even with cluttered gestures.