Recognition of Handwritten Mathematical Characters on Whiteboards Using Colour Images

Abstract

Automatic handwriting recognition has enjoyed significant improvements in the past decades. In particular, online recognition of mathematical formulas has seen a number of important advancements both for pen input devices as well as for smart boards. However, in reality most mathematics is still taught and developed on regular whiteboards and that the offline recognition still remains a challenging task. In this paper we are therefore concerned with the offline recognition of handwritten notes on whiteboards, presenting a novel way of transforming offline data via image analysis into equivalent online data. We use trajectory recovery techniques and statistical classification on high quality colour images to extract information on the strokes composing a character, such as start or end points and stroke direction. This data is then appropriately prepared and passed to an online character recogniser specialising on mathematical characters for the actual recognition task. We demonstrate the effectiveness of our new technique with experiments on a collection of 1000 whiteboard images of different mathematical symbols, Latin and Greek characters that have been obtained from a variety of writers using different types of pens.