CameraKeyboard: A Novel Interaction Technique for Text Entry Through Smartphone Cameras

Abstract

We present CameraKeyboard, a text entry technique that uses smartphone cameras to extract and digitalise text from physical sources such as business cards and identification documents. After taking a picture of the text of interest, the smartphone recognises the text through OCR technologies (specifically, Google Cloud Vision API) and organises it in the same way in which it is displayed in the original source. Next, users can select the required text - which is split into lines, and further split into words - by tapping on it. CameraKeyboard is designed to easily complement the standard keyboard when users need to digitalise text. In fact, the camera is integrated directly in the standard mobile QWERTY keyboard and can be accessed by tapping a button. CameraKeyboard is general-purpose like any other smartphone keyboard: it works with any application (e.g., Gmail, Whatsapp) and is able to extract text from any kind of source. We evaluated CameraKeyboard through a user study with 18 participants who carried out four different text entry tasks, and compared it to a standard smartphone keyboard (Google Keyboard) and a standard (physical) desktop keyboard. Results show that CameraKeyboard is the faster one in most cases.