Text Entry System Research Articles

Imperfect Surrogate Users: Understanding Performance Implications of Augmentative and Alternative Communication Systems through Bounded Rationality, Human Error, and Interruption Modeling

Nonspeaking individuals with motor disabilities frequently rely on augmentative and alternative communication (AAC) systems that allow users to communicate through a text entry interface coupled with a speech synthesizer. Such systems are notoriously difficult to evaluate with end-users. However, recent research has proposed envelope analysis as a method to estimate text entry rates and keystroke savings by simulating the interaction of an expert surrogate user entering sentences on a conceptual word-predictive text entry system. While only a part of the evaluation process of an AAC system, this method enables AAC designers to benefit from quantitative insights early on in the design process. This paper extends prior work by (1) demonstrating how to incorporate natural language generation, such as sentence generation, in such analyses; (2) presenting a model of an imperfect surrogate user that incorporates bounded rationality, human error, and interruptions to provide a more realistic simulation of text entry behavior; and (3) demonstrating how to estimate model parameters by observing users' actual typing behavior. We validate the model with data collected from eight participants using an AAC system on a touchscreen.

MyoKey: Inertial Motion Sensing and Gesture-Based QWERTY Keyboard for Extended Realities

Usability challenges and social acceptance of textual input in a context of extended realities (XR) motivate the research of novel input modalities. We investigate the fusion of inertial measurement unit (IMU) control and surface electromyography (sEMG) gesture recognition applied to text entry using a QWERTY-layout virtual keyboard. We design, implement and evaluate our proposed multi-modal solution, MyoBoard, which is an IMU-driven and EMG-based text entry system based on the QWERTY keyboard layout for XR headsets. The user can select characters with a combination of arm movements and hand gestures. MyoBoard employs a light-weight convolutional neural network classifier that can be deployed on a mobile device with insignificant inference time. We demonstrate the practicality of interruption-free text entry in mobile scenarios with MyoBoard, by recruiting 12 participants and by testing three sets of grasp micro-gestures in three scenarios - empty hand text input, input holding a pen, and a large object, such as an umbrella. With MyoBoard, users achieve an average text entry rate of 9.33 words per minute (WPM), 8.76 WPM, and 8.35 WPM for the freehand, pen, and umbrella conditions, respectively.

CrowbarLimbs: A Fatigue-Reducing Virtual Reality Text Entry Metaphor.

Text entry remains challenging in virtual environments, where users may quickly experience physical fatigue in some body parts using existing methods. In this paper, we propose "CrowbarLimbs," a novel virtual reality (VR) text entry metaphor with two deformable extended virtual limbs. By using a crowbar-like metaphor and placing the virtual keyboard at a user-preferred location based on the user's physical stature, our method can assist the user in placing their hands and arms in a comfortable posture, thus effectively reducing the physical fatigue in various body parts, such as hands, wrists, and elbows. In an initial user study, we found that CrowbarLimbs achieved text entry speed, accuracy, and system usability comparable to those of previous VR typing methods. To investigate the proposed metaphor in more depth, we further conducted two additional user studies to explore the ergonomically user-friendly shapes of CrowbarLimbs and virtual keyboard locations. The experimental results indicate that the shapes of CrowbarLimbs have significant effects on the fatigue ratings in various body parts and text entry speed. Furthermore, placing the virtual keyboard near the user and at half their height can lead to a satisfactory text entry rate of 28.37 words per minute.

What is supposed to help, actually hinders: about the impact of the GUI dynamics on the efficiency of gaze text entry

The research aimed to test the influence on text entry efficiency of supporting features in gaze text entry systems (aka gaze keyboards), leading to dynamic changes in key layout and size. We examined three gaze-controlled on-screen keyboards, that differ in the degree of support, which translates into the amount of change that occurs in the arrangement of the keys and their size - three degrees of dynamism of keyboards’ graphical user interface. We also attempted to examine the importance of user attention while using dynamic interface and its impact on text input performance.

EchoWrite 2.0: A Lightweight Zero-Shot Text-Entry System Based on Acoustics

Limited by size, shape, and other factors, it is rather inconvenient to interact with new smart devices by traditional methods. Acoustic-based methods following a machine learning approach have been put forward to resolve this problem in previous works. But they possess limitations of heavy training overhead, low performance for unseen users, and intensive computation cost. Following our previous work in this area, we further overcome shortcomings of existing work and propose a lightweight and zero-shot text-entry system for unseen users based on acoustic sensing. The key novelty of this work is proposing a new model training strategy including dataset construction and augmentation methods to effectively enhance generalization ability of a simple learning model with as few training data as possible, based on our insight into the problem. We design and implement a real-time Android application system called <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EchoWrite 2.0</monospace> to validate our idea with extensive experiments. Results show that <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EchoWrite 2.0</monospace> can recognize digits, English letters, and words with an accuracy of 85.3%, 73.2%, and 96.9%, respectively, for unseen users without providing any data to the learning model. The comparison with related work in different aspects shows overall superiority of <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EchoWrite 2.0</monospace> .

Write, Attend and Spell

Text entry on a smartwatch is challenging due to its small form factor. Handwriting recognition using the built-in sensors of the watch (motion sensors, microphones, etc.) provides an efficient and natural solution to deal with this issue. However, prior works mainly focus on individual letter recognition rather than word recognition. Therefore, they need users to pause between adjacent letters for segmentation, which is counter-intuitive and significantly decreases the input speed. In this paper, we present 'Write, Attend and Spell' (WriteAS), a word-level text-entry system which enables free-style handwriting recognition using the motion signals of the smartwatch. First, we design a multimodal convolutional neural network (CNN) to abstract motion features across modalities. After that, a stacked dilated convolutional network with an encoder-decoder network is applied to get around letter segmentation and output words in an end-to-end way. More importantly, we leverage a multi-task sequence learning method to enable handwriting recognition in a streaming way. We construct the first sequence-to-sequence handwriting dataset using smartwatch. WriteAS can yield 9.3% character error rate (CER) on 250 words for new users and 3.8% CER for words unseen in the training set. In addition, WriteAS can handle various writing conditions very well. Given the promising performance, we envision that WriteAS can be a fast and accurate input tool for smartwatch.

WITHDRAWN: Implementation of TTS with tactile mobile based assistive device for visually impaired people

Abstract The creation of interactive bit devices poses a brand-new set of text-entry challenges. Once considering blind folks, these are additional evident, as a result of bit screens lack the tactile feedback, they're accustomed once human action with phones. The solutions offered to permit non-visual text input resort to a broad vary of goals, complicated interaction strategies or unfamiliar layouts. We tend to propose Braille sort, a Braille language-based, text-entry system. In favor of a simplified single-finger interface, Braille sort avoids multi-touch gestures, that includes few and enormous targets. The intention of this project is to bridge a spot among the blind and even the scientific development in the world of telecommunications. We’ve got deter-mined to style a system wherever data is distributed via GSM that is any sent to Braille pad. Once the message is received, a buzzer is ready to alert that the visually handicapped person will reach the Braille pad to induce the message. As per the received message, individual switch is pushed victimization servo motor and therefore the message received may be understood by sensing the push buttons. Associate degree automaton app is meant, wherever they received text is reborn to voice message so it becomes additional convenient for the visually handicapped person to listen to and perceive the received message by victimization Wi-Fi.

A Low Latency On-Body Typing System through Single Vibration Sensor

Nowadays, smart wristbands have become one of the most prevailing wearable devices, as they are small and portable. However, due to the limited size of the touch screens, smart wristbands typically have poor interactive experience. There are a few works appropriating the human body as a surface to type on. Yet, by using multiple sensors at high sampling rates, they are not portable and are energy-consuming in practice. To break this stalemate, we proposed a portable, cost efficient text-entry system, termed ViType, which first leverages a single small form factor sensor to achieve a practical user input with much lower sampling rates. To enhance the input accuracy with less vibration information introduced by lower sampling rates, ViType designs a set of novel mechanisms, including a fine-grained feature extraction to process the vibration signals, and a runtime calibration and adaptation scheme to recover from the error due to temporal instability. Extensive experiments have been conducted on 30 human subjects. The results demonstrate that ViType is robust against various confounding factors. The average recognition accuracy is 95 percent with an initial training sample size of 20 for each key. The accuracy is 1.54 times higher than the state-of-the-art on-body typing system. Furthermore, when turning on the runtime calibration and adaptation system to update and enlarge the training sample size, the accuracy can reach around 98 percent on average during one month.

Learning to Recognize Handwriting Input with Acoustic Features

For mobile or wearable devices with a small touchscreen, handwriting input (instead of typing on the touchscreen) is highly desirable for efficient human-computer interaction. Previous passive acoustic-based handwriting solutions mainly focus on print-style capital input, which is inconsistent with people's daily habits and thus causes inconvenience. In this paper, we propose WritingRecorder, a novel universal text entry system that enables free-style lowercase handwriting recognition. WritingRecorder leverages the built-in microphone of the smartphones to record the handwritten sound, and then designs an adaptive segmentation method to detect letter fragments in real-time from the recorded sound. Then we design a neural network named Inception-LSTM to extract the hidden and unique acoustic pattern associated with the writing trajectory of each letter and thus classify each letter. Moreover, we adopt a word selection method based on language model, so as to recognize legislate words from all possible letter combinations. We implement WritingRecorder as an APP on mobile phones and conduct the extensive experimental evaluation. The results demonstrate that WritingRecorder works in real-time and can achieve 93.2% accuracy even for new users without collecting and training on their handwriting samples, under a series of practical scenarios.

From seen to unseen: Designing keyboard-less interfaces for text entry on the constrained screen real estate of Augmented Reality headsets

Text input is a very challenging task in the constrained screen real-estate of Augmented Reality headsets. Typical keyboards spread over multiple lines and occupy a significant portion of the screen. In this article, we explore the feasibility of single-line text entry systems for smartglasses. We first design FITE, a dynamic keyboard where the characters are positioned depending on their probability within the current input. However, the dynamic layout leads to mediocre text input and low accuracy. We then introduce HIBEY, a fixed 1-line solution that further decreases the screen real-estate usage by hiding the layout. Despite its hidden layout, HIBEY surprisingly performs much better than FITE, and achieves a mean text entry rate of 9.95 words per minute (WPM) with 96.06% accuracy, which is comparable to other state-of-the-art approaches. After 8 days, participants achieve an average of 13.19 WPM. In addition, HIBEY only occupies 13.14% of the screen real estate at the edge region, which is 62.80% smaller than the default keyboard layout on Microsoft Hololens.

Distance-based weighted sparse representation to classify motor imagery EEG signals for BCI applications

Motor imagery (MI) based brain-computer interface systems (BCIs) are highly in need for a large number of real-time applications such as hands and touch-free text entry system, movement of a wheelchair, movement of a cursor, prosthetic arm movement, virtual reality systems, etc. In recent years, sparse representation-based classification (SRC) is a growing technique and has been a successful technique on classifying MI-based Electroencephalography (EEG) signals. To further boost the proficiency of SRC technique, in this paper, a weighted SRC (WSRC) has been proposed for classifying MI signals. In WSRC approach, a weighted dictionary has been constructed according to the dissimilarity information between a test data and training samples. Then for the given test data, the sparse coefficients are computed over the weighted dictionary using l0-minimization problem. The sparse solution obtained using WSRC gives discriminative information and as a consequence, WSRC proves to be superior for MI-based EEG classification. The experimental results substantiate that WSRC is more efficient and accurate than SRC.

Quadmetric Optimized Thumb-to-Finger Interaction for Force Assisted One-Handed Text Entry on Mobile Headsets

Augmented reality head-worn computers often feature small-sized touch interfaces that complicate interaction with content, provide insufficient space for comfortable text input, and can be awkward to use in social situations. This paper presents a novel one-handed thumb-to-finger text entry solution for augmented reality head-worn computers. We design a glove composed of 12 force-sensitive nodes featuring an ambiguous keyboard layout. We first explore the viability of force disambiguation to evaluate the force division within the force spectrum. We select a 3-level force division as it allows to considerably reduce the number of keys while featuring a high (83.9%) accuracy. Following this pilot study, we map the 26 English characters onto the 9 nodes located on the index, middle and ring fingers in a 3-3-3 configuration, and attribute the space, enter and backspace keys to the remaining three nodes. We consider text entry performance as a quadmetric optimization problem considering the following criteria: goodness of character pairs, layout similarity to the QWERTY keyboard, easiness of force interaction, and comfort level of thumb reach. The resulting layout strikes a balance between performance and usability. We finally evaluate the quadmetric optimized layout over 6 sessions with 12 participants. The participants achieve an average text entry rate of 6.47 WPM with 6.85% error rate in the final session, which is significantly faster than existing thumb-to-finger solutions. In addition, our one-handed text entry system enhances the user mobility compared to other state-of-the-art solutions by freeing one hand, while allowing the user to direct his visual attention to other activities.

CamType: assistive text entry using gaze with an off-the-shelf webcam

As modern assistive technology advances, eye-based text entry systems have been developed to help a subset of physically challenged people to improve their communication ability. However, speed of text entry in early eye-typing system tends to be relatively slow due to dwell time. Recently, dwell-free methods have been proposed which outperform the dwell-based systems in terms of speed and resilience, but the extra eye-tracking device is still an indispensable equipment. In this article, we propose a prototype of eye-typing system using an off-the-shelf webcam without the extra eye tracker, in which the appearance-based method is proposed to estimate people’s gaze coordinates on the screen based on the frontal face images captured by the webcam. We also investigate some critical issues of the appearance-based method, which helps to improve the estimation accuracy and reduce computing complexity in practice. The performance evaluation shows that eye typing with webcam using the proposed method is comparable to the eye tracker under a small degree of head movement.

A New PC-Based Text Entry System Based on EOG Coding

Some disadvantages of optical eye tracking systems have increased the interest to EOG (Electrooculography) based Human Computer Interaction (HCI). However, text entry attempts using EOG have been slower than expected because the eyes should move several times for entering a character. In order to improve the writing speed and accuracy of EOG based text entry, a new method based on the coding of eye movements has been suggested in this study. In addition, a real time EOG based HCI system has developed to implement the method. In our method all characters have been encoded by single saccades in 8 directions and different dwell time. In order to standardize dwell times and facilitate the coding process, computer assisted voice guidance was used. A number of experiments have been conducted to examine the effectiveness of the proposed method and system. At the end of the fifth trials, an experienced user was able to write at average 13.2 wpm (5 letters = 1 word) with 100% accuracy using the developed system. The results of our experiments have shown that text entry with the eye can be done quickly and efficiently with the proposed method and system.

Efficient eye typing with 9-direction gaze estimation

Vision based text entry systems aim to help disabled people achieve text communication using eye movement. Most previous methods have employed an existing eye tracker to predict gaze direction and design an input method based upon that. However, these methods can result in eye tracking quality becoming easily affected by various factors and lengthy amounts of time for calibration. Our paper presents a novel efficient gaze based text input method, which has the advantage of low cost and robustness. Users can type in words by looking at an on-screen keyboard and blinking. Rather than estimate gaze angles directly to track eyes, we introduce a method that divides the human gaze into nine directions. This method can effectively improve the accuracy of making a selection by gaze and blinks. We build a Convolutional Neural Network (CNN) model for 9-direction gaze estimation. On the basis of the 9-direction gaze, we use a nine-key T9 input method which is widely used in candy bar phones. Bar phones were very popular in the world decades ago and have cultivated strong user habits and language models. To train a robust gaze estimator, we created a large-scale dataset with images of eyes sourced from 25 people. According to the results from our experiments, our CNN model is able to accurately estimate different people's gaze under various lighting conditions by different devices. In considering disable people's needs, we removed the complex calibration process. The input methods can run in screen mode and portable off-screen mode. Moreover, The datasets used in our experiments are made available to the community to allow further experimentation.

Passive Haptic Training to Improve Speed and Performance on a Keypad

Learning text entry systems is challenging, yet necessary. Many layouts and keyboards exist, but they rely on laborious learning techniques. Passive haptic learning (PHL) has already demonstrated some benefit for learning the Braille text entry system. Could this computing-enabled technique be used to improve desktop keyboard typing skills? It is unknown whether passive haptic training can improve speed on a motor task (as opposed to initial learning). We use a randomized numeric keypad to examine users’ typing performance with or without passive haptic training. When users were prevented from looking at the keyboard, the PHL group demonstrated consistent accuracy (-0.011 KSPC) while those in the control group greatly increased their error (+1.26 KSPC on average). This result is consistent with the finding that PHL users looked significantly less at the keyboard. In a second, longer study, users exposed to PHL were found to significantly improve their typing speed (mean increase of 11 WPM) versus control (mean increase of 2.2 WPM).

Text Entry System for Alternative Speech Communications

Text Entry System for Alternative Speech Communications

A Japanese Text Entry System for One-handed Index Finger Using Back-of-device Interaction

A Japanese Text Entry System for One-handed Index Finger Using Back-of-device Interaction

A brain–computer interface for the Dasher alternative text entry system

To investigate whether the alternative text entry system, Dasher, is useful to physically and intellectually disabled students when controlled by a brain---computer interface (BCI) a new software tool was developed to allow subjects to type words onto a computer screen via Dasher using their thoughts. A case study approach was adopted. Subjects were selected based on their suitability for the experiment, and the potential benefit to them of this system, by their head teacher. Subjects entered literacy level-matched phrases onto a computer using QWERTY keyboard, Dasher-mouse and Dasher-BCI. A researcher recorded qualitative and quantitative data, including characters entered per minute and their system preferences. Informed written consent was given for seven subjects to participate (aged 14---19 years, five male, with a range of physical and intellectual disabilities). After a short training period, all subjects had some degree of control over the Dasher-BCI system. With regard to typing speed, Dasher-BCI performed relatively poorly (3.9 ± 1.5 characters per minute), and QWERTY keyboard performed the best (31.9 ± 21.9 characters per minute). Dasher-BCI was the most preferred method. Areas of weakness in Dasher and the BCI hardware were highlighted and suggestions for improvement given. BCI-based text entry is not yet ready to compete with more established methods for students with combined cognitive and physical disabilities. Although underpowered, this study suggests that for people whose predominant disability is physical (cerebral palsy), BCI technology shows great potential as a viable text entry alternative. Suggestions for further research are discussed.

Text messaging language: A comparison of undergraduates' naturalistic textism use in four consecutive cohorts

Text messaging is by now a ubiquitous form of computer-mediated communication, especially among university students. An increasing body of research has examined the written language used in text messaging, but little or no data exist regarding how the use of non-conventional spellings, or ‘textisms’, may change over time as people adapt their written language to fit with advances in communication technology. This study reports on naturalistic messages gathered in four consecutive years from four cohorts of Australian first-year undergraduates. Textism use (in particular, ‘contractives’ such as <wht> for <what>) decreased during this time period, in step with the use of newer text-entry systems (e.g., predictive software). Compared to males, females used significantly more textisms in general, and more expressive textisms (letters added for expressive intent, such as <whaaat?!!> for <what>) in particular. Overall, our results suggest that textism use may continue to decrease as abbreviated language loses its practicality in a landscape of improved technology.