Virtual Keyboard Research Articles

Boamente: A Natural Language Processing-Based Digital Phenotyping Tool for Smart Monitoring of Suicidal Ideation.

People at risk of suicide tend to be isolated and cannot share their thoughts. For this reason, suicidal ideation monitoring becomes a hard task. Therefore, people at risk of suicide need to be monitored in a manner capable of identifying if and when they have a suicidal ideation, enabling professionals to perform timely interventions. This study aimed to develop the Boamente tool, a solution that collects textual data from users’ smartphones and identifies the existence of suicidal ideation. The solution has a virtual keyboard mobile application that passively collects user texts and sends them to a web platform to be processed. The platform classifies texts using natural language processing and a deep learning model to recognize suicidal ideation, and the results are presented to mental health professionals in dashboards. Text classification for sentiment analysis was implemented with different machine/deep learning algorithms. A validation study was conducted to identify the model with the best performance results. The BERTimbau Large model performed better, reaching a recall of 0.953 (accuracy: 0.955; precision: 0.961; F-score: 0.954; AUC: 0.954). The proposed tool demonstrated an ability to identify suicidal ideation from user texts, which enabled it to be experimented with in studies with professionals and their patients.

A comprehensive benchmark dataset for SSVEP-based hybrid BCI

The use of hybrid brain–computer interface (HBCI) systems have received much attention in recent years. However, there are few published datasets for HBCI systems. This paper provides a benchmark dataset for these systems. The provided dataset consist of data corresponding to three speller systems, including SSVEP-based BCI system and HBCI systems based on SSVEP-EMG and SSVEP-EOG. These spellers have been implemented using electroencephalogram (EEG) and other biosignals, including electromyogram (EMG) and electrooculogram (EOG). The virtual keyboard of spellers was composed of 9 visual flickers with stimulation frequencies ranged from 5.58 Hz to 11.11 Hz. The stimuli presentation was controlled by the psychophysics toolbox of Matlab. We conducted three experiments, using three speller systems under the similar situation. Ten subjects participated in a copy-spelling task experiment. For each system, the data included ten sessions corresponding to 10 times, which the subject spells the determined phrase. These HBCI systems were validated using NASA-TLX workload index and performance evaluation parameters, including the accuracy and the information transfer rate. Results showed that combining two control signals only when signals are derived from one organ of the body, might increase the workload. These datasets can be used as a comprehensive dataset to evaluate the performance of different HBCI systems. They are freely available from Http://maleki.semnan.ac.ir/.

Scoring Essays on an iPad Versus a Desktop Computer: An Exploratory Study

Recognizing the appealing features of a tablet (e.g., an iPad), including size, mobility, touch screen display, and virtual keyboard, more educational professionals are moving away from larger laptop and desktop computers and turning to the iPad for their daily work, such as reading and writing. Following the results of a recent survey of individuals who serve as ETS raters, more than 40% reported that they would prefer to use an iPad or other type of tablet to score essays. However, iPad‐based essay scoring could affect scoring accuracy and scoring time because the smaller screen and other features of an iPad may also affect raters' reading comprehension and score assigning processes. To address this issue, we invited 10 experienced raters to score holistically 40 essays for a graduate admission test using a desktop computer and an iPad following a counterbalanced design. We compared the raters' scores against the criterion scores and analyzed scoring times, scoring behaviors, and raters' answers to a structured interview after the scoring experiment. The results reveal no obvious differences between the two devices in the scoring accuracy or average scoring time per essay, which suggests that scoring on an iPad may not reduce scoring quality or scoring productivity for essays that are holistically scored as compared to scoring the essays on a desktop computer. We also found a few iPad‐specific issues that raters reported, including issues associated with the invisible scrolling bar and the extra scrolling needed to reach the score‐assignment panel, difficulty navigating between the prompt and the essay response, and oversensitivity of the touch screen.

PressTapFlick: Exploring a gaze and foot-based multimodal approach to gaze typing

Text entry is extremely difficult or sometimes impossible in the scenarios of situationally-induced impairments and disabilities, and for individuals with motor impairments (physical impairments and disabilities) by birth or due to an injury. As a remedy, many rely on gaze typing with dwell-based selection as it allows for hands-free text entry. However, dwell-based gaze typing could be limited by usability issues, reduced typing speed, high error rate, steep learning curve, and visual fatigue with prolonged usage. Addressing these issues is crucial for improving the usability and performance of gaze typing.In our work, we present a dwell-free, multimodal approach to gaze typing where the gaze input is supplemented with a foot input modality. Our combined gaze and foot-based typing system comprises of an enhanced virtual QWERTY keyboard (VKB), and a footwear augmenting wearable device that provides the foot input. In this multi-modal setup, the user points her gaze at the desired character, and selects it with the foot input. We further investigated two approaches to foot-based selection, a foot gesture-based selection and a foot press-based selection, which are compared against the standard dwell-based selection.We evaluated our gaze typing system through a comparative study involving three experiments (51 participants), where each experiment used one of the three target selection methods, and had 17 participants in it. In the first experiment the participants used dwell-based selection, second, foot gesture-based selection, and third, foot press-based selection for gaze typing. We found that with dwell-based selection the highest mean typing speed of 11.65 WPM (max 14.83 WPM) was achieved when using a dwell time of 400 ms. Similarly, among foot-based selection methods the highest mean typing speed of 14.98 WPM (max 18.18 WPM) was achieved with foot press-based selection. Furthermore, ANOVA tests revealed that the difference in the typing speeds between the three selection methods is significant, however, no significant difference was found in the error rate.Overall, based on both typing performance and qualitative feedback the results suggest that gaze and foot-based typing is convenient, easy to learn, and addresses the usability issues associated with dwell-based typing. Furthermore, toe tapping is the most preferred foot gesture of all the four gestures (toe tapping, heel tapping, right flick and left flick) we used in the study. Also, we found that when using foot-based selection users quickly develop a rhythm in focusing at a character with gaze and selecting it with the foot, and this familiarity reduces the errors significantly. We believe, our findings would encourage further research in leveraging a supplemental foot input in gaze typing, or in general, would assist in the development of rich foot-based interactions.

Gesture objects detection and tracking for virtual text entry keyboard interface

Gesture objects detection and tracking for virtual text entry keyboard interface

A comparative analysis of two immersive virtual reality systems in the integration and visualization of natural hand interaction

It is generally accepted that the use of natural interaction provides a positive impact in Virtual Reality (VR) applications. Therefore, it is important to understand what is the best way to integrate and visualize this feature in VR. For this reason, this paper presents a comparative study of the integration of natural hand interaction in two immersive VR systems: a Cave Audio Visual Experience (CAVE) system –where users’ real hands are visible– and a non-see-through Head-Mounted Display (HMD) system –where only a virtual representation of the hands is possible–. In order to test the suitability of using this type of interaction in a CAVE and compare it to an HMD, we raise six research questions related to task performance, usability and perception differences regarding natural hand interaction with these two systems. To answer these questions, we designed an experiment where users have to complete a pick-and-place task with virtual balls and a text-typing task with virtual keyboards. In both systems, the same tracking technology, based on a Leap Motion device, was used. To the best of our knowledge this is the first academic work addressing a comparison of this type. Objective and subjective data were collected during the experiments. The results show that the HMD has a performance, preference and usability advantage over the CAVE with respect to the integration of natural hand interaction. Nevertheless, the results also show that the CAVE system can be, as well, successfully used in combination with an optical hand tracking device.

Creating and Augmenting Keyboards for Extended Reality with the K eyboard A ugmentation T oolkit

This article discusses the Keyboard Augmentation Toolkit (KAT), which supports the creation of virtual keyboards that can be used both for standalone input (e.g., for mid-air text entry) and to augment physically tracked keyboards/surfaces in mixed reality. In a user study, we firstly examine the impact and pitfalls of visualising shortcuts on a tracked physical keyboard, exploring the utility of virtual per-keycap displays. Supported by this and other recent developments in XR keyboard research, we then describe the design, development, and evaluation-by-demonstration of KAT. KAT simplifies the creation of virtual keyboards (optionally bound to a tracked physical keyboard) that support enhanced display —2D/3D per-key content that conforms to the virtual key bounds; enhanced interactivity —supporting extensible per-key states such as tap, dwell, touch, swipe; flexible keyboard mappings that can encapsulate groups of interaction and display elements, e.g., enabling application-dependent interactions; and flexible layouts —allowing the virtual keyboard to merge with and augment a physical keyboard, or switch to an alternate layout (e.g., mid-air) based on need. Through these features, KAT will assist researchers in the prototyping, creation and replication of XR keyboard experiences, fundamentally altering the keyboard’s form and function.

Tracking and Classification of Head Movement for Augmentative and Alternative Communication Systems.

The use of assistive technologies can mitigate or reduce the challenges faced by individuals with motor disabilities to use computer systems. However, those who feature severe involuntary movements often have fewer options at hand. This work describes an application that can recognize the user’s head using a conventional webcam, track its motion, model the desired functional movement, and recognize it to enable the use of a virtual keyboard. The proposed classifier features a flexible structure and may be personalized for different user need. Experimental results obtained with participants with no neurological disorders have shown that classifiers based on Hidden Markov Models provided similar or better performance than a classifier based on position threshold. However, motion segmentation and interpretation modules were sensitive to involuntary movements featured by participants with cerebral palsy that took part in the study.

New Textual Authentication Method to Resistant Shoulder-Surfing Attack

Using textual passwords suffer from the balance between security and usability. Password policies are usually adopted by system administrators to force users to choose strong passwords. However, users often use a simple password to make it easy to remember, which reduces the password strength and make it vulnerable to information security threats. When users enter their passwords in public places like airports or cafes, they become exposed to shoulder surfing attacks which are considered as a kind of social engineering. With a little effort, an attacker can capture a password by recording the individual’s authentication session or by direct observation. To overcome this vulnerability, we propose a new textual-password approach that uses camouflage characters and a virtual keyboard which leads to generating strong and easy to remember passwords. The perspective of usability and security was evaluated by experimental studies conducted with 65 users and then compared with recent studies. The results showed that the proposed technique has the lowest shoulder surfing success rate with just 3.63% with reasonable usability.

MmKey: Universal Virtual Keyboard Using A Single Millimeter-Wave Radio

Keyboard acts as one of the most commonly used mediums for human–computer interaction. Today, massive Internet-of-Things (IoT) devices are designed without a physical keyboard as they go tiny, but are almost all equipped with a wireless module for networks. In this work, we aim to enable a universal virtual keyboard using wireless signals, which would allow a typing interface for tiny IoT devices or serve as a portable alternative to the unwieldy physical keyboards. To this end, we present <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">mmKey</i> , the first universal virtual keyboard system using a single millimeter-wave (mmWave) radio. By leveraging the unique advantages of mmWave signals, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">mmKey</i> converts any flat surface, with a printed paper keyboard, into an effective typing medium. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">mmKey</i> enables concurrent keystrokes and supports multiple keyboard layouts (e.g., computer keyboard, piano keyboard, or phone keypad). We design a novel signal processing pipeline to detect, segment and separate, and finally, recognize keystrokes. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">mmKey</i> does not need any training except for a minimal one-time effort of only three key-presses for keyboard calibration upon the initial setup. We prototype <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">mmKey</i> using a commodity 802.11ad/ay chipset, customized to support radar-like operations, and evaluate it with different keyboard layouts under various settings. Experimental results with ten participants demonstrate a keystroke recognition accuracy of >95% for single-key case and >90% for multikey scenario, which leads to a word recognition accuracy of >97%.

Investigating the Effects of Virtual Keyboard Position and Angle on Text Entry Performance and Subjective Evaluation

Investigating the Effects of Virtual Keyboard Position and Angle on Text Entry Performance and Subjective Evaluation

A Smart Wireless IoT Ring for Real-Time Keystroke Recognition Using Edge Computing

Enabling a single ring to recognize 3D keystrokes is a daunting challenge, especially for real-time keystrokes recognition. In this work, we integrated edge computing and machine learning algorithms into a microcontroller unit (MCU) of a smart ring so that keystrokes motion can be accurately recognized in real-time. We developed a multi-level decision (MLD) algorithm and embed a lightweight support vector machine (SVM) algorithm to execute computation for keystroke recognition on the edge of the smart ring. With this method, we can reduce the time for data transmission and avoid the data redundancy problem with the huge dispersion calculation workload; thus, improving the real-time performance of the smart ring device. Consequently, the application of this smart ring-based virtual keyboard system has minimum requirements for hardware, such as memory space and computing capacity. This study demonstrates that the use of low-performance chips in future virtual keyboard systems is possible in order to achieve lower development cost, reduced device size, and improved ease of use. The proposed smart ring is expected to provide a novel and convenient technology for real-world human-computer-interface applications in the future.

Romanian Slavonic in Unicode: problems and solutions

Editors of Slavonic and Slavonic–Romanian text can make use of a large variety of tools (fonts, physical and virtual keyboard layouts, word processors, operating systems) for transcribing and digitizing these texts in a uniform manner. The uniformity of the transcripts is based on Unicode standardization. Our study aims at explaining the place of Slavonic in Unicode and at briefly describing the most accessible tools. To this end, we shall describe the working tools from a historical and functional perspective and then provide examples in which those tools can be or have already been used to obtain a more accurate transcript. The user can choose from the existing methods and tools according to his/her purposes, needs and means. A better understanding of technical data can reduce the working time, improve transcription, accelerate learning times and generally make an editor’s work much easier.

Word recall differences between writing by hand and typing

This study determined how writing style affected people's recall and recognition of words. The three writing techniques used were handwriting using a pen on paper, typing on a conventional laptop keyboard, and typing on an iPad touch keyboard. A properly counterbalanced within-subjects experimental design was used, and 36 females between the ages of 19 and 54 participated. In this study, participants were given a wordlist and directed to use one of three distinct writing modalities to record words that were read aloud to them. For this study, the wordlist paradigm was applied. A person's memory for words typed using handwriting, a standard keyboard, and a virtual iPad keyboard was tested using oral free recall and recognition. For the analysis of the data, non-parametric statistics were used. Following analyses revealed that participants' free memory of words written in the handwriting condition was considerably higher than that of both keyboard writing conditions for the free recall measure. The findings demonstrate an all-encompassing influence of the writing medium. The findings also reveal that participants' free recall of terms written in handwriting was much better. There was no obvious effect of writing mode in the recognition condition. This shows that handwriting may have unique cognitive benefits that may not be completely retained while writing on a keyboard, especially concerning word memory components. One of these advantages might be word recall. The implications of the study's findings on cognition and education are investigated.

Eye gaze controlled adaptive virtual keyboard for users with SSMI

BACKGROUND: Users with Severe Speech and Motor Impairment (SSMI) often use a communication chart through their eye gaze or limited hand movement and care takers interpret their communication intent. There is already significant research conducted to automate this communication through electronic means. Developing electronic user interface and interaction techniques for users with SSMI poses significant challenges as research on their ocular parameters found that such users suffer from Nystagmus and Strabismus limiting number of elements in a computer screen. This paper presents an optimized eye gaze controlled virtual keyboard for English language with an adaptive dwell time feature for users with SSMI. OBJECTIVE: Present an optimized eye gaze controlled English virtual keyboard that follows both static and dynamic adaptation process. The virtual keyboard can automatically adapt to reduce eye gaze movement distance and dwell time for selection and help users with SSMI type better without any intervention of an assistant. METHODS: Before designing the virtual keyboard, we undertook a pilot study to optimize screen region which would be most comfortable for SSMI users to operate. We then proposed an optimized two-level English virtual keyboard layout through Genetic algorithm using static adaptation process; followed by dynamic adaptation process which tracks users’ interaction and reduces dwell time based on a Markov model-based algorithm. Further, we integrated the virtual keyboard for a web-based interactive dashboard that visualizes real-time Covid data. RESULTS: Using our proposed virtual keyboard layout for English language, the average task completion time for users with SSMI was 39.44 seconds in adaptive condition and 29.52 seconds in non-adaptive condition. Overall typing speed was 16.9 lpm (letters per minute) for able-bodied users and 6.6 lpm for users with SSMI without using any word completion or prediction features. A case study with an elderly participant with SSMI found a typing speed of 2.70 wpm (words per minute) and 14.88 lpm (letters per minute) after 6 months of practice. CONCLUSIONS: With the proposed layout for English virtual keyboard, the adaptive system increased typing speed statistically significantly for able bodied users than a non-adaptive version while for 6 users with SSMI, task completion time reduced by 8.8% in adaptive version than nonadaptive one. Additionally, the proposed layout was successfully integrated to a web-based interactive visualization dashboard thereby making it accessible for users with SSMI.

High-Precision and Customized Ring-Type Virtual Keyboard Based on Layout Redesign

Virtual keyboards could potentially replace traditional physical keyboards for text input and human-computer-interface due to their portability and comfortable usage characteristics. However, typical virtual keyboards such as wearable keyboards and camera-based keyboards have unsatisfactory input recognition accuracy, and hence, preventing them from gaining pervasive acceptance by users. In this study, we demonstrated that the layout optimization of keyboards can potentially improve recognition accuracy for ring-type motion sensing virtual keyboards, and analyzed several layout designs for the configuration of typing keys. Moreover, we proved that the new layout designs have significant advantages in reducing muscle oppression for the users. The layout designs were mainly improved through three aspects: arc and split design, key position adjustment, and letter arrangement. Three new layouts were tested, which show significant improvements ranging from 5.74% to 7.51% in recognition accuracy, i.e., achieving up to 97.57% accuracy for ring-type virtual keyboards. Additionally, muscle oppression test results (using surface myoelectricity data) indicate that our new keyboard layouts could provide a significant difference ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{p}&lt; 0.05$ </tex-math></inline-formula> ) in reducing muscle oppression from traditional QWERTY keyboards. As for the moving distance of finger tips, our customized layouts could even reduce half of the movement distance compared with traditional keyboards. The experimental results demonstrate that our motion sensing rings combined with customized keyboard layouts could greatly overcome several disadvantages of traditional QWERTY keyboards.

Complex Interaction as Emergent Behaviour: Simulating Mid-Air Virtual Keyboard Typing using Reinforcement Learning.

Accurately modelling user behaviour has the potential to significantly improve the quality of human-computer interaction. Traditionally, these models are carefully hand-crafted to approximate specific aspects of well-documented user behaviour. This limits their availability in virtual and augmented reality where user behaviour is often not yet well understood. Recent efforts have demonstrated that reinforcement learning can approximate human behaviour during simple goal-oriented reaching tasks. We build on these efforts and demonstrate that reinforcement learning can also approximate user behaviour in a complex mid-air interaction task: typing on a virtual keyboard. We present the first reinforcement learning-based user model for mid-air and surface-aligned typing on a virtual keyboard. Our model is shown to replicate high-level human typing behaviour. We demonstrate that this approach may be used to augment or replace human testing during the validation and development of virtual keyboards.

Triboelectric bending sensor based smart glove towards intuitive multi-dimensional human-machine interfaces

Recent advances in human-machine interface (HMI) lead to a renewed interest in creating intuitive and immersive interaction. Here, we designed a simple-structured and high-resolution bending angle triboelectric sensor named bending-angle triboelectric nanogenerator (BA-TENG) to construct a glove-based multi-dimensional HMI. With the assistance of a customized print circuit board (PCB), the glove-based HMI exhibits high sensitivity and low crosstalk in real-time multi-channel finger motion sensing. The signal-to-noise ratio (SNR) is improved by 19.36 dB. By systematically extracting and analyzing the multi-dimensional signal features of the BA-TENG, intuitive multi-dimensional HMIs were realized for smart-home, advanced robotic control, and a virtual keyboard with user recognition functionality. The classification accuracy of the virtual keyboard for seven users reached 93.1% by leveraging the advanced machine learning technique. The proposed BA-TENG-based smart glove reveals its potential as a solution for minimalist-design and intuitive multi-dimensional HMI, promising in diversified areas, including the Internet of things (IoT), assistive technology, and intelligent recognition systems. • Development of high bending angle resolution triboelectric nanogenerator (BA-TENG) using PDMS and silicon rubber. • The signal-to-noise ratio (SNR) is improved by 19.36 dB with the assistance of a trans-impedance amplifier. • Multi-dimensional signal features are extracted, including valley ( V ), width ( W ), hold time ( H ), and time interval ( T ). • Light-brightness control, robotic hand control, and a virtual keyboard with user identification were achieved. • Machine learning technique (SVM) is used for user identification with an accuracy of 93.1%.

Predicting Age Group and Gender of Smartphone Users Using Keystroke Biometrics

This paper investigated the impact of various keystroke dynamics features on a predictive biometric system. In this paper, keystroke dynamics data of 50 individuals have been acquired using an open-source data software application on an Android smartphone. A total number of 21 commonly used keystroke dynamics features were extracted from the raw data. The collected data was used in training a Random Forest algorithm using four different training sample sizes while the remaining portion of the data was used for classification. The algorithm was then used to determine the importance of 21 different keystroke dynamics features. The results showed that each features offers varying degree of importance in age-group and gender predictions. While such efforts have been made in the area of predictive keystroke dynamics using computer keyboards, literature on the same topic using touchscreen smartphone virtual keyboards have been limited.

Comparison of BlindLoginV2 and AudioBlindLogin with the common textual password authentication for the blind and visually impaired using smartphones

BlindLoginV2 and AudioBlindLogin were designed as memorable and usable graphical authentication systems for the blind and visually impaired, to reduce their difficulties in authentication as compared to alphanumeric or biometric authentication. The main aim of this paper is to compare the use of these two alternative graphical authentication systems with the traditional textual-based authentication, and to encourage the blind and visually impaired to use smartphone authentication, by overcoming the difficulties of maneuvering the virtual keyboard. It compares BlindLoginV2 and AudioBlindLogin with the 4-character common traditional textual authentication system, in terms of password creation time, mean login time performance and failure rate, and identifies the factors influencing the mean login time performance of BlindLoginV2 (object picture superiority effect) and AudioBlindLogin (auditory picture superiority effect) for the blind and visually impaired. Quantitative testing results show that both proposed systems are more accessible for the visually impaired as compared with the traditional text-based authentication system because the password creation time and mean login time is significantly shorter with lower failure rates. Results also show that the type of authentication method, the visual capability and the age of the user, and the interactions between these three factors significantly influences the mean login time performance.