Virtual Keyboard Research Articles

Detection of Dyslexic Children Using Machine Learning and Multimodal Hindi Language Eye-Gaze-Assisted Learning System

Children with dyslexia need specific instructions for spelling and word analysis from an early age. It is important to provide appropriate tools using technology for writing aids to such children that can help them to input text, while providing multiple feedback. However, it is unclear how children with dyslexia can efficiently use a gaze-based virtual keyboard (VK). In this study, we propose to use the typing performance of a multimodal Hindi language eye-gaze-assisted learning system based on a VK to help in the reduction of tracking errors for people with writing and reading deficiencies and to detect children with dyslexia. Performance was assessed at three levels: eye tracker, eye tracker with soft switch, and touchscreen as a baseline modality using a predefined copy-typing task. The system was validated through a series of experiments with 32 children (16 dyslexic and 16 control). The results show that the workload and the usability of the system are substantially different for children with dyslexia. Children with dyslexia have a lower typing performance when using the touchscreen modality or the eye tracker only. The detection of children with dyslexia from others was assessed with seven different types of classifiers using the typing speed on different words (AUC <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$&gt;$</tex-math></inline-formula> 0.9). These results highlight the need to have fully inclusive VKs. This work demonstrates the superior use of a multimodal system with participants having unique neuropsychological conditions and that the proposed system can be used to detect children with dyslexia.

Overview of Gesture Recognition Based Virtual Mouse and Keyboard

Abstract: Researchers around the globe nowadays are concentrating on making our devices more interactive and trying to make the devices more operational with minimal physical contact. Thai research proposes an interactive computer system that uses computer vision to create a virtual keyboard and mouse using only hand gestures. We can utilize a built-in camera or an external camera to track the image of different gestures performed by the person’s hand and according to different gestures conduct mouse cursor operations such as right and left clicks, as well as move the cursor. In addition to that, we can control the console using different gestures such as one finger to select the particular alphabet and a four-finger motion gesture to perform swipe operation in either the left or right direction. This proposed system with no wire or any other external device will act as virtual keyboard and mouse. The only hardware aspect of this system is the webcam which is used to capture images whereas the coding part will be done in python programming language

Motor Imagery BCI Classification with Frequency and Time-Frequency Features by Using Different Dimensions of the Feature Space Using Autoencoders

Brain-Computer Interfaces (BCIs) enable the users to directly communicate with machines based on various desired purposes through brain signals without moving any body parts. Thus, they have become very useful for prostheses, electric wheelchairs, virtual keyboards, and other studies like survey applications and emotion classifications. In this study, EEG signal processing was performed on the BCI Competition III-3a dataset, which contains motor imagery (MI) signals with four classes. Features of the non-stationary EEG signals belonging to three subjects were extracted using Power Spectral Density (PSD) with welch method, Wavelet Decomposition (WD), Empirical Mode Decomposition (EMD) and Hilbert-Huang Transform (HHT). From extracted 900 features, feature space dimension reduction was realized using Autoencoder, an unsupervised learning algorithm. The average accuracy obtained with Artificial Neural Network (ANN) is 74.5% for all binary classifications, which is generally a good result because of the non-stationary nature of EEG signals. 801 features yielded the best classification performance, obtained using an autoencoder with 400 hidden layer neurons.

Assessment of user-interaction strategies for neurosurgical data navigation and annotation in virtual reality

While virtual-reality (VR) has shown great promise in radiological tasks, effective user-interaction strategies that can improve efficiency and ergonomics are still under-explored and systematic evaluations of VR interaction techniques in the context of complex anatomical models are rare. Therefore, our study aims to identify the most effective interaction techniques for two common neurosurgical planning tasks in VR (point annotation and note-taking) from the state-of-the-arts, and propose a novel technique for efficient sub-volume selection necessary in neuroanatomical navigation. We assessed seven user-interaction methods with multiple input modalities (gaze, head motion, controller, and voice) for point placement and note-taking in the context of annotating brain aneurysms for cerebrovascular surgery. Furthermore, we proposed and evaluated a novel technique, called magnified selection diorama (Maserama) for easy navigation and selection of complex 3D anatomies in VR. Both quantitative and semi-quantitative (i.e., NASA Task Load Index) metrics were employed through user studies to reveal the performance of each interaction scheme in terms of accuracy, efficiency, and usability. Our evaluations demonstrated that controller-based interaction is preferred over eye-tracking-based methods for point placement while voice recording and virtual keyboard typing are better than freehand writing for note-taking. Furthermore, our new Maserama sub-volume selection technique was proven to be highly efficient and easy-to-use. Our study is the first to provide a systematic assessment of existing and new VR interaction schemes for neurosurgical data navigation and annotation. It offers valuable insights and tools to guide the design of future VR systems for radiological and surgical applications.

ThumbAir

Typing while wearing a standalone Head Mounted Display (HMD)---systems without external input devices or sensors to support text entry---is hard. To address this issue, prior work has used external trackers to monitor finger movements to support in-air typing on virtual keyboards. While performance has been promising, current systems are practically infeasible: finger movements may be visually occluded from inside-out HMD based tracking systems or, otherwise, awkward and uncomfortable to perform. To address these issues, this paper explores an alternative approach. Taking inspiration from the prevalence of thumb-typing on mobile phones, we describe four studies exploring, defining and validating the performance of ThumbAir, an in-air thumb-typing system implemented on a commercial HMD. The first study explores viable target locations, ultimately recommending eight targets sites. The second study collects performance data for taps on pairs of these targets to both inform the design of a target selection procedure and also support a computational design process to select a keyboard layout. The final two studies validate the selected keyboard layout in word repetition and phrase entry tasks, ultimately achieving final WPMs of 27.1 and 13.73. Qualitative data captured in the final study indicate that the discreet movements required to operate ThumbAir, in comparison to the larger scale finger and hand motions used in a baseline design from prior work, lead to reduced levels of perceived exertion and physical demand and are rated as acceptable for use in a wider range of social situations.

Hand writing versus Keyboard Writing: Effect on Word Recall

This study determined how writing style affected people's recall and recognition of words. The three writing techniques used were handwriting using a pen on paper, typing on a conventional laptop keyboard, and typing on an iPad touch keyboard. A properly counterbalanced within-subjects experimental design was used, and 36 females between the ages of 19 and 54 participated. In this study, participants were given a wordlist and directed to use one of three distinct writing modalities to record words that were read aloud to them. For this study, the wordlist paradigm was applied. A person's memory for words typed using handwriting, a standard keyboard, and a virtual iPad keyboard was tested using oral free recall and recognition. For the analysis of the data, non-parametric statistics were used. Following analyses revealed that participants' free memory of words written in the handwriting condition was considerably higher than that of both keyboard writing conditions for the free recall measure. The findings demonstrate an all-encompassing influence of the writing medium. The findings also reveal that participants' free recall of terms written in handwriting was much better. There was no obvious effect of writing mode in the recognition condition. This shows that handwriting may have unique cognitive benefits that may not be completely retained while writing on a keyboard, especially concerning word memory components. One of these advantages might be word recall. The implications of the study's findings on cognition and education are investigated.

Gesture Recognition Based Virtual Mouse and Keyboard

In this project, an optical mouse and keyboard are created utilizing hand motions and computer vision. The computer's camera will scan the image of various hand gestures made by a user, and the computer's mouse or pointer will move in accordance with the movements of the Users can even conduct right and left clicks using various gestures. Similar to this, several gestures can be used to operate the keyboard, such as the one-finger gesture for selecting an alphabet and the four- figure motion for swiping left and right. Without a wire or other external devices, it will function as a virtual mouse and keyboard. The project's webcam is its sole piece of hardware, and Python is used to code on the Anaconda platform. Here, the Convex hull defects are first constructed, and then an algorithm is generated and maps the mouse and keyboard functions to the defects using the defect calculations. By mapping a few of them with the mouse and keyboard, the computer will recognise the user's gesture and respond appropriately. Index Terms - Optical Mouse, Face Recognition, Virtual Mouse And Keyboard, Anaconda Platform

Ally: Understanding Text Messaging to Build a Better Onscreen Keyboard for Blind People

Millions of people worldwide use smartphones every day, but the standard issue QWERTY keyboard is poorly optimized for non-sighted input. In this article, we document the variety of methods blind people use to enter text into their smartphones, and focus on one particular need: sending text messages. We analyze two modern corpora of text messages and contrast them with an older text message corpus, as well as other corpora gathered from news articles, chat rooms, and books. We present a virtual keyboard for blind people optimized for sending text messages called Ally. To evaluate Ally, we conducted two user studies with blind participants. Our first study found increasing speeds and our second study found that half of participants reached comparable speeds to QWERTY, suggesting it may be a viable replacement. We conclude with a discussion of future work for non-sighted text-entry of text messages.

BOARD-AI: A goal-aware modeling interface for systems engineering, combining machine learning and plan recognition

Paper and pens remain the most commonly used tools by systems engineers to capture system models. They improve productivity and foster collaboration and creativity as the users do not need to conform to formal notations commonly present in Computer-Aided Systems Engineering (CASE) tools for system modeling. However, digitizing models sketched on a whiteboard into CASE tools remains a difficult and error-prone activity that requires the knowledge of tool experts. Over the past decade, switching from symbolic reasoning to machine learning has been the natural choice in many domains to improve the performance of software applications. The field of natural sketching and online recognition is no exception to the rule and most of the existing sketch recognizers rely on pre-trained sets of symbols to increase the confidence in the outcome of the recognizers. However, that performance improvement comes at the cost of trust. The lack of trust directly stems from the lack of explainability of the outcomes of the neural networks, which hinders its acceptance by systems engineering teams. A solution shall not only combine the performance and robustness but shall also earn unreserved support and trust from human users. While most of the works in the literature tip the scale in favor of performance, there is a need to better include studies on human perception into the equation to restore balance. This study presents an approach and a Human-machine interface for natural sketching that allows engineers to capture system models using interactive whiteboards. The approach combines techniques from symbolic AI and machine learning to improve performance while not compromising explainability. The key concept of the approach is to use a trained neural network to separate, upstream from the global recognition process, handwritten text from geometrical symbols, and to use the suitable technique (OCR or automated planning) to recognize text and symbols individually. Key advantages of the approach are that it does not resort to any other interaction modalities (e.g., virtual keyboards) to annotate model elements with textual properties and that the explainability of the outcomes of the modeling assistant is preserved. A user experiment validates the usability of the interface.

Designing Mobile MR Workspaces

Virtual Reality (VR) is increasingly used in everyday contexts for a variety of tasks. We particularly look at the confined space for passengers inside cars, where head-mounted displays (HMDs) could complement the prevalent use of mobile devices for work. In a field study (N=19), we tested three mobile workspace setups along the reality-virtuality continuum (Mounted Tablet, Augmented Focus Bubble, and Virtual Office) and let users re-position the virtual keyboard and display while typing on a physical keyboard in a parked car. The results revealed that using HMDs lowered users' awareness of their real surroundings but increased their perceived workload with a performance impairment of text entry rate compared to just using a tablet. Letting users customize their workspace layout improved their perceived performance and decreased pitch-axis head movements for switching between the virtual display and keyboard. This paper discusses challenges and strategies for future work regarding dynamic incorporation of productivity tools, adaptive mixed reality (MR) work environment designs, and optimizing upper thresholds of physical discomfort in mobile MR workspaces.

Spatial Model Personalization in Gboard

We introduce a framework for adapting a virtual keyboard to individual user behavior by modifying a Gaussian spatial model to use personalized key center offset means and, optionally, learned covariances. Through numerous real-world studies, we determine the importance of training data quantity and weights, as well as the number of clusters into which to group keys to avoid overfitting. While past research has shown potential of this technique using artificially-simple virtual keyboards and games or fixed typing prompts, we demonstrate effectiveness using the highly-tuned Gboard app with a representative set of users and their real typing behaviors. Across a variety of top languages, we achieve small-but-significant improvements in both typing speed and decoder accuracy.

The Effects of Background Noise on User Experience and Performance of Mixed Reality Voice Dictation

From education to manufacturing and medicine, the Microsoft HoloLens 2 mixed reality headset has been increasingly adopted. Text entry is a core functionality of the HoloLens 2, enabling communication and search features. The device tracks hand and finger movements interacting with a virtual keyboard but is slow and cumbersome. Speech-to-text dictation is a promising alternative requiring less physical effort and time. However, the HoloLens 2 dictation performance in environmental settings with varying noise levels is unclear. This study investigated the text input accuracy (WER), speed (WPM), and user perceptions of the HoloLens 2 dictation across three background noise levels (40 dB, 55 dB, and 70 dB). The 70 dB condition resulted in inferior outcomes for text input speed, perceived usability, mental demand, temporal demand, and effort. Users also reported lengthy processing delays in the 70 dB condition, while accuracy remained consistent (2–5% WER) across all noise conditions.

PinchText: One-Handed Text Entry Technique Combining Pinch Gestures and Hand Positions for Head-Mounted Displays

This paper presents PinchText, a mid-air technique with a condensed keys-based keyboard, which combines hand positions and pinch gestures, enabling one-handed text entry for Head-mounted displays (HMDs). Firstly, we conduct Study 1 to collect and analyze the typing data of PinchText with two arm postures and two movement directions, obtaining the range of hand position corresponding to the middle key set. Then, we conduct Study 2, a 6-block experiment, finding that PinchText with Hand-Up Vertical (UpV) and Hand-Down Vertical (DownV) modes could achieve a speed of 12.71 words-per-minute (WPM) and 11.14 WPM respectively with both uncorrected error rates less than 0.5%, which is 71% faster than the index finger pinch-based technique. Finally, Study 3 is conducted to explore the potential of reducing the size of the decoupled visual keyboard of PinchText, verifying that the occlusion of the virtual keyboard can be decreased. Overall, PinchText is an efficient, easy-to-learn, and comfortable text entry technique for HMDs.

Development of an Electrooculogram (EOG) and Surface Electromyogram (sEMG)-Based Human Computer Interface (HCI) Using a Bone Conduction Headphone Integrated Bio-Signal Acquisition System

Human–computer interface (HCI) methods based on the electrooculogram (EOG) signals generated from eye movement have been continuously studied because they can transmit the commands to a computer or machine without using both arms. However, usability and appearance are the big obstacles to practical applications since conventional EOG-based HCI methods require skin electrodes outside the eye near the lateral and medial canthus. To solve these problems, in this paper, we report development of an HCI method that can simultaneously acquire EOG and surface-electromyogram (sEMG) signals through electrodes integrated into bone conduction headphones and transmit the commands through the horizontal eye movements and various biting movements. The developed system can classify the position of the eyes by dividing the 80-degree range (from −40 degrees to the left to +40 degrees to the right) into 20-degree sections and can also recognize the three biting movements based on the bio-signals obtained from the three electrodes, so a total of 11 commands can be delivered to a computer or machine. The experimental results showed the interface has accuracy of 92.04% and 96.10% for EOG signal-based commands and sEMG signal-based commands, respectively. As for the results of virtual keyboard interface application, the accuracy was 97.19%, the precision was 90.51%, and the typing speed was 5.75–18.97 letters/min. The proposed interface system can be applied to various HCI and HMI fields as well as virtual keyboard applications.

Virtual Keyboards With Real-Time and Robust Deep Learning-Based Gesture Recognition

In head-mounted display devices for augmented reality and virtual reality, external signals are often entered using a virtual keyboard (VKB). Among various user interfaces for VKBs, hand gestures are widely used because they are fast and intuitive. This work proposes a gesture-recognition (GR)-based VKB algorithm that is accurate in any environment and operates in real time. Specifically, the proposed ambidextrous VKB layouts reduce the total finger travel distance on one-hand VKB layouts. Additionally, a fast typing action is proposed to use characteristics when previous and current keys are adjacent. To be robust in any environment, we utilize a deep learning (DL)-based GR method in the proposed VKB algorithm. To train DL networks, seven classes are defined and an automated dataset generation method is proposed to reduce the necessary time and effort. The proposed one-hand VKB layout with the fast typing action shows a 1.5× faster typing speed than the popular ABC keyboard layout. Furthermore, the proposed ambidextrous VKB layout brings an additional 52% improvement compared with the proposed one-hand VKB layout. The proposed DL-based GR method implemented on the well-known YOLOv3 machine learning framework shows a mean average precision rate of 95% for images including background colors similar to skin color. The proposed DL-based GR method for one-hand and ambidextrous VKBs achieves around 41 frames per second on a software platform, which allows real-time processing.

Ability-Based Methods for Personalized Keyboard Generation.

This study introduces an ability-based method for personalized keyboard generation, wherein an individual’s own movement and human–computer interaction data are used to automatically compute a personalized virtual keyboard layout. Our approach integrates a multidirectional point-select task to characterize cursor control over time, distance, and direction. The characterization is automatically employed to develop a computationally efficient keyboard layout that prioritizes each user’s movement abilities through capturing directional constraints and preferences. We evaluated our approach in a study involving 16 participants using inertial sensing and facial electromyography as an access method, resulting in significantly increased communication rates using the personalized keyboard (52.0 bits/min) when compared to a generically optimized keyboard (47.9 bits/min). Our results demonstrate the ability to effectively characterize an individual’s movement abilities to design a personalized keyboard for improved communication. This work underscores the importance of integrating a user’s motor abilities when designing virtual interfaces.

Virtual Mouse And Keyboard For Computer Interaction

Mouse and keyboard are the wonderful inventions of Human-Computer Interaction (HCI) technology. Currently, wireless mouse or a Bluetooth mouse still uses devices and is not free of devices completely since it uses a battery for power and a dongle to connect it to the PC. In the proposed virtual mouse and keyboard system, this limitation can be overcome by employing webcam or a built-in camera for capturing of hand gestures and coloured object. Method used for Hand gesture recognition is based on Convolutional Neural Networks (CNNs).The computer can be controlled virtually and can perform left click, right click, scrolling functions,volumecontrol,brightness control and other computer cursor function without the use of a physical mouse.A virtual keyboard is controlled by tracking a coloured object and it is based on image processing technique. The proposed system will mostly can use on the conference to present topics using projector and avoid COVID-19 spread by eliminating the human intervention and dependency of devices to control the computer.Main goal is to make the interaction between human and computer as natural as the interactionbetween humans.

Influence analysis of piano music immersion virtual reality cooperation based on mapping equation

Abstract In order to explore the influence of piano music immersive virtual reality cooperation, a method of VR image recognition based on mapping equation is proposed. Firstly, through the feature extraction of the image, the main color of the target image is obtained, and the extracted results are smoothed. Through the two steps of color space conversion and tone reverse mapping, the morphological transformation of the image is realized to obtain the transition image. The K-means clustering matching operation is performed on the transition image, and the optimal energy equation of color migration is obtained by hierarchical migration and global migration. The reverse mapping color migration algorithm is put into the virtual reality technology. A virtual piano is developed with HTC Vive suite and Leap Motion sensor fixed on the helmet as the hardware platform, and Unity3D and related StreamVR plug-ins and Leap Motion plug-ins as the software platform. The virtual piano forms a virtual keyboard with a cube component. By compiling the script response function of the cube component being approached and pressed or released, the performance events of the virtual piano keyboard are recorded, and the sound of the virtual piano is further realized with the help of MIDI. The experimental results show that 400 groups of data are collected for the four gesture movements, of which 300 groups are used for training modeling and 100 groups for recognition. The gesture recognition results based on HMM are shown in figure 3, and the visible recognition rate can reach more than 88%. It is proved that the immersion of 3D virtual piano is much higher than that of 2D virtual piano displayed on ordinary computer screen; Moreover, it can be played naturally with both hands, and the sense of comfort and fluency is much stronger than the 2D virtual piano played with the help of mouse and keyboard.

Virtual Keyboard Design of Lampung Script Based on Android

Lampung script fonts, in general, can only be used for typing on a computer system and have not been developed for typing on Android-based smartphones. Therefore, efforts to modernize and build typing using Lampung script for smartphone needs are very much needed on mobile phones so they can be used flexibly. This research makes a virtual keyboard / android keyboard application that can do typing using Lampung script. The virtual keyboard layout is designed by adopting several layouts found on the QWERTY keyboard by displaying all Aksara letters on one virtual keyboard display and grouping Aksara letters in a more attractive composition. To test the Lampung virtual keyboard script, the researchers used the usability testing method, namely aspects of effectiveness, efficiency, and user satisfaction, by 25 respondents. Through this test, an assessment of the effectiveness of the typing test on the success rate of respondents in doing the task was obtained, which was 75%. Evaluation of the typing efficiency aspect using time calculation gets the average result on the WPM score analysis, which is 34 WPM. This result is still below the average typing speed of people, which is 38 WPM. Testing on the aspect of user satisfaction using the system usability scale (SUS) method gets a SUS value of 76, and this result is included in the Acceptable/Good category Acceptable/Good so that users can type Lampung script through Android phones. The conclusion is that it can produce an optimal, efficient virtual keyboard layout and provide comfort for the user while using it with Android. It also could introduce the Lampung script to the younger generation.

Pseudo-haptics and self-haptics for freehand mid-air text entry in VR

Despite the lack of haptic feedback, two-finger typing remains as a common freehand mid-air text entry method in VR. To compensate for this limitation, we investigated the effect of pseudo-haptics and thumb-to-finger pinch self-haptics. Three types of virtual keyboards were developed for comparison: a normal virtual keyboard without any haptic features, a pseudo-haptic keyboard providing a haptic illusion by adding pseudo-haptic features to the normal virtual keyboard, and a pinch keyboard providing passive self-haptic feedback when the index finger touches the thumb to type bubble-shaped keys. A follow-up user study was conducted on the three keyboards to measure typing performance, perceived workload, and user experience. The results showed that the pseudo-haptic and pinch keyboards were comparable to the normal one in typing performance and workload, but beneficial in user experience and preference. The findings suggest that pseudo-haptics and self-haptics could achieve experiential benefits in VR text entry without obtrusive extra devices.