Soft Keyboard Research Articles

Machine learning-based novel continuous authentication system using soft keyboard typing behavior and motion sensor data

Smartphones utilize various authentication methods, including passwords, fingerprints, and face recognition. While this information is quite practical and easy to remember, it introduces several security issues. The primary concerns involve theft, password forgetfulness, or unauthorized password copying. Implementing behavioral biometrics for user authentication adds an extra layer of security. The main contribution of this study is the utilization of soft keyboard typing behavior, a behavioral biometric, for continuous user recognition. To achieve this, the phone's grip style and typing characteristics of users are scrutinized using data collected from motion sensors and the touchscreen panel. Another challenge in mobile device authentication pertains to recognition accuracy and processing time. To expedite and optimize data classification, a hybrid classification structure is suggested. This structure incorporates correlation-based feature selection and a straightforward logistic regression method, offering rapid and highly accurate classification outcomes—a further contribution of this study. Experimental results demonstrate that user identification can be accomplished in as little as 0.03 ms, with a classification accuracy of up to 93%. Continuous authentication systems offer greater security compared to one-time authentication systems. Nevertheless, these systems might not always yield the most precise results. Overcoming this challenge necessitates the development of an efficient software architecture. In line with this, an additional contribution of this study is an explanation of how to construct a continuous authentication system using the developed architecture.

Which Thumb, the Left or Right, Touches the Letter Keys on a Smartphone QWERTY Soft Keyboard during Two-Thumb Key Entry?

This study aims to observe which thumb, the left or right, is used for keystrokes and examine the patterns during two-thumb key entry on a smartphone QWERTY soft keyboard. A total of 36 college students, including 18 left-handed and 18 right-handed, were recruited for testing, and they had 9.7 years of smartphone use experience on average. A smartphone application was implemented, and whether the left or right thumb was used for touch interactions was recorded for each of the 26 letter keys. As a result, it was found that there were slightly more letter keys that were statistically more often tapped by the left thumb during the two-thumb key entry on the QWERTY soft keyboard, regardless of the participant’s handedness. In addition, all the letter keys were touched statistically more often with the relatively closer one of both thumbs, except for the letter keys G and V in the center. It seemed that the distance between keys and thumbs was regarded as the most important factor influencing the thumb choice for keystrokes, followed by the habituated experience of using physical QWERTY keyboards.

One Line Plus Soft Keyboard Layout for Smartwatches Text Entry

The "One Line Plus" soft keyboard layout inspired by One Line and T9 layouts. This innovative design condenses the button layout into a single line, maximizing button size for ease of use. The implementation includes essential features like enter and autocorrect buttons to enhance user experience. Performance testing involving 31 participants was conducted to evaluate the effectiveness of the layout. The results showed an average words per minute (WPM) of 8.8 and a total character count of 43.48, with the highest WPM of 16 and character count of 89, and the lowest WPM of 4 and character count of 16. The analysis grouped participants based on their background, experience with smartwatches, and typing intensity. Notably, the group with frequent typing experience on smartwatches achieved the highest WPM of 11.3 and character count of 53.4, indicating the layout's efficacy for proficient users. The System Usability Scale (SUS) evaluation yielded an average score of 89.35, indicating the layout's successful usability. These findings demonstrate the potential of the One Line Plus soft keyboard layout for enhancing text entry on smartwatches.

STAACS3: Simulation Tool for AAC with Single-Switch Scanning.

Communication for people with motor impairments is a difficult, yet necessary, task in daily activities. To do so, soft keyboards, digital counterpart of physical keyboards, are used. Depending on their features, several designs can be considered, however their thoughtful evaluation in real-life is not feasible. Therefore, it becomes necessary to filter the possible configurations wisely, taking advantage of simulation tools. This paper presents STAACS3, a Simulation Tool for AAC with Single-Switch scanning, that allows to model and simulate the performance of any type of scanning keyboard regardless of its keys configuration, scanning strategy or prediction systems.

Investigation into Touch Performance on a QWERTY Soft Keyboard on a Smartphone: Touch Time, Accuracy, and Satisfaction in Two-Thumb Key Entry

This research aims to examine touch performance and user-satisfaction depending on key location in a QWERTY soft keyboard during two-thumb key entry on a smartphone. Thirty-three college students who were smartphone users were recruited, and an experimental program was implemented to measure their task completion time, the number of touch errors, and user-satisfaction during key entry. The QWERTY layout was split into 15 zones to assign absolute positions for reliable statistical analysis. The results showed that the zones with significantly longer task completion times were observed more prevalently in the zones in the periphery (p < 0.0001). In addition, relatively higher subjective satisfaction ratings were found in the zones in the center area of the QWERTY layout (p < 0.0001). It seemed that both of the results were improved in the zones that participants could immediately see without moving the thumbs, before touch interaction. Meanwhile, touch error frequencies failed to show statistical significance among the zones (p = 0.3195).

Federated Learning for Privacy Preservation of Healthcare Data From Smartphone-Based Side-Channel Attacks.

Federated learning (FL) has recently emerged as a striking framework for allowing machine and deep learning models with thousands of participants to have distributed training to preserve the privacy of users' data. Federated learning comes with the pros of allowing all participants the possibility of creating robust models even in the absence of sufficient training data. Recently, smartphone usage has increased significantly due to its portability and ability to perform many daily life tasks. Typing on a smartphone's soft keyboard generates vibrations that could be abused to detect the typed keys, aiding side-channel attacks. Such data can be collected using smartphone hardware sensors during the entry of sensitive information such as clinical notes, personal medical information, username, and passwords. This study proposes a novel framework based on federated learning for side-channel attack detection to secure this information. We collected a dataset from 10 Android smartphone users who were asked to type on the smartphone soft keyboard. We convert this dataset into two windows of five users to make two clients training local models. The federated learning-based framework aggregates model updates contributed by two clients and trained the Deep Neural Network (DNN) model individually on the dataset. To reduce the over-fitting factor, each client examines the findings three times. Experiments reveal that the DNN model achieves an accuracy of 80.09%, showing that the proposed framework has the potential to detect side-channel attacks.

Continuous Nonintrusive Mobile Device Soft Keyboard Biometric Authentication

Mobile banking, shopping, and in-app purchases utilize persistent authentication states for access to sensitive data. One-shot authentication permits access for a fixed time period. For instance, a username/password-based authentication allows a user access to all the shopping and payments data in the Amazon shopping app. Traditional user passwords and lock screens are easily compromised. Snooping attacks—observing an unsuspecting user entering passwords—and smudge attacks—examining touchscreen finger oil residue—enable compromised user authentication. Mobile device interactions provide robust human and device identity data. Such biometrics enhance authentication. In this paper, behavioral attributes during user input constitute the password. Adversary password reproduction difficulty increases since pure observation is insufficient. Current mobile continuous authentication schemes use, among others, touchscreen–swipe interactions or keyboard input timing. Many of these methods require cumbersome training or intrusive authentication. Software keyboard interactions provide a consistent biometric data stream. We develop biometric profiles using touch pressure, location, and timing. New interactions authenticate against a profile using a divergence measure. In our limited user–device data sets, the classification achieves virtually perfect accuracy.

Evaluating Soft Keyboard Layouts by Using Paper Mockups

Two experiments to compare text-entry were carried out using Soft Keyboard Layouts. Paper mockups were used and we carried out manual timing in a relaxed situation, even though within a classroom environment. Volunteer students were paired and made to work in such groups, serving as participants and also helping out in the timing process. One set of students served as participants at first and the other timed them. They then switched roles halfway and exchanged the tasks of participation and timing. Participants tapped the phrase “the quick brown fox jumps over the lazy dog” on each keyboard layouts five times using stylus. The time for word entry was measured and used to determine the speeds; which were found to be 24.8 to 34.3 for QWERTY Keyboard Layout and 11.9 to 14.8 for Opti Layout. We discussed the advantages and shortfalls of the methods of evaluation.

Towards a Unified and Efficient Command Selection Mechanism for Touch-Based Devices Using Soft Keyboard Hotkeys

We advocate for the usage of hotkeys on touch-based devices by capitalising on soft keyboards through four studies. First, we evaluated visual designs and recommended icons with command names for novices while letters with command names for experts. Second, we investigated the discoverability by asking crowdworkers to use our prototype, with some tasks only doable upon successfully discovering the technique. Discovery rates were high regardless of conditions that vary the familiarity and saliency of modifier keys. However, familiarity with desktop hotkeys boosted discoverability. Our third study focused on how prior knowledge of hotkeys could be leveraged and resulted in a 5% selection time improvement and identified the role of spatial memory in retention. Finally, we compared our soft keyboard layout with a grid layout similar to FastTap. The latter offered a 12–16% gain on selection speed, but at a high cost in terms of screen estate and low spatial stability.

Modeling Touch Point Distribution with Rotational Dual Gaussian Model.

Touch point distribution models are important tools for designing touchscreen interfaces. In this paper, we investigate how the finger movement direction affects the touch point distribution, and how to account for it in modeling. We propose the Rotational Dual Gaussian model, a refinement and generalization of the Dual Gaussian model, to account for the finger movement direction in predicting touch point distribution. In this model, the major axis of the prediction ellipse of the touch point distribution is along the finger movement direction, and the minor axis is perpendicular to the finger movement direction. We also propose using projected target width and height, in lieu of nominal target width and height to model touch point distribution. Evaluation on three empirical datasets shows that the new model reflects the observation that the touch point distribution is elongated along the finger movement direction, and outperforms the original Dual Gaussian Model in all prediction tests. Compared with the original Dual Gaussian model, the Rotational Dual Gaussian model reduces the RMSE of touch error rate prediction from 8.49% to 4.95%, and more accurately predicts the touch point distribution in target acquisition. Using the Rotational Dual Gaussian model can also improve the soft keyboard decoding accuracy on smartwatches.

Area Design of Keyboard Layout for Comfortable Texting Ability with the Thumb Jacobian Matrix

ABSTRACT The objective of this paper is to develop a theory for designing the keyboard area of smartphones, and endowing the users with comfortable texting ability via one thumb. In light of the fact that most users tend to hold the smartphone in one hand and manipulate with one thumb in daily life, this paper extracts the thumb kinematic chain from the thumb anatomy of the human hand, and defines the manipulation comfort ellipsoid of the thumb. Then some indexes such as the morphology and volume of the comfort ellipsoid, and the condition number of the thumb Jacobian are presented to evaluate the manipulation comfort of the thumb. Finally, the design problem of the smartphone’s keyboard area is transformed into an optimization problem that determines the most comfortable manipulation space for the human thumb. The proposed quantitative study method of the thumb manipulation comfort is verified by 12 subjects who hold a smartphone with one hand and manipulate the screen with one thumb, and the most comfortable keyboard layout of smartphone is determined for the 12 subjects. The proposed method for the manipulation comfort of the human thumb bridges the gap between the layout planning of soft keyboard of smartphones and the comfort of single thumb manipulation.

การลดความผิดพลาดในการพิมพ์ภาษาไทยบนแป้นพิมพ์เสมือนด้วยวิธีปรับพื้นที่ปุ่มเป้าหมายและการใช้แบบจำลองทางภาษา LSTM

การลดความผิดพลาดในการพิมพ์ภาษาไทยบนแป้นพิมพ์เสมือนด้วยวิธีปรับพื้นที่ปุ่มเป้าหมายและการใช้แบบจำลองทางภาษา LSTM

One Line Soft Keyboard with T9-keys Layout Proposal on Smartwatch Touch Screen Device

A smartwatch is a mini-sized digital computer device that functions apart from being a watch and can also perform intelligent computing tasks. Due to screen size limitations, a smartwatch requires special design techniques on its input and output to allow interaction with its users. Text entry is often required by many applications, but because the smartwatch screen is small, text entry techniques commonly used on personal computers are impractical when applied to smartwatches. In this study, we introduce a design concept proposal for a soft keyboard called One Line Soft Keyboard with T9-Keys Layout. The soft keyboard design concept is expected to reduce the space on the keyboard on the smartwatch screen, along with the use of widely used button layouts that are familiar to users.

WiPass: 1D-CNN-based smartphone keystroke recognition Using WiFi signals

Keystroke privacy is crucial for smartphone system security and user privacy. User’s private information may be leaked if the sequences of keystrokes on the numeric soft keyboard are obtained by an adversary in a certain way. Different keystrokes may lead to different finger movements, causing diverse interference to WiFi signals, which can be indicated by the fluctuation of Channel State Information (CSI) waveforms. In this paper, we propose WiPass, a keystroke recognition system for classifying numeric keyboard inputs on smartphones, which consists of a transmitter (e.g. a WiFi router) and a receiver (e.g. a desktop computer with a Commercial Off-The-Shelf (COTS) WiFi NIC). The key inspiration comes from the fact that while performing a certain keystroke near a receiver, the CSI values received by the receiver vary in a unique pattern. WiPass can extract and analyze the CSI Data generated by user’s keystroke operations on the smartphone, thus inferring the users’ numeric keystrokes by comparing and classifying the CSI waveforms of the different keystrokes. Distinct from the previous keystroke inference approaches, WiPass employs 1D Convolutional Neural Network (1D-CNN) model as the classification model instead of other machine learning models. The experimental results show that the accuracy rate of the WiPass in detecting a keystroke and in classifying keystrokes reaches over 95% and 85% respectively.

Efficient Typing on Ultrasmall Touch Screens With In Situ Decoder and Visual Feedback

Typing on a smartwatch is challenging because of the fat-finger problem. Rising to the challenge, we present a soft keyboard for ultrasmall touch screen devices with efficient visual feedback integrated with autocorrection and prediction techniques. After exploring the design space to support efficient typing on smartwatches, we designed a novel and space-saving text entry interface based on an in situ decoder and prediction function that can run in real-time on a smartwatch such as LG Watch Style. We outlined the details implemented through performance optimization techniques and released interface code, APIs, and libraries as open source. We examined the design decisions with the simulations and studied the visual feedback methods in terms of performance and user preferences. The experiment showed that users could type more accurately and quickly on the target device with our best-performing visual feedback design and implementation. The simulation result showed that the single word suggestion could yield a sufficiently high hit ratio using the optimized word suggestion algorithm.

Your Eyes Reveal Your Secrets: An Eye Movement Based Password Inference on Smartphone

The widespread use of smartphones has brought great convenience to our daily lives, while at the same time we have been increasingly exposed to security threats. Keystroke security is essential to user privacy protection. In this paper, we present GazeRevealer, a novel side-channel based keystroke inference framework to infer sensitive inputs on smartphone from video recordings of victim's eye patterns captured from smartphone front camera. We observe that eye movements typically follow the keystrokes typing on the number-only soft keyboard during password input. By exploiting eye movement patterns, we are able to infer the passwords being entered. We propose a novel algorithm to extract sensitive eye images from video streams, and classify these images with Support Vector Classification. We also propose a novel classification enhancement algorithm to further improve classification accuracy. Compared with prior keystroke detection approaches, GazeRevealer does not require any external auxiliary devices, and it only relies on smartphone front camera. We evaluate the performance of GazeRevealer on several smartphones under different real-life usage scenarios. The results show that GazeRevealer achieves an inference rate of 77.89 percent for single key number and an inference rate of 84.38 percent for 6-digit password in the ideal case.

Multi-walled carbon nanotubes functionalized silk fabrics for mechanical sensors and heating materials

MWCNTs@silk fabric (CSF) with single-side, good conductive property was prepared via a surface micro-dissolution strategy. In this method, the partial dissolution of silk fibers was regulated by the concentration formic acid and the interaction between the protein and its solvent. The dissolved silk protein served as the physical crosslinking agent that connected dispersed carbon nanoparticles to form a continuous conductive network, avoiding complex chemical treatments and the use of physical adhesives. CSF possessed good electrical conductivity (468 Ω cm−1) and durability. Moreover, CSF had a unique single-side loading characteristic and its resistance changed from 98.4% to 102% of the initial value with the bending angle varying from −112.5° to 112.5°, so CSF could be used as sensors to monitor the mechanical activities effectively. Patterned conductive MWCNTs@silk fabric (PCSF) was fabricated by using patterned templates during the deposition process and used for wider applications, such as flexible switches (F-shaped PCSF), soft keyboards (keyboard-like PCSF), electric heating materials (S-shaped PCSF) and microwave-assisted heating materials (butterfly-like PCSF).

AlphaLogger: detecting motion-based side-channel attack using smartphone keystrokes

Due to the advancement in technologies and excessive usability of smartphones in various domains (e.g., mobile banking), smartphones became more prone to malicious attacks.Typing on the soft keyboard of a smartphone produces different vibrations, which can be abused to recognize the keys being pressed, hence, facilitating side-channel attacks. In this work, we develop and evaluate AlphaLogger- an Android-based application that infers the alphabet keys being typed on a soft keyboard. AlphaLogger runs in the background and collects data at a frequency of 10Hz/sec from the smartphone hardware sensors (accelerometer, gyroscope and magnetometer) to accurately infer the keystrokes being typed on the soft keyboard of all other applications running in the foreground. We show a performance analysis of the different combinations of sensors. A thorough evaluation demonstrates that keystrokes can be inferred with an accuracy of 90.2% using accelerometer, gyroscope, and magnetometer.

I-Keyboard: Fully Imaginary Keyboard on Touch Devices Empowered by Deep Neural Decoder.

Text entry aims to provide an effective and efficient pathway for humans to deliver their messages to computers. With the advent of mobile computing, the recent focus of text-entry research has moved from physical keyboards to soft keyboards. Current soft keyboards, however, increase the typo rate due to a lack of tactile feedback and degrade the usability of mobile devices due to their large portion on screens. To tackle these limitations, we propose a fully imaginary keyboard (I-Keyboard) with a deep neural decoder (DND). The invisibility of I-Keyboard maximizes the usability of mobile devices and DND empowered by a deep neural architecture allows users to start typing from any position on the touch screens at any angle. To the best of our knowledge, the eyes-free ten-finger typing scenario of I-Keyboard which does not necessitate both a calibration step and a predefined region for typing is first explored in this article. For the purpose of training DND, we collected the largest user data in the process of developing I-Keyboard. We verified the performance of the proposed I-Keyboard and DND by conducting a series of comprehensive simulations and experiments under various conditions. I-Keyboard showed 18.95% and 4.06% increases in typing speed (45.57 words per minute) and accuracy (95.84%), respectively, over the baseline.

An examination of the effort hypothesis for the perceived strength of passwords

Previous work has examined whether keyboards that have multiple levels of entry (e.g., soft keyboards, found on mobile devices) increase users’ perceptions of password security. That work found that passwords that required greater numbers of keyboard transitions had higher perceived strength. Unfortunately, it was impossible to determine whether this increase in perceived strength was due solely to the extra keyboard transition steps or whether the inclusion of special characters in the passwords (which is required in order to increase the number of keyboard transitions) was driving that perception. The research described in this paper aims to disambiguate these two possibilities. In these experiments, rather than typing the passwords on keyboards, the users simply spoke the password. In this case, there is no additional effort or keypress required in order to enter the special character(s). Results showed that there was a positive correlation between the perceived password strength and the number of keyboard transition for spoken passwords, in a fashion nearly identical to the typed password study. This suggests that the increase in perceived password strength cannot be explained simply as a function of the additional expended effort required to enter additional keyboard transition steps, although it may play a role.