Touchscreen Gestures Research Articles

TIUI: Touching Live Video for Telepresence Operation

This paper presents a framework for telepresence operation by touching live video on a touchscreen. Our goal is to enable users to use a smartpad to teleoperate everyday objects by touching the objects’ live video they are watching. To this end, we coined the term “teleinteractive device” to describe such an object with an identity, an actuator, and a communication network. We developed a touchable live video image-based user interface (TIUI) that empowers users to teleoperate any teleinteractive device by touching its live video with touchscreen gestures. The TIUI contains four modules — touch, control, recognition, and knowledge — to perform live video understanding, communication, and control for telepresence operation. We implemented a telepresence operation system that consists of a telepresence robot and teleinteractive devices at a local site, a smartpad with the TIUI at a remote site, and communication networks connecting the two sites. We demonstrated potential applications of the system in remotely controlling telepresence robots, opening doors with access control panels, and pushing power wheelchairs. We conducted user studies to show the effectiveness of the proposed framework.

Touchscreen gestures as images. A transfer learning approach for soft biometric traits recognition

Mobile devices are nowadays ubiquitous. They are equipped with a variety of sensors, each designed for capturing specific signals which can be exploited to acquire discriminating user’s traits, thus allowing to recognize, for instance, the authorized users. In this regard, we focus on capturing soft biometric traits from smartphones. Soft biometric information extracted from a human body (e.g., gender and age) is ancillary information proved to improve the performance of biometric authentication systems, and it has drawn a great deal of attention for its applications in healthcare, smart spaces and digital world as a whole. This paper presents an approach to gender and age-group recognition, namely TGSB, leveraging a transfer learning strategy applied to state-of-the-art Convolutional Neural Networks (CNNs) fed with image-based representations of touch gestures performed by users on mobile devices. We perform experiments considering, one at a time, touch gestures of the same kind, and combinations thereof, with intermediate and late fusion learning strategies. Experiments prove that TGSB is a promising approach, with up to 94% accuracy for the gender recognition and up to 99% for the age-group recognition. We highlight the most useful touch gesture for gender and age-group recognition, that is Scroll with 81% and 96% accuracy, respectively. We show that combining multiple touch gestures (intermediate fusion) with a joint latent subspace learning mechanism in CNNs improves the TGSB performance, up to 99% accuracy when considering a combination of two Scroll. Compared to previous works, TGSB exhibits much better performance in both gender and age-group recognition.

Exploring the use of Image Target Based Virtual Controller for Mobile Augmented Reality

Abstract: Mobile phones have made the Augmented Reality technology easily available to large audience and people are developing applications for it but the interaction with mobile AR is limited by touchscreen gesture, hand-based gesture can be used but it needs additional sensors like depth sensor and better processing power to work properly and lower end mobile phone devices cannot handle it and also external hardware can be build but around it but it adds to additional cost and is also not affordable to everyone. In this project a prototype application is developed to demonstrate the use of image targets to design a virtual controller that can be used to interact with AR content and implement basic tasks such as touching and moving virtual objects available in the AR space with the help of virtual controller and further the paper discusses about the benefits and drawbacks of such implementation and its future usage.

The Effect of Tablet Computer Configurations and Touchscreen Gestures on Human Biomechanics, Performance, and Subjective Assessment

As the increasing usage scenarios of tablet computers led to more non-neutral postures, optimizing the touchscreen gesture input became imminent to improve system performance and users’ well-being. Therefore, we conducted a study to investigate the influence of four tablet configurations and seven touchscreen gestures on electromyography, performance, and subjective assessment. Our results indicated that muscular loads of shoulder decreased under the Stand-Hand configuration while it increased under the Sit-Table during gesture interaction. We also found that Drag-Up and Drag-Left tended to possess higher muscular loads of shoulder while Drag-Down caused greater muscular loads of index finger. Besides, two-touch gestures spent longer duration when performing long-distance movements. Dragging in the inner direction was supposed to be more efficient than that in the outer direction. Our findings could provide a scientific basis for guiding the appropriate selection and the use of touchscreen interaction in the future HCI field.

Memory load differentially influences younger and older users’ learning curve of touchscreen gestures

In this study, we employed a recall test to investigate how memory load affects the learning curve of gesture-letter pairs for younger and older users. The gesture-letter pairs were carefully designed to mimic real-world gesture-function/command associations on a touchscreen mobile phone. Both younger and older user groups showed lower recall accuracy as the memory load of gesture-letter pairs increased, and recall performance improved with repeated memory training. More specifically, younger users improved rapidly over repeated memory sessions under all memory loads, whereas older users benefited much less from repeated memory sessions except the lowest memory load of 6 gesture-letter pairs. These results reveal that the memory load differentially modulated younger and older users’ learning curves of gesture-letter pairs. Thus, our work suggests an upper limit when adding new gesture-function associations on mobile phones and special attention should be devoted to old users.

Searching Names in Contact List by Three Touch-Screen Gestures

We compared three list scrolling gestures including sliding to scroll, moving a regular scrollbar, and scrolling through alphabetic index, in a name locating task on mobile phone. We found that scrolling through alphabetic index was the fastest independent of target locations. Moreover. for near-reaching targets RT was shorter for sliding to scroll than moving scrollbar, but it took sliding longer than moving scrollbar for far-reaching targets. Gesture evaluation scores of alphabetic index increased and those of sliding to scroll decreased over time. The different pre- and post-task rating patterns for gestures are due to their differential fatigue and emotional changes. The present study provides insights into users’ mental models for three list scrolling gestures and implications for future gesture design.

The effectiveness of zoom touchscreen gestures for authentication and identification and its changes over time

This paper focuses on how zoom touchscreen gestures can be used to continuously authenticate and identify smartphone users. The zoom gesture is critically under-researched as a behavioral biometric despite richness of data found in this gesture. Furthermore, analysing how the zoom gesture performs over time is a novel line of inquiry. Zoom samples from three different data collection sessions were sourced. In these sessions, each participant zoomed in and out on three images. Eighty-five features were extracted from each gesture. The classification models used were Support Vector Machine (SVM), Random Forest (RF), and K-nearest Neighbor (KNN). The best authentication performance of AUC 0.937 and EER 10.6% were achieved using the SVM classifier. The best identification performance of 65.5% accuracy, 69.6% precision, and 67.9% recall were achieved using the RF classifier. In terms of stability over time, SVM proved to be the most stable classifier, with an AUC degradation of only 0.007 after two weeks had elapsed. This analysis proves that zoom gestures demonstrate promise for use in continuous smartphone authentication and identification applications.

Correction to: Examining the usability of touchscreen gestures for adults with DS

Correction to: Examining the usability of touchscreen gestures for adults with DS

Child–smartphone interaction: relevance and positive and negative implications

An increasing number of children are now using smartphones, and the phenomenon is being studied keenly by computer scientists, pediatricians, developmental psychologists and educationists. We reviewed the research literature on what we call child–smartphone interaction. We found that children start actively using smartphones at around the age of two. Two- and three-year-old children can tap the screen at intended places, four-to-six-year-old children can perform single-finger touchscreen gestures and follow instructions provided in non-textual modes, and seven- and eight-year-old children can perform multi-finger touchscreen gestures, follow instructions provided as text, navigate hypertext and use simple augmented reality apps. Suitably designed apps can facilitate child development. For example, video chatting apps can help children communicate with parents, drawing apps and storytelling apps can nurture creativity, vocabulary apps can help children learn new words, and concepts of science and mathematics can also be explained to children using appropriate apps. At the same time, overuse of smartphones has negative consequences such as sleep disorders and may also contribute to obesity in children. Children as young as nine years now access the Internet. Reckless use of the Internet can expose children to inappropriate materials and cyberbullying. Therefore, content filtering software should be installed in smartphones used by children. We recommend objective assessment of children’s ability to use smartphones, designing of apps according to the capabilities and interests of children belonging to different age groups, effective use of apps in child rearing and parental oversight on children’s use of digital technologies.

Analysing Touchscreen Gestures: A Study Based on Individuals with Down Syndrome Centred on Design for All.

There has been a conscious shift towards developing increasingly inclusive applications. However, despite this fact, most research has focused on supporting those with visual or hearing impairments and less attention has been paid to cognitive impairments. The purpose of this study is to analyse touch gestures used for touchscreens and identify which gestures are suitable for individuals living with Down syndrome (DS) or other forms of physical or cognitive impairments. With this information, app developers can satisfy Design for All (DfA) requirements by selecting adequate gestures from existing lists of gesture sets. Twenty touch gestures were defined for this study and a sample group containing eighteen individuals with Down syndrome was used. A tool was developed to measure the performance of touch gestures and participants were asked to perform simple tasks that involved the repeated use of these twenty gestures. Three variables are analysed to establish whether they influence the success rates or completion times of gestures, as they could have a collateral effect on the skill with which gestures are performed. These variables are Gender, Type of Down syndrome, and Socioeconomic Status. Analysis reveals that significant difference is present when a pairwise comparison is performed, meaning individuals with DS cannot perform all gestures with the same ease. The variables Gender and Socioeconomic Status do not influence success rates or completion times, but Type of DS does.

Examining the usability of touchscreen gestures for adults with DS

This document is part of a global investigation that aims to establish best practices in usability testing of mobile applications, that fit the specific needs of persons with cognitive disabilities. The motivating factor is to improve the quality of life of people with special needs. As a first step, we want to discover what are the skills of people with DS (DS) when using a mobile device. Thanks to its direct manipulation interaction style, multi-touch technology is the ideal mechanism for learning activities with a view to the social inclusion of people with DS. This paper investigates the most common touchscreen gestures on commercial existing software. A commercial analysis was carried out to discover the touch screen gestures used by 103 free software Apps running on a mobile touch screen tablet of 11 where the applications run”. The commercial analysis showed that most applications support tap and drag operations on multi-touch technology. Additionally, the research sought to discover the adults with DS (19–54 years of age) ability to perform other gestures on multi-touch surfaces. A DS user skill study was performed to assess the ability of this user segment to interact with multi-touch surfaces. The analysis involved 53 participants, aged between 19 and 54 years, from two vocational training centers attended by people with DS in Madrid. Authors used the Gesture Games App for experimenting with multi-touch gestures such as tap, double tap, long press, drag, scale up, scale down, and one-finger rotation. Tap, double tap, and drag were the three gestures that the most participant could use. In contrast, participants had difficulty performing one-finger rotation and long press gesture. Our statistical analysis showed that the ability to perform each gesture was independent of gender, age group, and previous experience with touchscreen of a participant.

Getting smart: towards critical digital literacy pedagogies

ABSTRACT Many researchers agree that awareness of agency and how it is enacted through different participants holds a key role in developing digital literacy. Recognition of the ways in which digital tools can appropriate and shape humans’ semiotic work and even act out roles in communication is part of critical digital literacy. In digital environments, humans interact with the content (not just consume it) through interfaces that may include touchscreen, voice commands, eye movement and hand gesture, often accompanied by non-human agents (e.g. bots). The need to make transparent the hidden design intentions of these non-human liaisons is becoming more and more essential, particularly for formal educational scenarios where digital tools are commonly used. This study explores how different roles of human and non-human agents in an online environment can be made salient and how teachers can promote learner awareness of hidden agents at the layer of digital interfaces. Drawing on the notion of “exchange structure” from conversational analysis and functional linguistics, audio recordings and re-constructed screenshots are analyzed. Results highlight the actions and intentionality of human and non-human agents that form multimodal turn-taking sequences. Suggestions for developing critical digital literacy pedagogies (CDLP), including making hidden agents more salient, are proposed.

Mobile Match on Card Active Authentication Using Touchscreen Biometric

With the wide use of personal consumer electronics devices such as smartphones, people store sensitive and confidential information more on their devices. Active authentication (AA) systems continuously authenticate users to reduce possible attacks after a successful login on the device. In this article, we propose match-on-card (MOC) approach for a secure active authentication scheme using touchscreen for smartphones to enhance the security and privacy and decrease the performance overhead on the consumer device. We train a Deep Neural Network (DNN) model, and store the model on the smart card available on the device for user authentication. To implement the user verification on smart cards, we quantize inputs to the model and the model’s parameters. A speed-up technique is added to the verification phase to improve the execution time. Evaluation results show that with a well configured DNN model, our on-card authentication reaches an Equal Error Rate (EER) of 2.6% for 15 strokes and verification time of 0.65 second for each stroke. Considering the average user’s stroke frequency of 1 stroke/s, our proposed scheme shows the potential for mobile MOC active authentication using touchscreen gestures on consumer devices.

A Survey on Applying Automated Recognition of Touchscreen Stroke Gestures to Children’s Input

Abstract Gesture recognition algorithms help designers create intelligent user interfaces for a number of application areas. However, these recognition algorithms are usually designed to recognize the gestures of adults, not children, and as such they generally do not perform as well for children as adults. Recognition of younger children’s gestures is particularly poor when compared to recognition of older children’s and adults’ gestures. Researchers have begun to examine the aspects of children’s gesture articulation patterns that make recognition difficult. This paper extends the initial work examining child-specific recognition approaches by considering general-purpose approaches and how they might apply to the problem of recognizing children’s touchscreen gestures. This paper presents a survey of existing recognition and analysis techniques for gestures of both adults and children from a human-centered perspective, highlighting ways in which improved recognition can lead to a better experience for children using touchscreen gestures in a variety of contexts.

User-Silicon Entangled Mobile Identity Authentication

We explore mobile device touchscreen characteristics to build a user-device (UD) biometric physical unclonable function (PUF). Human user touchscreen interaction induces dynamic capacitive differences. Sensors detect current differences which are a function of both (1) a human biometric of how a shape is traced and (2) silicon foundry process transistor-level variability embedded in the touchscreen grid. This forms a physical function with input x defining a shape and output y abstracted from the measured current value stream. We argue and establish that this physical function has PUF attributes. Moreover, it provides a robust user-device biometric-based authentication mechanism. Authentication is based on geometric shapes (challenges) drawn on the touchscreen. Users trace them. The authentication layer creates a response abstract, and validates it against a user profile. Authentication accuracy is affected by the complexity of geometric shapes as well as the validation algorithm. We consider polyline shapes (simple gestures) and complex closed geometric shapes (complex gestures). Complex gestures offer higher response entropy, but are computationally less efficient with a slightly lower validation accuracy. Complex gestures achieve 99.6% accuracy compared with 100% for simple gestures. User profiles exhibit physical unclonable function (PUF) properties. Touchscreen gestures are quantized into binary strings. Gesture hamming distance is 60+ bits for 128-bit strings for different user-device profiles; it is 0 bits for the same profile. This demonstrates variability and reproducibility respectively. Montreal TestU01 tests binary string pseudorandom characteristics; the majority of tests pass showing pseudorandom number generator (PRG) characteristics.

Ability of children to perform touchscreen gestures and follow prompting techniques when using mobile apps.

BackgroundChildren today get access to smartphones at an early age. However, their ability to use mobile apps has not yet been studied in detail.PurposeThis study aimed to assess the ability of children aged 2–8 years to perform touchscreen gestures and follow prompting techniques, i.e., ways apps provide instructions on how to use them.MethodsWe developed one mobile app to test the ability of children to perform various touchscreen gestures and another mobile app to test their ability to follow various prompting techniques. We used these apps in this study of 90 children in a kindergarten and a primary school in New Delhi in July 2019. We noted the touchscreen gestures that the children could perform and the most sophisticated prompting technique that they could follow.ResultsTwo- and 3-year-old children could not follow any prompting technique and only a minority (27%) could tap the touchscreen at an intended place. Four- to 6-year-old children could perform simple gestures like a tap and slide (57%) and follow instructions provided through animation (63%). Seven- and 8-year-old children could perform more sophisticated gestures like dragging and dropping (30%) and follow instructions provided in audio and video formats (34%). We observed a significant difference between the number of touchscreen gestures that the children could perform and the number of prompting techniques that they could follow (F=544.0407, P<0.05). No significant difference was observed in the performance of female versus male children (P>0.05).ConclusionChildren gradually learn to use mobile apps beginning at 2 years of age. They become comfortable performing single-finger gestures and following nontextual prompting techniques by 8 years of age. We recommend that these results be considered in the development of mobile apps for children.

Touching Products Virtually: Facilitating Consumer Mental Imagery with Gesture Control and Visual Presentation

Gesture-based interaction has greatly changed the way in which we interact with online products by allowing users to control digital systems with hand movements. This study investigates how gesture-based interaction modes, namely, mid-air gesture and touchscreen gesture, compared with mouse-based interaction, affect consumers’ virtual product experiences (VPE) by eliciting mental imagery (i.e., haptic imagery and spatial imagery). Furthermore, we explore how visual product presentation can be designed to facilitate different types of interaction modes. Through a lab experiment, we find that touchscreen gesture outper- forms mid-air gesture and mouse-based interaction in terms of eliciting haptic imagery, and this effect is mitigated when 3D presentation is used. We also find that mid-air gesture outperforms touchscreen gesture and mouse-based interaction in terms of eliciting spatial imagery when 3D presentation is used. Both haptic imagery and spatial imagery can further reduce consumers’ product uncertainty. Our results extend prior work on interactivity design of VPE and further contribute to the emerging literature on gesture-based interaction.

Gender recognition in smartphones using touchscreen gestures

This paper presents an approach for gender recognition in smartphones using touchscreen gestures performed by the user. The primary behavioral data comprising readings from the accelerometer, gyroscope, and orientation sensors are acquired while the user interacts with the touchscreen device. These measurements are further enriched by deriving a secondary set of gesture attributes such as swipe length and point curvature. The GIST descriptor-based features are then extracted from two-dimensional maps of the gesture attributes. Finally, a k-nearest neighbor (k-NN) classifier recognizes the user's gender based on a subset of features identified through feature selection. We have evaluated the performance of the proposed approach on two datasets, which consist of 2268 touch gestures from 126 subjects, collected using two different touchscreen devices. Our experiments show that the approach achieves higher gender classification accuracy compared to the existing method. In addition, the performance of our approach is consistent as it provides classification accuracy of 93.65% and 92.96% on the first and second datasets, respectively when multiple gestures are combined for gender recognition. Our study demonstrates that holistic image features considered in this work provide reliable information for smartphone-based gender classification.

Examining the Usability of Touch Screen Gestures for Children With Down Syndrome

Spanish Ministry of Economy and Competitiveness and funded by the European Development Regional Fund (EDRF-FEDER) with the project TIN2014-60077-R (SUPEREMOS). This work was also supported by a pre-doctoral fellowship within the Formacion de Profesorado Universitario (FPU) program from the Spanish Ministry of Education, Culture and Sports to V. Nacher (FPU14/00136) and by a pre-doctoral scholarship given by the SENESCYT (Secretaria Nacional de Educacion Superior, Ciencia y Tecnologia e Innovacion) of the government of Ecuador (No. 381-2012) to Doris Caliz.

Dynamic Authentication of Smartphone Users Based on Touchscreen Gestures

This paper proposes a dynamic authentication of smartphone users based on their gestures on touchscreen. The user authentication consists of three stages: collecting touch gestures’ data, extracting features, and classification. Tapping, scrolling, dragging and zooming gestures’ data are acquired using a developed android application. Then, features from these gestures are extracted. Finally, three different classifiers, medians vector proximity (MVP), k-nearest neighbor (k-NN) and random forest (RF), are applied to the extracted features. The performance of these classifiers are investigated and compared considering a single-touch gesture and all possible combinations of the extracted touch gestures on smartphone. The experimental results show that the MVP classifier brings the best results when using single gestures. When two gestures are combined together, the k-NN gives the best results. The k-NN classifier reaches an equal error rate of 0% using only three gestures. The RF is not ideal to be used on smartphone users’ authentication as it gives the worst results.