Input extraction via motion-sensor behavior analysis on smartphones

Abstract

Smartphone onboard sensors, such as the accelerometer and gyroscope, have greatly facilitated people’s life, but these sensors may bring potential security and privacy risk. This paper presents an empirical study of analyzing the characteristics of accelerometer and magnetometer data to infer users’ input on Android smartphones. The rationale behind is that the touch input actions in different positions would cause different levels of posture and motion change of the smartphone. In this work, an Android application was run as a background process to monitor data of motion sensors. Accelerometer data were analyzed to detect the occurrence of input actions on touchscreen. Then the magnetometer data were fused with accelerometer data for inferring the positions of user inputs on touchscreen. Through the mapping relationship from input positions and common layouts of keyboard or number pad, one can easily obtain the inputs. Analyses were conducted using data from three types of smartphones and across various operational scenarios. The results indicated that users’ inputs can be accurately inferred from the sensor data, with the accuracies of 100% for input-action detection and 80% for input inference in some cases. Additional experiments on the effect of smartphone screen size, sampling rate, and training data size were provided to further examine the reliability and practicability of our approach. These findings suggest that readings from accelerometer and magnetometer data could be a powerful side channel for inferring user inputs.