Smartphone Inertial Sensors Research Articles

Security and privacy enhanced smartphone-based gait authentication with random representation learning and digital lockers

Gait data captured by inertial sensors of smartphone have demonstrated promising results on user authentication. However, most existing models stored the enrolled gait pattern in plaintext for matching with the pattern being validated, thus, posed critical security and privacy issues. In this study, we present a gait cryptosystem that generates from gait data captured by smartphone sensors the random keys for user authentication, meanwhile, secures the gait pattern. First, we propose a revocable and random binary string extraction method using deep neural network followed by feature-wise binarization. A novel loss function for network optimization is also designed, to tackle not only the intra-user stability but also the inter-user randomness. Second, we propose a new biometric key generation scheme, namely Irreversible Error Correct and Obfuscate (IECO), improved from the Error Correct and Obfuscate (ECO) scheme, to securely generate from the binary string a random and irreversible key. The model was evaluated with two benchmark datasets as OU-ISIR and whuGAIT. The evaluation showed that our model could generate the key of 139 bits from 5-second data sequence with zero False Acceptance Rate (FAR) and False Rejection Rate (FRR) smaller than 5.441%. In addition, the security and user privacy analyses showed that our model was secure against existing attacks on biometric template protection, and fulfilled the irreversibility and unlinkability requirements.

Dynamic forecasting of severe acute graft-versus-host disease after transplantation

Forecasting of severe acute graft-versus-host disease (aGVHD) after transplantation is a challenging ‘large p, small n’ problem that suffers from nonuniform data sampling. We propose a dynamic probabilistic algorithm, daGOAT, that accommodates sampling heterogeneity, integrates multidimensional clinical data and continuously updates the daily risk score for severe aGVHD onset within a two-week moving window. In the studied cohorts, the cross-validated area under the receiver operator characteristic curve (AUROC) of daGOAT rose steadily after transplantation and peaked at ≥0.78 in both the adult and pediatric cohorts, outperforming the two-biomarker MAGIC score, three-biomarker Ann Arbor score, peri-transplantation features-based models and XGBoost. Simulation experiments indicated that the daGOAT algorithm is well suited for short time-series scenarios where the underlying process for event generation is smooth, multidimensional and where there are frequent and irregular data missing. daGOAT’s broader utility was demonstrated by performance testing on a remotely different task, that is, prediction of imminent human postural change based on smartphone inertial sensor time-series data.

Longitudinal Trend Monitoring of Multiple Sclerosis Ambulation Using Smartphones.

Goal: Smartphone and wearable devices may act as powerful tools to remotely monitor physical function in people with neurodegenerative and autoimmune diseases from out-of-clinic environments. Detection of progression onset or worsening of symptoms is especially important in people living with multiple sclerosis (PwMS) in order to enable optimally adapted therapeutic strategies. MS symptoms typically follow subtle and fluctuating disease courses, patient-to-patient, and over time. Current in-clinic assessments are often too infrequently administered to reflect longitudinal changes in MS impairment that impact daily life. This work, therefore, explores how smartphones can administer daily two-minute walking assessments to monitor PwMS physical function at home. Methods: Remotely collected smartphone inertial sensor data was transformed through state-of-the-art Deep Convolutional Neural Networks, to estimate a participant's daily ambulatory-related disease severity, longitudinally over a 24-week study. Results: This study demonstrated that smartphone-based ambulatory severity outcomes could accurately estimate MS level of disability, as measured by the EDSS score ([Formula: see text]: 0.56,[Formula: see text]0.001). Furthermore, longitudinal severity outcomes were shown to accurately reflect individual participants' level of disability over the study duration. Conclusion: Smartphone-based assessments, that can be performed by patients from their home environments, could greatly augment standard in-clinic outcomes for neurodegenerative diseases. The ability to understand the impact of disease on daily-life between clinical visits, through objective digital outcomes, paves the way forward to better measure and identify signs of disease progression that may be occurring out-of-clinic, to monitor how different patients respond to various treatments, and to ultimately enable the development of better, and more personalised care.

Smartphone Inertial Sensors Applications in Medical Care Field: A Comprehensive Review

Sensors of various types are built into the majority of current smartphones each with a distinct function that aids the device's performance. Micro-Electro-Mechanical System (MEMS) inertial sensors, such as gyroscopes (gyros) and accelerometers, are built into smartphones. These embedded inertial sensors have improved user experience and accelerated smartphone development and it can be used to detect events or collect information about human activities. Additionally, machine learning algorithms applied to smartphone sensor data could have a wide range of uses in entertainment, health care, finance and other fields. There have been massive financial gains when a smartphone is utilized to provide telemedicine and remote healthcare assistance. This paper provides a brief overview of all the recent work that have been done in the medical care field by utilizing inertial navigation sensors (gyroscopes and accelerometers) whether the inertial sensors were used together or separately. The goal of this review is to outline the most recent trends in four topics. First, we’ll go over fall detection systems released research publications. Second, we'll go over the most recent scholarly papers on human activity monitoring systems that have been published. Third, we’ll review the most recently released research publications on epilepsy detection. Last, we’ll look over the recently released papers on Parkinson’s disease. This work also includes some important applications that use external inertial sensors for comparison purposes.

Human Behavior Recognition Model Based on Feature and Classifier Selection.

With the rapid development of the computer and sensor field, inertial sensor data have been widely used in human activity recognition. At present, most relevant studies divide human activities into basic actions and transitional actions, in which basic actions are classified by unified features, while transitional actions usually use context information to determine the category. For the existing single method that cannot well realize human activity recognition, this paper proposes a human activity classification and recognition model based on smartphone inertial sensor data. The model fully considers the feature differences of different properties of actions, uses a fixed sliding window to segment the human activity data of inertial sensors with different attributes and, finally, extracts the features and recognizes them on different classifiers. The experimental results show that dynamic and transitional actions could obtain the best recognition performance on support vector machines, while static actions could obtain better classification effects on ensemble classifiers; as for feature selection, the frequency-domain feature used in dynamic action had a high recognition rate, up to 99.35%. When time-domain features were used for static and transitional actions, higher recognition rates were obtained, 98.40% and 91.98%, respectively.

Human Activity Recognition for Indoor Localization Using Smartphone Inertial Sensors.

With the fast increase in the demand for location-based services and the proliferation of smartphones, the topic of indoor localization is attracting great interest. In indoor environments, users’ performed activities carry useful semantic information. These activities can then be used by indoor localization systems to confirm users’ current relative locations in a building. In this paper, we propose a deep-learning model based on a Convolutional Long Short-Term Memory (ConvLSTM) network to classify human activities within the indoor localization scenario using smartphone inertial sensor data. Results show that the proposed human activity recognition (HAR) model accurately identifies nine types of activities: not moving, walking, running, going up in an elevator, going down in an elevator, walking upstairs, walking downstairs, or going up and down a ramp. Moreover, predicted human activities were integrated within an existing indoor positioning system and evaluated in a multi-story building across several testing routes, with an average positioning error of 2.4 m. The results show that the inclusion of human activity information can reduce the overall localization error of the system and actively contribute to the better identification of floor transitions within a building. The conducted experiments demonstrated promising results and verified the effectiveness of using human activity-related information for indoor localization.

Indoor Localization Fusing WiFi With Smartphone Inertial Sensors Using LSTM Networks

Smartphone-based indoor localization has attracted considerable attentions in both research and industrial areas. However, the localization accuracy and robustness are still challenging problems due to low-cost noisy devices, especially in those complicated localization environments. Considering that pedestrian dead-reckoning (PDR) devices are widely equipped in recent smartphones, we propose a novel indoor localization fusing algorithm that integrates both wireless fidelity (WiFi) features and PDR features. By formulating the fusing indoor localization as a recursive function approximation problem, a sliding-window-based displacement scheme is designed to generate a time-series-based feature data set. We further apply the long short-term memory (LSTM) network for data fusion and localization on this data set by taking advantage of its benefits in time-series prediction and characterization. To evaluate the performance of the proposed algorithm, we compare it with state-of-the-art filter-based localization algorithms in three typical movements and three postures of holding smartphones. Extensive experiment results demonstrate the accuracy and robustness of the proposed algorithm in indoor localization, even in some extreme environments.

Expert-novice classification of mobile game player using smartphone inertial sensors

The gaming industry has seen a tremendous growth in the last decade due to an exponential increase in the number of smartphone users. Embedded smartphone sensors provide solutions for automatic game controls during game-play. In this paper, we present an experimental study for the expertise classification of a game (mobile-based) player using smartphone inertial sensors, while they are simultaneously used for game controls. The game expertise level of participants is either labeled as expert or novice using game scores. Towards this end, data from 38 participants are curated during Traffic Racer game-play (in three different trials) using the embedded gyroscope and accelerometer sensors of the smartphone. These signals are pre-processed using Savitzky-Golay smoothing filter to remove noise. Twenty time domain features are extracted from the pre-processed data and are subjected to the wrapper-based feature selection method to select an optimum subset of features. Three classifiers, including k-nearest neighbor (k-NN), random forest, and the Naive Bayes, are evaluated towards the classification of player’s expertise level, i.e., expert and novice. The best average accuracy of 92.1% is achieved with k-NN classifier using the fusion of gyroscope and accelerometer data, which outperforms the existing state-of-the-art methods.

Pose-Invariant Inertial Odometry for Pedestrian Localization

Inertial sensors in smartphones enable the relative state measurements for pedestrian localization without global positioning system (GPS) signals or beacons. Any such navigation method should notice the reality that the phone is placed at uncontrolled variant poses with respect to the human body. In this work, we focus on pedestrian position estimation from the raw inertial measurement unit (IMU) measurements without any constraint on the device carrying manners and propose a novel deep inertial odometry solution. By presenting the expression of continuous rotating, we are able to release the reliance on the unreliable orientation provided by the sensor application program interface (API). Moreover, we propose a novel loss formulation by representing the velocity as the average velocity magnitude and the moving direction. The proposed approach was assessed via the public RoNIN dataset. The localization performance of the public dataset trained network model was then evaluated by real scenario trials in the CUHK campus. Experimental results show that our model is capable of providing robust velocity estimates and generating more accurate trajectories than state-of-the-art inertial odometry methods. Specifically, the localization evaluation in the CUHK campus contains 60-min inertial signals with a length of 3 km. The trained odometry network is with the 30th percentile accuracy of 2.26 m and the 50th percentile accuracy of 4.98 m.

A deep learning‐based indoor‐positioning approach using received strength signal indication and carrying mode information

AbstractIndoor smartphone positioning is one of the key information and cummunication technology techniques enabling new opportunities for indoor navigation and mobile location‐based services to enrich our everyday lives. Generally, the development of an indoor positioning system heavily relies on wireless sensor network. Since wireless sensors can estimate the probable distance between radio source and the sensors themselves by evaluating the strengths of wireless signals received from radio sources, such as received strength signal indications of Wi‐Fi and Bluetooth. However, the radio signals could be influenced by indoor and outdoor objects, such as walls and furniture, and carrying mode of a user's smartphone, like in‐pocket or in‐backpack. But, according to the best of our knowledge, up to present, people do not know how carrying mode information (CMI) influences the positioning accuracy of a positioning system. Therefore, in this study, we propose an indoor positioning scheme, named LEarning‐based Indoor Positioning System (LEIPS), which identifies the carrying mode of a user's smartphone by using this smartphone's inertial sensors and deep learning algorithms, aiming to increase indoor positioning accuracy. Our experimental results demonstrate that this system reaches 96% of positioning accuracy. CMI is also validated, showing that it is able to improve indoor prediction accuracy.

Smartphone Sensors-Based Abnormal Driving Behaviors Detection: Serial-Feature Network

One of the important factors leading to traffic accidents is the abnormal driving behavior of drivers. Early detection of abnormal driving behaviors can effectively reduce the occurrence of traffic accidents. At present, most of the mainstream driving behavior detection methods are based on the data of a single moment, which separates the continuity of driving behavior. In this article, a driving behavior recognition algorithm based on Serial-Feature Network (SF-Net) and smart phone inertial sensor is proposed, which fully considers the continuity of driving events and uses adjacent multi time data to identify driving status. The data used in this article are collected from GPS data, 3-axis acceleration and gyroscope data of smart phone. Through the preprocessing operation, SF-net makes the input vector not only contain the current sensor data, but also fuse the relevant information of adjacent time. In SF-net, deep convolution neural network is used for feature extracting, and 10 different driving behaviors can be identified by fusing multi-level and multi-time feature information. The field test results show that the accuracy rate of the serial feature network is 97.1%, and the recall rate is 98.4%, which is better than other test network models. When the number of training samples is small, the sequential feature network can still maintain a high recognition rate, and the network model is relatively stable.

Smartphone-Based Monitoring of Parkinson Disease: Quasi-Experimental Study to Quantify Hand Tremor Severity and Medication Effectiveness.

BackgroundHand tremor typically has a negative impact on a person’s ability to complete many common daily activities. Previous research has investigated how to quantify hand tremor with smartphones and wearable sensors, mainly under controlled data collection conditions. Solutions for daily real-life settings remain largely underexplored.ObjectiveOur objective was to monitor and assess hand tremor severity in patients with Parkinson disease (PD), and to better understand the effects of PD medications in a naturalistic environment.MethodsUsing the Welch method, we generated periodograms of accelerometer data and computed signal features to compare patients with varying degrees of PD symptoms.ResultsWe introduced and empirically evaluated the tremor intensity parameter (TIP), an accelerometer-based metric to quantify hand tremor severity in PD using smartphones. There was a statistically significant correlation between the TIP and self-assessed Unified Parkinson Disease Rating Scale (UPDRS) II tremor scores (Kendall rank correlation test: z=30.521, P<.001, τ=0.5367379; n=11). An analysis of the “before” and “after” medication intake conditions identified a significant difference in accelerometer signal characteristics among participants with different levels of rigidity and bradykinesia (Wilcoxon rank sum test, P<.05).ConclusionsOur work demonstrates the potential use of smartphone inertial sensors as a systematic symptom severity assessment mechanism to monitor PD symptoms and to assess medication effectiveness remotely. Our smartphone-based monitoring app may also be relevant for other conditions where hand tremor is a prevalent symptom.

Deep Learning for Heterogeneous Human Activity Recognition in Complex IoT Applications

With continued improvements in wireless sensing technology, the notion of the Internet of Things (IoT) has been widely adopted and has become pervasive owing to its broad applications in scenarios such as ambient assisted living, smart healthcare, and smart homes. In that regard, human activity recognition (HAR) is a vital element of intelligent systems to undertake persistent surveillance of human behavior. Due to the omnipresent impact of smartphones in each person’s life, smartphone inertial sensors are used as a case study for this research. Most of the conventional approaches regard HAR as a time-series classification problem; yet, the accuracy of recognition degrades for heterogeneous sensors. In this article, we investigate encoding sensory heterogeneous HAR (HHAR) data into three-channel image representation (i.e., RGB), hence treat the HHAR task as an image classification problem. Since present convolutional network models are computationally heavy when deployed in the IoT environment, we propose a lightweight model image encoded HHAR, called multiscale image-encoded HHAR (MS-IE-HHAR). The model employs a hierarchical multiscale extraction (HME) module followed by an improved spatialwise and channelwise attention (ISCA) module to form the main architecture of the model. The HME module is formed by a group of residually connected shuffle group convolutions (SG-Conv) to extract and learn image representations from different receptive fields while reducing the number of network parameters. The ISCA module combines a lightweight spatialwise attention (SwA) block and an improved channelwise attention (CwA) module to enable the network to pay instructive attention to spatial correlations as well as channel interdependency information. Finally, two widely available HHAR public data sets (i.e., HHAR UCI, and MHEALTH) were used to evaluate the performance of the proposed models with accuracy over 98% and 99%, respectively, demonstrating the model superiority for modeling HAR from heterogeneous data sources.

Kinematic Mobile Drop Jump Analysis at Different Heights Based on a Smartphone Inertial Sensor.

The purpose of this study was to describe the acceleration variables in a plyometric jump test using the inertial sensor built into an iPhone 4S® smartphone, and the jumping variables from a contact mat. A cross-sectional study was conducted involving 16 healthy young adults. Linear acceleration, flight time, contact time and jump height were measured in a drop jump test from 60 cm and from 30 cm. Greater acceleration values were found in the drop jump test from 60 cm; the same was observed for the values from the contact mat. Multiple regression analysis was performed for each drop jump test: jump height was used as the dependent variable, and the most relevant variables were used as predictor variables (weight and maximum angular velocity in the Y axis for analysis of the drop jump from 60 cm, and weight and maximum acceleration in the Z axis for the drop jump from 30 cm). We found a significant regression model for the drop jump test from 60 cm (R2 = 0.515, p “ 0.001) and for the test from 30 cm (R2 = 0.460, p “ 0.01). According to the results obtained in this study, the built-in iPhone 4S® inertial sensor is able to measure acceleration for healthy young adults performing a vertical drop jump test. The acceleration kinematic variables are higher in the drop jump test from 60 cm than from 30 cm.

Urban Wi-Fi fingerprinting along a public transport route

Abstract The outreach of Wi-Fi localization is extended in this study for urban wide applications as they provide the high potential to employ them for numerous applications for localization and guidance in urban environments. The selected application presented in this paper is the localization and routing of public transport smartphone users. For the conducted investigations, Received Signal Strength Indicator (RSSI) values are collected for users who are travelling from home in a residential neighbourhood to work in the city centre and return along the same route. Special tramway trains are selected which provide two on-board Wi-Fi Access Points (APs). Firstly, the availability, visibility and RSSI stability of the Wi-Fi signal behavior of these APs and the APs in the surrounding environment along the routes is analyzed. Then the trajectories are estimated based on location fingerprinting. A first analyses reveals that significant differences exists between the six employed smartphones as well as times of the day, e. g. in the morning at peak hours or at off-peak hours. From the long-time observations it is seen that the two on-board APs show a high stability of the RSSI signals at the same times of the day and along the whole route. It is therefore currently investigated how they can confirm and validate user localization along the route and if they can contribute to constrain the overall positioning solution in combination with the inertial smartphone sensors. Moreover, the railway track can serve as a further constraint. As an outlook on future work, the development of a Simultaneous Localization and Mapping (SLAM) solution with a fusion with the smartphone inertial sensors is proposed.

An Accurate Fall Detection System for the Elderly People Using Smartphone Inertial Sensors

In developed countries, the number of elderly people living alone is continuously increasing. These people are more vulnerable to serious health issues, such as falling down. A sensor-based system, augmented to mobile phones, can provide a much-needed prediction to the falls, where injuries and fracture possibilities can be significantly decreased. The purpose of this study is to develop a fall recognition system based on smartphone inertial sensors, which is a combination of accelerometer and gyroscope. The system can distinguish between falls and other activity daily livings (ADLs). The data output from the inertial sensor have been used by two different classifiers; artificial neural network (ANN) and support vector machine (SVM), where the objective is to find an accurate falling classifier using smartphone inertial sensors. Results show that SVM based classifier offers an accuracy of 99.27%, which outperforms the state of the art results that use smartphone data.

Feature Extraction Methods Proposed for Speech Recognition Are Effective on Road Condition Monitoring Using Smartphone Inertial Sensors

The objective of our project is to develop an automatic survey system for road condition monitoring using smartphone devices. One of the main tasks of our project is the classification of paved and unpaved roads. Assuming recordings will be archived by using various types of vehicle suspension system and speeds in practice, hence, we use the multiple sensors found in smartphones and state-of-the-art machine learning techniques for signal processing. Despite usually not being paid much attention, the results of the classification are dependent on the feature extraction step. Therefore, we have to carefully choose not only the classification method but also the feature extraction method and their parameters. Simple statistics-based features are most commonly used to extract road surface information from acceleration data. In this study, we evaluated the mel-frequency cepstral coefficient (MFCC) and perceptual linear prediction coefficients (PLP) as a feature extraction step to improve the accuracy for paved and unpaved road classification. Although both MFCC and PLP have been developed in the human speech recognition field, we found that modified MFCC and PLP can be used to improve the commonly used statistical method.

Human Activity Recognition Using Inertial Sensors in a Smartphone: An Overview.

The ubiquity of smartphones and the growth of computing resources, such as connectivity, processing, portability, and power of sensing, have greatly changed people’s lives. Today, many smartphones contain a variety of powerful sensors, including motion, location, network, and direction sensors. Motion or inertial sensors (e.g., accelerometer), specifically, have been widely used to recognize users’ physical activities. This has opened doors for many different and interesting applications in several areas, such as health and transportation. In this perspective, this work provides a comprehensive, state of the art review of the current situation of human activity recognition (HAR) solutions in the context of inertial sensors in smartphones. This article begins by discussing the concepts of human activities along with the complete historical events, focused on smartphones, which shows the evolution of the area in the last two decades. Next, we present a detailed description of the HAR methodology, focusing on the presentation of the steps of HAR solutions in the context of inertial sensors. For each step, we cite the main references that use the best implementation practices suggested by the scientific community. Finally, we present the main results about HAR solutions from the perspective of the inertial sensors embedded in smartphones.

Hierarchical classification scheme for real-time recognition of physical activities and postural transitions using smartphone inertial sensors.

This paper introduces a novel approach for real-time classification of human activities using data from inertial sensors embedded in a smartphone. We propose a hierarchical classification scheme to recognize seven classes of activities including postural transitions. Its structure has three internal nodes composed of three Support Vector Machines (SVMs) classifiers, each one is associated with a set of activities. Moreover, each SVMs is fed with a feature vector from an adapted and optimal frequency band. Experimental results conducted on a challenging publicly available dataset named SBHAR show that our method is effective and outperforms various state-of-the-art approaches. We also show the suitability of our method to recognize postural transitions.

A Novel Semisupervised Deep Learning Method for Human Activity Recognition

Human activity recognition (HAR) based on inertial sensors has been investigated for many industrial informatics applications, such as healthcare and ubiquitous computing. Existing methods mainly rely on supervised learning schemes, which require large labeled training data. However, labeled data are sometimes difficult to acquire, while unlabeled data are readily available. Thus, we intend to make use of both labeled and unlabeled data with semisupervised learning for accurate HAR. In this paper, we propose a semisupervised deep learning approach, using temporal ensembling of deep long short-term memory, to recognize human activities with smartphone inertial sensors. With the deep neural network processing, features are extracted for local dependencies in the recurrent framework. Besides, with an ensemble approach based on both labeled and unlabeled data, we can combine together the supervised and unsupervised losses, so as to make good use of unlabeled data that the supervised learning method cannot leverage. Experimental results indicate the effectiveness of our proposed semisupervised learning scheme, when compared to several state-of-the-art semisupervised learning approaches.