User-dependent Models Research Articles

Multimodal emotion classification using machine learning in immersive and non-immersive virtual reality

Affective computing has been widely used to detect and recognize emotional states. The main goal of this study was to detect emotional states using machine learning algorithms automatically. The experimental procedure involved eliciting emotional states using film clips in an immersive and non-immersive virtual reality setup. The participants’ physiological signals were recorded and analyzed to train machine learning models to recognize users’ emotional states. Furthermore, two subjective ratings emotional scales were provided to rate each emotional film clip. Results showed no significant differences between presenting the stimuli in the two degrees of immersion. Regarding emotion classification, it emerged that for both physiological signals and subjective ratings, user-dependent models have a better performance when compared to user-independent models. We obtained an average accuracy of 69.29 ± 11.41% and 71.00 ± 7.95% for the subjective ratings and physiological signals, respectively. On the other hand, using user-independent models, the accuracy we obtained was 54.0 ± 17.2% and 24.9 ± 4.0%, respectively. We interpreted these data as the result of high inter-subject variability among participants, suggesting the need for user-dependent classification models. In future works, we intend to develop new classification algorithms and transfer them to real-time implementation. This will make it possible to adapt to a virtual reality environment in real-time, according to the user’s emotional state.

Real-Time Locomotion Recognition Algorithm for an Active Pelvis Orthosis to Assist Lower-Limb Amputees

Powered hip exoskeletons, in combination with passive prostheses, have been recently proposed to improve the economy and pattern of walking of lower-limb amputees within clinical scenarios. However, for everyday life support, a real-time control strategy that can accurately recognize different locomotion modes and transitions is required. In this letter, we proposed a novel locomotion recognition algorithm for an Active Pelvis Orthosis designed to assist people with lower-limb amputation, in quasi-static (sit-to-stand/stand-to-sit) and dynamic locomotion modes (walking and stairs negotiation). Two finite-state machines were combined to recognize in real-time the participants’ locomotion, one was a rule-based algorithm and one was based on four linear discriminant analysis classifiers. Four transfemoral amputees took part in the experiments and performed a circuit of tasks in two conditions, namely in <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">transparent mode</i> (the exoskeleton was controlled to provide null output impedance), and in <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">assistive mode</i> (the exoskeleton was controlled to output an assistive torque consistently with the locomotion mode recognized by the algorithm), to test the algorithm in real-time conditions. The median (25 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">th</sup> , 75 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">th</sup> percentile) between-subjects recognition accuracy was 94.8% (93.4%, 96.5%) with user-dependent models. Offline analysis on user-independent models with leave-one-subject-out validation resulted in between-subjects recognition accuracy equal to 95.9% (94.0%, 97.8%). The results of this study pave the way for future experimentations of the technology in ecological scenarios.

EEG-Based Emotion Recognition Using a 2D CNN with Different Kernels.

Emotion recognition is receiving significant attention in research on health care and Human-Computer Interaction (HCI). Due to the high correlation with emotion and the capability to affect deceptive external expressions such as voices and faces, Electroencephalogram (EEG) based emotion recognition methods have been globally accepted and widely applied. Recently, great improvements have been made in the development of machine learning for EEG-based emotion detection. However, there are still some major disadvantages in previous studies. Firstly, traditional machine learning methods require extracting features manually which is time-consuming and rely heavily on human experts. Secondly, to improve the model accuracies, many researchers used user-dependent models that lack generalization and universality. Moreover, there is still room for improvement in the recognition accuracies in most studies. Therefore, to overcome these shortcomings, an EEG-based novel deep neural network is proposed for emotion classification in this article. The proposed 2D CNN uses two convolutional kernels of different sizes to extract emotion-related features along both the time direction and the spatial direction. To verify the feasibility of the proposed model, the pubic emotion dataset DEAP is used in experiments. The results show accuracies of up to 99.99% and 99.98 for arousal and valence binary classification, respectively, which are encouraging for research and applications in the emotion recognition field.

Discriminating between patients with unipolar disorder, bipolar disorder, and healthy control individuals based on voice features collected from naturalistic smartphone calls

It is of crucial importance to be able to discriminate unipolar disorder (UD) from bipolar disorder (BD), as treatments, as well as course of illness, differ between the two disorders. To investigate whether voice features from naturalistic phone calls could discriminate between (1) UD, BD, and healthy control individuals (HC); (2) different states within UD. Voice features were collected daily during naturalistic phone calls for up to 972days. A total of 48 patients with UD, 121 patients with BD, and 38 HC were included. A total of 115,483 voice data entries were collected (UD [n=16,454], BD [n=78,733], and HC [n=20,296]). Patients evaluated symptoms daily using a smartphone-based system, making it possible to define illness states within UD and BD. Data were analyzed using random forest algorithms. Compared with BD, UD was classified with a specificity of 0.84 (SD: 0.07)/AUC of 0.58 (SD: 0.07) and compared with HC with a sensitivity of 0.74 (SD: 0.10)/AUC=0.74 (SD: 0.06). Compared with BD during euthymia, UD during euthymia was classified with a specificity of 0.79 (SD: 0.05)/AUC=0.43 (SD: 0.16). Compared with BD during depression, UD during depression was classified with a specificity of 0.81 (SD: 0.09)/AUC=0.48 (SD: 0.12). Within UD, compared with euthymia, depression was classified with a specificity of 0.70 (SD 0.31)/AUC=0.65 (SD: 0.11). In all models, the user-dependent models outperformed the user-independent models. The results from the present study are promising, but as reflected by the low AUCs, does not support that voice features collected during naturalistic phone calls at the current state of art can be implemented in clinical practice as a supplementary and assisting tool. Further studies are needed.

Voice analyses using smartphone-based data in patients with bipolar disorder, unaffected relatives and healthy control individuals, and during different affective states

BackgroundVoice features have been suggested as objective markers of bipolar disorder (BD).AimsTo investigate whether voice features from naturalistic phone calls could discriminate between (1) BD, unaffected first-degree relatives (UR) and healthy control individuals (HC); (2) affective states within BD.MethodsVoice features were collected daily during naturalistic phone calls for up to 972 days. A total of 121 patients with BD, 21 UR and 38 HC were included. A total of 107.033 voice data entries were collected [BD (n = 78.733), UR (n = 8004), and HC (n = 20.296)]. Daily, patients evaluated symptoms using a smartphone-based system. Affective states were defined according to these evaluations. Data were analyzed using random forest machine learning algorithms.ResultsCompared to HC, BD was classified with a sensitivity of 0.79 (SD 0.11)/AUC = 0.76 (SD 0.11) and UR with a sensitivity of 0.53 (SD 0.21)/AUC of 0.72 (SD 0.12). Within BD, compared to euthymia, mania was classified with a specificity of 0.75 (SD 0.16)/AUC = 0.66 (SD 0.11). Compared to euthymia, depression was classified with a specificity of 0.70 (SD 0.16)/AUC = 0.66 (SD 0.12). In all models the user dependent models outperformed the user independent models. Models combining increased mood, increased activity and insomnia compared to periods without performed best with a specificity of 0.78 (SD 0.16)/AUC = 0.67 (SD 0.11).ConclusionsVoice features from naturalistic phone calls may represent a supplementary objective marker discriminating BD from HC and a state marker within BD.

Identification of Free and WHO-Compliant Handwashing Moments Using Low Cost Wrist-Worn Wearables

Hand washing is the simplest and most effective gesture, when correctly performed, for the prevention of many infections. For this reason, the World Health Organization (WHO) has defined a washing procedure that guarantees effective and safe cleaning. This organization recommends that States promote this activity and monitor it continuously. Based on this fact, this article presents a work oriented to study the feasibility of identifying the moments in which a person carried out a hand washing, determining its beginning and duration, as well as if these washings were compliant with the WHO guidelines. The identification of washing moments is made through the analysis, by means of Machine Learning techniques, of the data that can be collected from the inertial sensors of the smartwatch the person is wearing. This study was carried out with the participation of 15 volunteers. Data was not only collected in controlled settings but, also, more than 600 hours of sensor measurements come from free-live conditions. The results of the study showed that it is feasible to build a solid solution based on the use of low cost wearables for the identification of washing moments. The solution is very effective (with F1 over 95%) with user-dependent models. Also, with user-independent models, the identification of WHO washings is also very effective (with F1 above 85%), but more limited in the detection of free washings (F1 around 55%).

Detecting Drowsy Learners at the Wheel of e-Learning Platforms With Multimodal Learning Analytics

Learners are expected to stay wakeful and focused while interacting with e-learning platforms. Although wakefulness of learners strongly relates to educational outcomes, detecting drowsy learning behaviors only from log data is not an easy task. In this study, we describe the results of our research to model learners’ wakefulness based on multimodal data generated from heart rate, seat pressure, and face recognition. We collected multimodal data from learners in a blended course of informatics and conducted two types of analysis on them. First, we clustered features based on learners’ wakefulness labels as generated by human raters and ran a statistical analysis. This analysis helped us generate insights from multimodal data that can be used to inform learner and teacher feedback in multimodal learning analytics. Second, we trained machine learning models with multiclass-Support Vector Machine (SVM), Random Forest (RF) and CatBoost Classifier (CatBoost) algorithms to recognize learners’ wakefulness states automatically. We achieved an average macro-F1 score of 0.82 in automated user-dependent models with CatBoost. We also showed that compared to unimodal data from each sensor, the multimodal sensor data can improve the accuracy of models predicting the wakefulness states of learners while they are interacting with e-learning platforms.

Machine Learning Model Comparisons of User Independent &amp; Dependent Intent Recognition Systems for Powered Prostheses

Developing intelligent prosthetic controllers to recognize user intent across users is a challenge. Machine learning algorithms present an opportunity to develop methods for predicting user's locomotion mode. Currently, linear discriminant analysis (LDA) offers the standard solution in the state-of-the-art for subject dependent models and has been used in the development of subject independent applications. However, the performance of subject independent models differ radically from their dependent counterpart. Furthermore, most of the studies limit the evaluation to a fixed terrain with individual stair height and ramp inclination. In this study, we investigated the use of the XGBoost algorithm for developing a subject independent model across 8 individuals with transfemoral amputation. We evaluated the performance of XGBoost across different stair heights and inclination angles and found that it generalizes well across preset conditions. Our findings suggest that XGBoost offers a potential benefit for both subject independent and subject dependent algorithms outperforming LDA and NN (DEP SS Error: 2.93% ± 0.49%, DEP TS Error: 7.03% ± 0.74%, IND SS Error: 10.12% ± 3.16%, and IND TS Error: 15.78% ± 2.39%)(p <; 0.05). We were also able to show that with the inclusion of extra sensors the model performance could continually be improved in both user dependent and independent models (p <; 0.05). Our study provides valuable information for future intent recognition systems to make them more reliable across different users and common community ambulation modes.

Accurate Measurement of Handwash Quality Using Sensor Armbands: Instrument Validation Study.

BackgroundHand hygiene is a crucial and cost-effective method to prevent health care–associated infections, and in 2009, the World Health Organization (WHO) issued guidelines to encourage and standardize hand hygiene procedures. However, a common challenge in health care settings is low adherence, leading to low handwashing quality. Recent advances in machine learning and wearable sensing have made it possible to accurately measure handwashing quality for the purposes of training, feedback, or accreditation.ObjectiveWe measured the accuracy of a sensor armband (Myo armband) in detecting the steps and duration of the WHO procedures for handwashing and handrubbing.MethodsWe recruited 20 participants (10 females; mean age 26.5 years, SD 3.3). In a semistructured environment, we collected armband data (acceleration, gyroscope, orientation, and surface electromyography data) and video data from each participant during 15 handrub and 15 handwash sessions. We evaluated the detection accuracy for different armband placements, sensor configurations, user-dependent vs user-independent models, and the use of bootstrapping.ResultsUsing a single armband, the accuracy was 96% (SD 0.01) for the user-dependent model and 82% (SD 0.08) for the user-independent model. This increased when using two armbands to 97% (SD 0.01) and 91% (SD 0.04), respectively. Performance increased when the armband was placed on the forearm (user dependent: 97%, SD 0.01; and user independent: 91%, SD 0.04) and decreased when placed on the arm (user dependent: 96%, SD 0.01; and user independent: 80%, SD 0.06). In terms of bootstrapping, user-dependent models can achieve more than 80% accuracy after six training sessions and 90% with 16 sessions. Finally, we found that the combination of accelerometer and gyroscope minimizes power consumption and cost while maximizing performance.ConclusionsA sensor armband can be used to measure hand hygiene quality relatively accurately, in terms of both handwashing and handrubbing. The performance is acceptable using a single armband worn in the upper arm but can substantially improve by placing the armband on the forearm or by using two armbands.

Toward User-Independent Emotion Recognition Using Physiological Signals

Many techniques have been developed to improve the flexibility and the fit of detection models beyond user-dependent models, yet detection tasks continue to be complex and challenging. For emotion, which is known to be highly user-dependent, improvements to the emotion learning algorithm can greatly boost predictive power. Our aim is to improve the accuracy rate of the classifier using peripheral physiological signals. Here, we present a hybrid sensor fusion approach based on a stacking model that allows for data from multiple sensors and emotion models to be jointly embedded within a user-independent model. WMD-DTW, which is a weighted multi-dimensional DTW, and the k-nearest neighbor’s algorithm k-NN are used to classify the emotions as a base model. The ensemble methods were used to learn a high-level classifier on top of the two base models. We applied a meta-learning approach to the data set and showed that the ensemble approach outperforms any individual method. We also compared the results using two data sets. Our proposed system achieved an overall accuracies of 65.6% for all users for the E4-data set and 94.0% and 93.6% for recognizing valence and arousal emotions, respectively, using the MAHNOB data set.

Using Built-In Sensors to Predict and Utilize User Satisfaction for CPU Settings on Smartphones

Understanding user experience/satisfaction with mobile systems in order to manage computational resources has become a popular approach in recent years. One of the key issues in this area is to gauge user satisfaction. In this paper, we propose and evaluate a system to save energy by altering CPU core count and frequency while keeping users satisfied. Specifically, the system uses the sensor data collected from two popular personal devices: a smartphone and a smartwatch. In the proposed architecture, we first develop prediction models by collecting sensor data along with user performance satisfaction inputs. Then, our system predicts users' current satisfaction and sets CPU core/frequency based on these predictions in real-time. We observe that sensor data gathered from these two devices are highly correlated with users' instantaneous satisfaction of the phone. We evaluate the proposed system by developing and comparing two different models. First, we develop a user-independent (user-oblivious) model by using data gathered from 10 users. Second, we develop user-dependent (personal) models for 20 different users. We demonstrate that both models can predict satisfaction with over 97% accuracy on average when a binary satisfaction model is used (i.e., users indicating satisfied versus unsatisfied). The prediction accuracy is over 91% on average if a 3-level satisfaction model is used. Our results also show that when compared to default scheme, the user-independent and user-dependent models save 8.96% and 10.12% of the total system energy on average, respectively, without impacting user satisfaction.

Activity Recognition Using Community Data to Complement Small Amounts of Labeled Instances.

Human Activity Recognition (HAR) is an important part of ambient intelligence systems since it can provide user-context information, thus allowing a greater personalization of services. One of the problems with HAR systems is that the labeling process for the training data is costly, which has hindered its practical application. A common approach is to train a general model with the aggregated data from all users. The problem is that for a new target user, this model can perform poorly because it is biased towards the majority type of users and does not take into account the particular characteristics of the target user. To overcome this limitation, a user-dependent model can be trained with data only from the target user that will be optimal for this particular user; however, this requires a considerable amount of labeled data, which is cumbersome to obtain. In this work, we propose a method to build a personalized model for a given target user that does not require large amounts of labeled data. Our method uses data already labeled by a community of users to complement the scarce labeled data of the target user. Our results showed that the personalized model outperformed the general and the user-dependent models when labeled data is scarce.

손목 부착형 카메라를 이용한 손 모양 인식에서의 사용자 적응 방법

This paper proposes a robust hand segmentation method using view-invariant characteristic of a wrist-mounted camera, and deals with a hand shape recognition system based on segmented hand information. We actively utilize the advantage of the proposed camera device that provides view-invariant images physically, and segment hand region using a Bayesian rule based on adaptive histograms. We construct HSV histograms from RGB histograms, and update HSV histograms using hand region information from a current image. We also propose a user adaptation method by which hand models gradually approach user-dependent models from user-independent models as the user uses the system. The proposed method was evaluated using 16 Korean manual alphabet, and we obtained increases of 27.91% in recognition success rate.

Detecting Naturalistic Expressions of Nonbasic Affect Using Physiological Signals

Signals from peripheral physiology (e.g., ECG, EMG, and GSR) in conjunction with machine learning techniques can be used for the automatic detection of affective states. The affect detector can be user-independent, where it is expected to generalize to novel users, or user-dependent, where it is tailored to a specific user. Previous studies have reported some success in detecting affect from physiological signals, but much of the work has focused on induced affect or acted expressions instead of contextually constrained spontaneous expressions of affect. This study addresses these issues by developing and evaluating user-independent and user-dependent physiology-based detectors of nonbasic affective states (e.g., boredom, confusion, curiosity) that were trained and validated on naturalistic data collected during interactions between 27 students and AutoTutor, an intelligent tutoring system with conversational dialogues. There is also no consensus on which techniques (i.e., feature selection or classification methods) work best for this type of data. Therefore, this study also evaluates the efficacy of affect detection using a host of feature selection and classification techniques on three physiological signals (ECG, EMG, and GSR) and their combinations. Two feature selection methods and nine classifiers were applied to the problem of recognizing eight affective states (boredom, confusion, curiosity, delight, flow/-engagement, surprise, and neutral). The results indicated that the user-independent modeling approach was not feasible; however, a mean kappa score of 0.25 was obtained for user-dependent models that discriminated among the most frequent emotions. The results also indicated that k-nearest neighbor and Linear Bayes Normal Classifier (LBNC) classifiers yielded the best affect detection rates. Single channel ECG, EMG, and GSR and three-channel multimodal models were generally more diagnostic than two--channel models.