Unseen Activities Research Articles

3D Knee and Hip Angle Estimation With Reduced Wearable IMUs via Transfer Learning During Yoga, Golf, Swimming, Badminton, and Dance.

Wearable lower-limb joint angle estimation using a reduced inertial measurement unit (IMU) sensor set could enable quick, economical sports injury risk assessment and motion capture; however the vast majority of existing research requires a full IMU set attached to every related body segment and is implemented in only a single movement, typically walking. We thus implemented 3-dimensional knee and hip angle estimation with a reduced IMU sensor set during yoga, golf, swimming (simulated lower body swimming in a seated posture), badminton, and dance movements. Additionally, current deep-learning models undergo an accuracy drop when tested with new and unseen activities, which necessitates collecting large amounts of data for the new activity. However, collecting large datasets for every new activity is time-consuming and expensive. Thus, a transfer learning (TL) approach with long short-term memory neural networks was proposed to enhance the model's generalization ability towards new activities while minimizing the need for a large new-activity dataset. This approach could transfer the generic knowledge acquired from training the model in the source-activity domain to the target-activity domain. The maximum improvement in estimation accuracy (RMSE) achieved by TL is 23.6 degrees for knee flexion/extension and 22.2 degrees for hip flexion/extension compared to without TL. These results extend the application of motion capture with reduced sensor configurations to a broader range of activities relevant to injury prevention and sports training. Moreover, they enhance the capacity of data-driven models in scenarios where acquiring a substantial amount of training data is challenging.

CALIBRATION OF HOLZAPFEL-GASSER-OGDEN COLLATERAL LIGAMENT PROPERTIES IN A HYBRID POST-ARTHROPLASTY KNEE JOINT MODEL FOR LAXITY TESTING

This study aims to create a novel computational workflow for frontal plane laxity evaluation which combines a rigid body knee joint model with a non-linear implicit finite-element model wherein collateral ligaments are anisotropically modelled using subject-specific, experimentally calibrated Holzpfel-Gasser-Ogden (HGO) models.The framework was developed based on CT and MRI data of three cadaveric post-TKA knees. Bones were segmented from CT-scans and modelled as rigid bodies in a multibody dynamics simulation software (MSC Adams/view, MSC Software, USA). Medial collateral and lateral collateral ligaments were segmented based on MRI-scans and are modelled as finite elements using the HGO model in Abaqus (Simulia, USA). All specimens were submitted varus/valgus loading (0-10Nm) while being rigidly fixed on a testing bench to prevent knee flexion. In subsequent computer simulations of the experimental testing, rigid bodies kinematics and the associated soft-tissue force response were computed at each time step. Ligament properties were optimised using a gradient descent approach by minimising the error between the experimental and simulation-based kinematic response to the applied varus/valgus loads. For comparison, a second model was defined wherein collateral ligaments were modelled as nonlinear no-compression spring elements using the Blankevoort formulation.Models with subject-specific, experimentally calibrated HGO representations of the collateral ligaments demonstrated smaller root mean square errors in terms of kinematics (0.7900° +/− 0.4081°) than models integrating a Blankevoort representation (1.4704° +/− 0.8007°).A novel computational workflow integrating subject-specific, experimentally calibrated HGO predicted post-TKA frontal-plane knee joint laxity with clinically applicable accuracy. Generally, errors in terms of tibial rotation were higher and might be further reduced by increasing the interaction nodes between the rigid body model and the finite element software. Future work should investigate the accuracy of resulting models for simulating unseen activities of daily living.

Generalized Zero-Shot Activity Recognition with Embedding-Based Method

Sensor-based human activity recognition aims to recognize the activities performed by people with the sensor readings. Most of existing works in this area rely on supervised classification algorithms, and can only recognize activities covered by the training data. Whereas, in many practical applications, while performing activity recognition, not only the activities covered by the training data, but also some previously unseen activities need to be recognized. In this paper, we study the problem of generalized zero-shot activity recognition. In this problem, the activities that need to be recognized contain both the activities covered by the training data and the previously unseen activities. We firstly give a formulation of this problem, and then propose an embedding-based method to address it. In this method, an embedding-compatibility model is learned. When performing activity recognition, the learned model and the calibrated stacking mechanism are employed. Extensive experiments on publicly available datasets demonstrate the effectiveness of our method.

GoPose

This paper presents GoPose, a 3D skeleton-based human pose estimation system that uses WiFi devices at home. Our system leverages the WiFi signals reflected off the human body for 3D pose estimation. In contrast to prior systems that need specialized hardware or dedicated sensors, our system does not require a user to wear or carry any sensors and can reuse the WiFi devices that already exist in a home environment for mass adoption. To realize such a system, we leverage the 2D AoA spectrum of the signals reflected from the human body and the deep learning techniques. In particular, the 2D AoA spectrum is proposed to locate different parts of the human body as well as to enable environment-independent pose estimation. Deep learning is incorporated to model the complex relationship between the 2D AoA spectrums and the 3D skeletons of the human body for pose tracking. Our evaluation results show GoPose achieves around 4.7cm of accuracy under various scenarios including tracking unseen activities and under NLoS scenarios.

A Segmentation Scheme for Knowledge Discovery in Human Activity Spotting.

Spotting the target activity in a sequence of activities and transitions without applying a predetermined window size is a challenging task. The sliding window method, which is the typical approach in segmenting the continuous data stream, operates with optimal segment sizes chosen considering the type and duration of the activities. Nevertheless, activity type and duration are usually unknown to the detection unit in practice. In this study, we proposed the nonpredetermined size windowing (NSW) scheme to spot the target activity performed in a sequence of unseen activities. NSW is built on classifying progress-based features in multilayer training and prediction stages where the time-domain progress is expressed in terms of polynomials. Thus, it operates without incorporating the information regarding duration and type of the activities. We verified our method with a proof-of-concept use case where data are acquired by a single wrist-worn 3-D accelerometer. We compared our method against fixed size windowing performed with varying window sizes and feature extraction schemes; windowed energy, peak frequency, Shannon entropy, and wavelet entropy. Our method outperforms the compared schemes, reaching a median accuracy of 89% and a median true positive rate of 1 in both intrasubject and intersubject cases based on an independent test set where each test instance contains a sequence of nine different activities.

Improving the Performance of Closet-Set Classification in Human Activity Recognition by Applying a Residual Neural Network Architecture

The Time Series Classification Residual Network (TSCResNet) deep learning model is introduced in this study to improve the classification performance in the human activity recognition (HAR) problem. Specifically in the context of closed-set classification where all labels or classes are present during the model training phase. This contrasts with open-set classification where new, unseen activities are introduced to the HAR system after the training phase. The proposed TSCResNet model is evaluated with the benchmark PAMAP2 Physical Activity Monitoring Dataset. By using the same quantitative methods and data preprocessing protocols as previous research in the field of closed-set HAR, results show that the TSCResNet model architecture is able to achieve improved classification results across accuracy, the weighted F1-score, and mean F1-score.

TN-ZSTAD: Transferable Network for Zero-Shot Temporal Activity Detection.

An integral part of video analysis and surveillance is temporal activity detection, which means to simultaneously recognize and localize activities in long untrimmed videos. Currently, the most effective methods of temporal activity detection are based on deep learning, and they typically perform very well with large scale annotated videos for training. However, these methods are limited in real applications due to the unavailable videos about certain activity classes and the time-consuming data annotation. To solve this challenging problem, we propose a novel task setting called zero-shot temporal activity detection (ZSTAD), where activities that have never been seen in training still need to be detected. We design an end-to-end deep transferable network TN-ZSTAD as the architecture for this solution. On the one hand, this network utilizes an activity graph transformer to predict a set of activity instances that appear in the video, rather than produces many activity proposals in advance. On the other hand, this network captures the common semantics of seen and unseen activities from their corresponding label embeddings, and it is optimized with an innovative loss function that considers the classification property on seen activities and the transfer property on unseen activities together. Experiments on the THUMOS'14, Charades, and ActivityNet datasets show promising performance in terms of detecting unseen activities.

Global Position Prediction for Interactive Motion Capture

We present a method for reconstructing the global position of motion capture where position sensing is poor or unavailable. Capture systems, such as IMU suits, can provide excellent pose and orientation data of a capture subject, but otherwise need post processing to estimate global position. We propose a solution that trains a neural network to predict, in real-time, the height and body displacement given a short window of pose and orientation data. Our training dataset contains pre-recorded data with global positions from many different capture subjects, performing a wide variety of activities in order to broadly train a network to estimate on like and unseen activities. We compare training on two network architectures, a universal network (u-net) and a traditional convolutional neural network (CNN) - observing better error properties for the u-net in our results. We also evaluate our method for different classes of motion. We observe high quality results for motion examples with good representation in specialized datasets, while general performance appears better in a more broadly sampled dataset when input motions are far from training examples.

Wi-Fringe: Leveraging Text Semantics in WiFi CSI-Based Device-Free Named Gesture Recognition.

The lack of adequate training data is one of the major hurdles in WiFi-based activity recognition systems. In this paper, we propose Wi-Fringe, which is a WiFi CSI-based device-free human gesture recognition system that recognizes named gestures, i.e., activities and gestures that have a semantically meaningful name in English language, as opposed to arbitrary free-form gestures. Given a list of activities (only their names in English text), along with zero or more training examples (WiFi CSI values) per activity, Wi-Fringe is able to detect all activities at runtime. We show for the first time that by utilizing the state-of-the-art semantic representation of English words, which is learned from datasets like the Wikipedia (e.g., Google's word-to-vector [1]) and verb attributes learned from how a word is defined (e.g, American Heritage Dictionary), we can enhance the capability of WiFi-based named gesture recognition systems that lack adequate training examples per class. We propose a novel cross-domain knowledge transfer algorithm between radio frequency (RF) and text to lessen the burden on developers and end-users from the tedious task of data collection for all possible activities. To evaluate Wi-Fringe, we collect data from four volunteers in a multi-person apartment and an office building for a total of 20 activities. We empirically quantify the trade-off between the accuracy and the number of unseen activities.

Zero-Shot Human Activity Recognition Using Non-Visual Sensors.

Due to significant advances in sensor technology, studies towards activity recognition have gained interest and maturity in the last few years. Existing machine learning algorithms have demonstrated promising results by classifying activities whose instances have been already seen during training. Activity recognition methods based on real-life settings should cover a growing number of activities in various domains, whereby a significant part of instances will not be present in the training data set. However, to cover all possible activities in advance is a complex and expensive task. Concretely, we need a method that can extend the learning model to detect unseen activities without prior knowledge regarding sensor readings about those previously unseen activities. In this paper, we introduce an approach to leverage sensor data in discovering new unseen activities which were not present in the training set. We show that sensor readings can lead to promising results for zero-shot learning, whereby the necessary knowledge can be transferred from seen to unseen activities by using semantic similarity. The evaluation conducted on two data sets extracted from the well-known CASAS datasets show that the proposed zero-shot learning approach achieves a high performance in recognizing unseen (i.e., not present in the training dataset) new activities.

Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning

Although promising results have been achieved in video captioning, existing models are limited to the fixed inventory of activities in the training corpus, and do not generalize to open vocabulary scenarios. Here we introduce a novel task, zeroshot video captioning, that aims at describing out-of-domain videos of unseen activities. Videos of different activities usually require different captioning strategies in many aspects, i.e. word selection, semantic construction, and style expression etc, which poses a great challenge to depict novel activities without paired training data. But meanwhile, similar activities share some of those aspects in common. Therefore, we propose a principled Topic-Aware Mixture of Experts (TAMoE) model for zero-shot video captioning, which learns to compose different experts based on different topic embeddings, implicitly transferring the knowledge learned from seen activities to unseen ones. Besides, we leverage external topic-related text corpus to construct the topic embedding for each activity, which embodies the most relevant semantic vectors within the topic. Empirical results not only validate the effectiveness of our method in utilizing semantic knowledge for video captioning, but also show its strong generalization ability when describing novel activities.

A framework for unsupervised change detection in activity recognition

PurposeThis purpose of this paper is to develop a change detection technique for activity recognition model. The approach aims to detect changes in the initial accuracy of the model after training and when the model is deployed for recognizing new unseen activities without access to the ground truth. The changes between the two sessions may occur because of differences in sensor placement, orientation and user characteristics such as age and gender. However, many of the existing approaches for model adaptation in activity recognition are blind methods because they continuously adapt the recognition model without explicit detection of changes in the model performance.Design/methodology/approachThe approach determines the variation between reference activity data belonging to different classes and newly classified unseen data. If there is coherency between the data, it means the model is correctly classifying the instances; otherwise, a significant variation indicates wrong instances are being classified to different classes. Thus, the approach is formulated as a two-level architectural framework comprising of the off-line phase and the online phase. The off-line phase extracts of Shewart Chart change parameters from the training data set. The online phase performs classification of new samples and the detection of the changes in each class of activity present in the data set by using the change parameters computed earlier.FindingsThe approach is evaluated using a real activity-recognition data set. The results show that there are consistent detections that correlate with the error rate of the model.Originality/valueThe developed approach does not use ground truth to detect classifier performance degradation. Rather, it uses a data discrimination method and a base classifier to detect the changes by using the parameters computed from the reference data of each class to discriminate outliers in the new data being classified to the same class. The approach is the first, to the best of the authors’ knowledge, that addresses the problem of detecting within-user and cross-user variations that lead to concept drift in activity recognition. The approach is also the first to use statistical process control method for change detection in activity recognition, with a robust integrated framework that seamlessly detects variations in the underlying model performance.

Smartphone-Based Patients' Activity Recognition by Using a Self-Learning Scheme for Medical Monitoring.

Smartphone based activity recognition has recently received remarkable attention in various applications of mobile health such as safety monitoring, fitness tracking, and disease prediction. To achieve more accurate and simplified medical monitoring, this paper proposes a self-learning scheme for patients' activity recognition, in which a patient only needs to carry an ordinary smartphone that contains common motion sensors. After the real-time data collection though this smartphone, we preprocess the data using coordinate system transformation to eliminate phone orientation influence. A set of robust and effective features are then extracted from the preprocessed data. Because a patient may inevitably perform various unpredictable activities that have no apriori knowledge in the training dataset, we propose a self-learning activity recognition scheme. The scheme determines whether there are apriori training samples and labeled categories in training pools that well match with unpredictable activity data. If not, it automatically assembles these unpredictable samples into different clusters and gives them new category labels. These clustered samples combined with the acquired new category labels are then merged into the training dataset to reinforce recognition ability of the self-learning model. In experiments, we evaluate our scheme using the data collected from two postoperative patient volunteers, including six labeled daily activities as the initial apriori categories in the training pool. Experimental results demonstrate that the proposed self-learning scheme for activity recognition works very well for most cases. When there exist several types of unseen activities without any apriori information, the accuracy reaches above 80% after the self-learning process converges.

Sade's Interior Motives: The Importance of the Unseen in the Château de Silling

Given the Marquis de Sade's well-deserved reputation as an author who continually sought to shock his audience, why does the work he most treasured insist on hiding certain details from the reader? In Les Cent vingt journees de Sodome, ou l'Ecole du libertinage, the medieval Chateau de Silling is transformed into a highly functional space dedicated solely to libertine sexual pleasures. Through a combination of various seductions and tortures, the characters in the novel are transformed into receptacles of pleasure, for both the libertine masters and the audience. While numerous scholars have called attention to the unseen activities that take place in Silling, none has fully considered the impact of these hidden moments on the reader as libertine-in-training. Sade's literary seduction of his reader pulls us into the spiral from public to private spaces, and from passive reading to active imagination. Although Sade happily provides salacious details about most activities that will take place in the chateau, he shelters certain privileged spaces from the readers gaze. It is time to look behind the closed door at what goes on in those rooms, and why Sade chooses to keep his audience out of them. Despite Sade's continuous refrain of il faut tout dire, his chef-d'oeuvre contains numerous moments of accepted silence and secret activities, seemingly in direct contrast with the copious details presented in other sections of the work. In a later novel, Sade warns his reader that ne connaitras rien si tu n'as pas tout connu. (1) Indeed, many of Sade's characters demonstrate this need to push the imagination in order to develop ever-more intense pleasures. While later readers might have easily filled in the blanks using other Sadean texts, the audience for Les Cent vingt journees, his first major work, would have had to rely heavily on the clues and instructions found within the novel. Les Cent vingt journees provides the reader, as libertine-in-training, with the means of accomplishing this quest for complete knowledge and unlimited imagination. Absorbing the various delights and tortures described in the text gradually allows the reader to become more knowledgeable about the libertine lifestyle. By the time of the first omission, the reader has already been provided with a large list of possible activities to suggest what might be happening in these hidden moments. From the beginning of the narrative, Sade provides explicit information on what is to come, including acts ranging from fondling to orgies. While the aforementioned deeds take place in public view, the reader is also required to embellish additional details in order to explain occasional acts requiring privacy. The reader's eventual willingness and ability to improvise during these key moments stand at the heart of Sade's project. Les Cent vingt journees de Sodome, ou l'Ecole du libertinage was written in 1785 and is considered by many critics to be Sade's masterpiece, although it was not published during the author's lifetime. Most of Sade's other works contain hidden locations and secret torture chambers, but the reader is almost always invited into these spaces to survey the action. The rare exceptions to this formula are conspicuous for their unwillingness to enlighten the reader. A classic example of this sort of hidden text is found in La Philosophie dans le boudoir (1795), when Dolmance takes Augustin into an adjoining cabinet and emerges later, only to explain that il est de certaines choses qui demandent absolument des voiles. (2) In this case, the female characters insist on knowing what will take place; they are told in a whisper and immediately react with revulsion. The reader is never let in on the secret, forcing him to fill in the details. (3) Obviously, Les Cent vingt journees is not the only work to employ this technique of titillation, but it stands out for the sheer number of times the reader's voyeurism is abruptly interrupted. …

“Sleep Attacks” in a Patient With Parkinson’s Disease and Subsequent Psychiatric Decompensation

To the Editor: Sleep disorders in neurodegenerative disorders such as Parkinson's disease are underappreciated. Misdiagnosing a parasomnia can contribute to a worsened quality of life and be devastating in a psychiatrically vulnerable population, as illustrated by the following case. Case report. Mr A, a 64-year-old man, was followed up regularly for his obsessive-compulsive disorder, which had originally been diagnosed 30 years earlier and had recently been in remission on treatment with 80 mg of citalopram daily. He had a history of Parkinson's disease treated with deep brain stimulation, liver transplant following cryptogenic liver cirrhosis, and obstructive sleep apnea treated with nasal continuous positive airway pressure (CPAP). In 2010, he reported having “sleep attacks” at increased frequency during the preceding 2 months. Therefore, he was terrified of falling or doing something “bad” during his sleep. He sustained minor injuries and experienced increasing anxiety about how he arrived in strange places and positions prior to waking up, usually on the bathroom floor. These episodes were not witnessed since he lived alone, and, without anyone to account for his activities, he produced rampant guilt-laden obsessive thoughts about his unseen activities. Intrusive thoughts paralyzed Mr A, and he was no longer able to care for himself. He stopped eating and his weight went from 115 to 95 lb. At this point, he no longer wished to live and began having persecutory delusions that involved punishment for his “bad” behavior. After being admitted to the geropsychiatry unit, Mr A adapted well to its monitored environment and the titration of low-dose quetiapine (12.5 mg in the morning and 25 mg in the evening). No sleep attacks were seen. A second sleep study revealed an elevated hypopnea index of 57.7, consistent with his known obstructive sleep apnea. In addition, there was evidence of rapid eye movement (REM) sleep behavior disorder (RBD) given periodic small limb movements during REM sleep. The catatonic state of REM sleep is disrupted in RBD, which allows for patients to act out their REM-induced dream state. The conventional treatment for RBD is high-dose melatonin or clonazepam,1 but this was not started because the patient's sleep attacks, anxiety, and suicidal ideation were treated with proper use of his nasal CPAP. The sleep attacks did not recur, but the patient remained delusional for 6 months. Parkinson's disease is plagued by sleep-related problems. Sleep architecture is usually experienced as light sleep with frequent awakenings. The majority of patients experience greater insomnia, restless legs syndrome, and vivid dreaming than the general population.2 “Sleep attacks” may be associated with narcolepsy, in which an increasing loss of hypocretin cells has been correlated with disease progression in Parkinson's disease.3,4 Additionally, excessive daytime sleepiness correlates with disease severity and in its extreme form can produce sleep attacks.2 Obstructive sleep apnea may not be more common in Parkinson's disease.5 This is a case of severe obstructive sleep apnea mimicking RBD, a phenomenon that has been termed pseudo-RBD.1 A study by Iranzo and Santamaria6 examined 16 patients with REM sleep–enacting behaviors in addition to a mean hypopnea index of 67.5. Video monitoring found that the abnormal sleep behaviors occurred only during apnea-induced arousals (hypnopompic). As in this case, they remitted with treatment of the underlying obstructive sleep apnea.