Multimodal Time-series Data Research Articles

New ecological knowledge of the bubblegum coral Paragorgia arborea (L. 1758) using computer vision

IntroductionModern stationary observatories, equipped with cameras and other sensors, make it possible to observe and study cold-water corals in situ over increasingly long periods of time. In this study we investigate data collected for a bubblegum coral (Paragorgia arborea) with the LoVe (Lofoten Vesteralen) observatory for a period of 117 days, including time series of images, current, water depth and temperature. This kind of multimodal time series data represent a new methodological challenge for marine technology and bioinformatics.MethodsIn this work, we continue on the path to computer-aided ocean data analysis and show how, in addition to polyp activity, morphological parameters can be extracted from image data with a new deep learning-based approach. Using customized visualizations we show how this image-derived data together with the environmental data can be analysed and new interpretations are supported.ResultsOur results indicate that the polyp activity and branch diameter variation followed a semi-cyclic pattern, with the polyps changing from fully retracted to fully expanded faster than the image sampling rate, resulting in an almost binary time series that showed positive correlation with the current strength, which may be related to food supply. The periods of full polyp expansion ranged from 1 to 4 h, with the majority around 1 h. The variation in branch diameter and volume of intra-skeletal channels probably relates to the intra-colonial transport of nutrients and waste products. During a 16 day period in March 2019, the polyps remained retracted for an extended period accompanied by little variation in branch diameter. This behaviour was not related to any of the measured environmental conditions. Further studies are required to determine if this could be linked to biological processes, such as reproduction.

IP3T: an interpretable multimodal time-series model for enhanced gait phase prediction in wearable exoskeletons.

Wearable exoskeletons assist individuals with mobility impairments, enhancing their gait and quality of life. This study presents the iP3T model, designed to optimize gait phase prediction through the fusion of multimodal time-series data. The iP3T model integrates data from stretch sensors, inertial measurement units (IMUs), and surface electromyography (sEMG) to capture comprehensive biomechanical and neuromuscular signals. The model's architecture leverages transformer-based attention mechanisms to prioritize crucial data points. A series of experiments were conducted on a treadmill with five participants to validate the model's performance. The iP3T model consistently outperformed traditional single-modality approaches. In the post-stance phase, the model achieved an RMSE of 1.073 and an R2 of 0.985. The integration of multimodal data enhanced prediction accuracy and reduced metabolic cost during assisted treadmill walking. The study highlights the critical role of each sensor type in providing a holistic understanding of the gait cycle. The attention mechanisms within the iP3T model contribute to its interpretability, allowing for effective optimization of sensor configurations and ultimately improving mobility and quality of life for individuals with gait impairments.

Toward Enhanced Prediction of High‐Impact Solar Energetic Particle Events Using Multimodal Time Series Data Fusion Models

AbstractSolar energetic particle (SEP) events, originating from solar flares and Coronal Mass Ejections, present significant hazards to space exploration and technology on Earth. Accurate prediction of these high‐energy events is essential for safeguarding astronauts, spacecraft, and electronic systems. In this study, we conduct an in‐depth investigation into the application of multimodal data fusion techniques for the prediction of high‐energy SEP events, particularly ∼100 MeV events. Our research utilizes six machine learning (ML) models, each finely tuned for time series analysis, including Univariate Time Series (UTS), Image‐based model (Image), Univariate Feature Concatenation (UFC), Univariate Deep Concatenation (UDC), Univariate Deep Merge (UDM), and Univariate Score Concatenation (USC). By combining time series proton flux data with solar X‐ray images, we exploit complementary insights into the underlying solar phenomena responsible for SEP events. Rigorous evaluation metrics, including accuracy, F1‐score, and other established measures, are applied, along with K‐fold cross‐validation, to ensure the robustness and generalization of our models. Additionally, we explore the influence of observation window sizes on classification accuracy.

Design of an Iterative Method for Enhanced Multimodal Time Series Analysis Using Graph Attention Networks, Variational Graph Autoencoders, and Transfer Learning

In the ever-evolving landscape of data analysis, the need to efficiently and accurately interpret multimodal time series data has become paramount. Traditional methods often fall short in addressing the complex dependencies and dynamics inherent in such data, limiting their effectiveness in real-world applications. This work introduces a comprehensive approach that leverages Graph Attention Networks (GATs), Variational Graph Autoencoders (VGAEs), transfer learning with pretrained transformers, and Bayesian state-space models to overcome these limitations. GATs are selected for their ability to dynamically focus on relevant modalities through attention mechanisms, thereby capturing the intricate relationships between different data modalities. This method significantly enhances the model's ability to integrate multimodal information, leading to notable improvements in classification, prediction, and anomaly detection tasks. VGAEs are utilized to learn latent representations within a graph-based framework, promoting unsupervised learning while unveiling the underlying data structure. The resultant embeddings are pivotal for downstream tasks like clustering and visualization, encapsulating the interactions within multimodal time series data effectively. Furthermore, this work incorporates transfer learning with pretrained transformers to harness extensive knowledge from large datasets, adapting it to multimodal time series analysis. This strategy excels in capturing long-range dependencies, thereby augmenting generalization and performance in data-scarce scenarios. Bayesian state-space models are employed to elucidate the temporal dynamics and uncertainties of time series data, offering a robust framework for probabilistic inference and enhancing the interpretability and reliability of forecasting and anomaly detection. The efficacy of the proposed model is rigorously evaluated using diverse datasets, including the Yahoo! Stock Dataset, Forest Cover Dataset, and an empirical collection of 100k time series data samples. The results demonstrate a significant leap in performance metrics, including a 9.5% increase in precision, 8.5% boost in accuracy, 8.3% rise in recall, 10.4% reduction in delay, 9.4% enhancement in AUC, and a 5.9% improvement in specificity, alongside superior pre-emption capabilities compared to existing methods. This work not only addresses the pressing need for advanced multimodal time series analysis techniques but also sets a new benchmark for efficiency and accuracy. The integration of GATs, VGAEs, transfer learning with pretrained transformers, and Bayesian state-space models presents a formidable approach that significantly advances the field, offering profound impacts on a wide array of applications.

Deep Belief Networks (DBN) with IoT-Based Alzheimer’s Disease Detection and Classification

Dementias that develop in older people test the limits of modern medicine. As far as dementia in older people goes, Alzheimer’s disease (AD) is by far the most prevalent form. For over fifty years, medical and exclusion criteria were used to diagnose AD, with an accuracy of only 85 per cent. This did not allow for a correct diagnosis, which could be validated only through postmortem examination. Diagnosis of AD can be sped up, and the course of the disease can be predicted by applying machine learning (ML) techniques to Magnetic Resonance Imaging (MRI) techniques. Dementia in specific seniors could be predicted using data from AD screenings and ML classifiers. Classifier performance for AD subjects can be enhanced by including demographic information from the MRI and the patient’s preexisting conditions. In this article, we have used the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset. In addition, we proposed a framework for the AD/non-AD classification of dementia patients using longitudinal brain MRI features and Deep Belief Network (DBN) trained with the Mayfly Optimization Algorithm (MOA). An IoT-enabled portable MR imaging device is used to capture real-time patient MR images and identify anomalies in MRI scans to detect and classify AD. Our experiments validate that the predictive power of all models is greatly enhanced by including early information about comorbidities and medication characteristics. The random forest model outclasses other models in terms of precision. This research is the first to examine how AD forecasting can benefit from using multimodal time-series data. The ability to distinguish between healthy and diseased patients is demonstrated by the DBN-MOA accuracy of 97.456%, f-Score of 93.187 %, recall of 95.789 % and precision of 94.621% achieved by the proposed technique. The experimental results of this research demonstrate the efficacy, superiority, and applicability of the DBN-MOA algorithm developed for the purpose of AD diagnosis.

Explainable machine learning models based on multimodal time-series data for the early detection of Parkinson’s disease

Background and objectivesParkinson’s Disease (PD) is a devastating chronic neurological condition. Machine learning (ML) techniques have been used in the early prediction of PD progression. Fusion of heterogeneous data modalities proved its capability to improve the performance of ML models. Time series data fusion supports the tracking of the disease over time. In addition, the trustworthiness of the resulting models is improved by adding model explainability features. The literature on PD has not sufficiently explored these three points. MethodsIn this work, we proposed an ML pipeline for predicting the progression of PD that is both accurate and explainable. We explore the fusion of different combinations of five time series modalities from the Parkinson’s Progression Markers Initiative (PPMI) real-world dataset, including patient characteristics, biosamples, medication history, motor, and non-motor function data. Each patient has six visits. The problem has been formulated in two ways: ❶ a three-class based progression prediction with 953 patients in each time series modality, and ❷ a four-class based progression prediction with 1,060 patients in each time series modality. The statistical features of these six visits were calculated from each modality and diverse feature selection methods were applied to select the most informative feature sets. The extracted features were used to train a set of well-known ML models including Support vector machines (SVM), random forests (RF), extra tree classifier (ETC), light gradient boosting machines (LGBM), and stochastic gradient descent (SGD). We examined a number of data-balancing strategies in the pipeline with different combinations of modalities. ML models have been optimized using the Bayesian optimizer. A comprehensive evaluation of various ML methods has been conducted, and the best models have been extended to provide different explainability features. ResultsWe compare the performance of ML models before and after optimization and using and without using feature selection. In the three-class experiment and with various modality fusions, the LGBM model produced the most accurate results with a 10-fold cross-validation (10-CV) accuracy of 90.73% using non-motor function modality. RF produced the best results in the four-class experiment with various modality fusions with a 10-CV accuracy of 94.57% using non-motor modality. With the fused dataset of non-motor and motor function modalities, the LGBM model outperformed the other ML models in both the 3-class and 4-class experiments (i.e., 10-CV accuracy of 94.89% and 93.73%, respectively). Using the Shapely Additive Explanations (SHAP) framework, we employed global and instance-based explanations to explain the behavior of each ML classifier. Moreover, we extended the explainability by implementing the LIME and SHAPASH local explainers. The consistency of these explainers has been explored. The resultant classifiers were accurate, explainable, and thus medically more relevant and applicable. ConclusionsThe select modalities and feature sets were confirmed by the literature and medical experts. The various explainers suggest that the bradykinesia (NP3BRADY) feature was the most dominant and consistent. By providing thorough insights into the influence of multiple modalities on the disease risk, the suggested approach is expected to help improve the clinical knowledge of PD progression processes.

Multimodal Few-Shot Target Detection Based on Uncertainty Analysis in Time-Series Images

The ability to interpret multimodal data, and map the targets and anomalies within, is important for an automatic recognition system. Due to the expensive and time-consuming nature of multimodal time-series data annotation in the training stage, multimodal time-series image understanding, from drone and quadruped mobile robot platforms, is a challenging task for remote sensing and photogrammetry. In this regard, robust methods must be computationally low-cost, due to the limited data on aerial and ground-based platforms, yet accurate enough to meet certainty measures. In this study, a few-shot learning architecture, based on a squeeze-and-attention structure, is proposed for multimodal target detection, using time-series images from the drone and quadruped robot platforms with a small training dataset. To build robust algorithms in target detection, a squeeze-and-attention structure has been developed from multimodal time-series images from limited training data as an optimized method. The proposed architecture was validated on three datasets with multiple modalities (e.g., red-green-blue, color-infrared, and thermal), achieving competitive results.

Spatio-Temporal Wildfire Prediction Using Multi-Modal Data

Due to severe societal and environmental impacts, wildfire prediction using multi-modal sensing data has become a highly sought-after data-analytical tool by various stakeholders (such as state governments and power utility companies) to achieve a more informed understanding of wildfire activities and plan preventive measures. A desirable algorithm should precisely predict fire risk and magnitude for a location in real time. In this paper, we develop a flexible spatio-temporal wildfire prediction framework using multi-modal time series data. We first predict the wildfire risk (the chance of a wildfire event) in real-time, considering the historical events using discrete mutually exciting point process models. Then we further develop a wildfire magnitude prediction set method based on the flexible distribution-free time-series conformal prediction (CP) approach. Theoretically, we prove a risk model parameter recovery guarantee, as well as coverage and set size guarantees for the CP sets. Through extensive real-data experiments with wildfire data in California, we demonstrate the effectiveness of our methods, as well as their flexibility and scalability in large regions.

Automatic detection of Alzheimer’s disease progression: An efficient information fusion approach with heterogeneous ensemble classifiers

Predicting Alzheimer’s disease (AD) progression is crucial for improving the management of this chronic disease. Usually, data from AD patients are multimodal and time series in nature. This study proposes a novel ensemble learning framework for AD progression incorporating heterogeneous base learners into an integrated model using the stacking technique. This framework is used to build a 4-class ensemble classifier, which predicts AD progression 2.5 years in the future based on the multimodal time-series data. Statistical measures have been extracted from the longitudinal data to be used by the conventional machine learning models. The examined ensemble members include k-nearest neighbor, extreme gradient boosting, support vector machine, random forest, decision tree, and multilayer perceptron. We utilize three time-series modalities and one static non-time series modality of 1371 subjects from the Alzheimer’s disease neuroimaging initiative (ADNI) to validate our model. Several homogeneous and heterogeneous combinations of ensemble members were implemented, and their performance compared. The balance between accuracy and diversity when selecting ensemble members was investigated. We found that both accuracy and diversity are equally critical metrics to obtain an optimal ensemble model. Furthermore, our testing showed that the proposed model achieves outstanding progression prediction performance. The proposed model achieved a high performance without using neuroimaging data, which means that the model could be implemented in low-cost healthcare environments. The proposed model has achieved superior results compared with the state-of-the-art techniques in Alzheimer’s and ensemble classifiers domains. The proposed framework can be used to implement efficient information fusion ensembles for other medical and non-medical problems.

Boundary-Focused Generative Adversarial Networks for Imbalanced and Multimodal Time Series

Class imbalance problems have been reported as a major issue in various applications. Classification becomes further complicated when an imbalance occurs in time series data sets. To address time series data, it is necessary to consider their characteristics (i.e., high dimensionality, high correlations, and multimodality). Oversampling is a well-known approach for addressing this problem; however, such an approach does not appropriately consider the characteristics of time series data. This paper addresses these limitations by presenting a model-based oversampling approach, a boundary-focused generative adversarial network (BFGAN). The proposed BFGAN employs a specifically designed additional label for reflecting the importance of a sample's position in data space. Furthermore, the BFGAN generates artificial samples after taking into consideration a sample's multimodality and importance by using a suitable modified GAN structure. We present empirical results that reveal a significant improvement in the quality of the generated data when the proposed BFGAN is used as an oversampling algorithm for an imbalanced multimodal time series data set.

Towards a wearable multi-modal seizure detection system in epilepsy: A pilot study

ObjectiveTo explore the possibilities of wearable multi-modal monitoring in epilepsy and to identify effective strategies for seizure-detection. MethodsThirty patients with suspected epilepsy admitted to video electroencephalography (EEG) monitoring were equipped with a wearable multi-modal setup capable of continuous recording of electrocardiography (ECG), accelerometry (ACM) and behind-the-ear EEG. A support vector machine (SVM) algorithm was trained for cross-modal automated seizure detection. Visualizations of multi-modal time series data were used to generate ideas for seizure detection strategies. ResultsThree patients had more than five seizures and were eligible for SVM classification. Classification of 47 focal tonic seizures in one patient found a sensitivity of 84% with a false alarm rate (FAR) of 8/24 h. In two patients each with nine focal nonmotor seizures it yielded a sensitivity of 100% and a FAR of 13/24 h and 5/24. Visual comparisons of features were used to identify strategies for seizure detection in future research. ConclusionsMulti-modal monitoring in epilepsy using wearables is feasible and automatic seizure detection may benefit from multiple modalities when compared to uni-modal EEG. SignificanceThis study is unique in exploring a combination of wearable EEG, ECG and ACM and can help inform future research on monitoring of epilepsy.

Multi-Channel Fusion Classification Method Based on Time-Series Data.

Time-series data generally exists in many application fields, and the classification of time-series data is one of the important research directions in time-series data mining. In this paper, univariate time-series data are taken as the research object, deep learning and broad learning systems (BLSs) are the basic methods used to explore the classification of multi-modal time-series data features. Long short-term memory (LSTM), gated recurrent unit, and bidirectional LSTM networks are used to learn and test the original time-series data, and a Gramian angular field and recurrence plot are used to encode time-series data to images, and a BLS is employed for image learning and testing. Finally, to obtain the final classification results, Dempster–Shafer evidence theory (D–S evidence theory) is considered to fuse the probability outputs of the two categories. Through the testing of public datasets, the method proposed in this paper obtains competitive results, compensating for the deficiencies of using only time-series data or images for different types of datasets.

Tracking agitation in people living with dementia in a care environment

Agitation is a symptom that communicates distress in people living with dementia (PwD), and that can place them and others at risk. In a long term care (LTC) environment, care staff track and document these symptoms as a way to detect when there has been a change in resident status to assess risk, and to monitor for response to interventions. However, this documentation can be time-consuming, and due to staffing constraints, episodes of agitation may go unobserved. This brings into question the reliability of these assessments, and presents an opportunity for technology to help track and monitor behavioural symptoms in dementia. In this paper, we present the outcomes of a 2 year real-world study performed in a dementia unit, where a multi-modal wearable device was worn by 20 PwD. In line with a commonly used clinical documentation tool, this large multi-modal time-series data was analyzed to track the presence of episodes of agitation in 8-hour nursing shifts. The development of a baseline classification model (AUC=0.717) on this dataset and subsequent improvement (AUC= 0.779) lays the groundwork for automating the process of annotating agitation events in nursing charts.

Automatic identification of differences in behavioral co-occurrence between groups

BackgroundRecent advances in sensing technologies have enabled us to attach small loggers to animals in their natural habitat. It allows measurement of the animals’ behavior, along with associated environmental and physiological data and to unravel the adaptive significance of the behavior. However, because animal-borne loggers can now record multi-dimensional (here defined as multimodal) time series information from a variety of sensors, it is becoming increasingly difficult to identify biologically important patterns hidden in the high-dimensional long-term data. In particular, it is important to identify co-occurrences of several behavioral modes recorded by different sensors in order to understand an internal hidden state of an animal because the observed behavioral modes are reflected by the hidden state. This study proposed a method for automatically detecting co-occurrence of behavioral modes that differs between two groups (e.g., males vs. females) from multimodal time-series sensor data. The proposed method first extracted behavioral modes from time-series data (e.g., resting and cruising modes in GPS trajectories or relaxed and stressed modes in heart rates) and then identified two different behavioral modes that were frequently co-occur (e.g., co-occurrence of the cruising mode and relaxed mode). Finally, behavioral modes that differ between the two groups in terms of the frequency of co-occurrence were identified.ResultsWe demonstrated the effectiveness of our method using animal-locomotion data collected from male and female Streaked Shearwaters by showing co-occurrences of locomotion modes and diving behavior recorded by GPS and water-depth sensors. For example, we found that the behavioral mode of high-speed locomotion and that of multiple dives into the sea were highly correlated in male seabirds. In addition, compared to the naive method, the proposed method reduced the computation costs by about 99.9%.ConclusionBecause our method can automatically mine meaningful behavioral modes from multimodal time-series data, it can be potentially applied to analyzing co-occurrences of locomotion modes and behavioral modes from various environmental and physiological data.

Forecasting office indoor CO2 concentration using machine learning with a one-year dataset

Modern buildings are expected to fulfill energy efficiency regulations while providing a healthy and comfortable living environment for their occupants. Demand-controlled heating, ventilation, and air conditioning can improve energy efficiency and well-being, especially if changes in indoor environmental factors can be forecast reliably enough. As a first step, this study afforded a baseline for CO2 forecasting evaluation by providing for the public use a comprehensive dataset covering a full year. Secondly, we studied the applicability of four machine learning methods, Ridge regression, Decision Tree, Random Forest, and Multilayer Perceptron, for modeling the future concentration of CO2 in indoors. We evaluated their prediction accuracy within different forecasting and history window time frames and the impact of multiple sensor modalities. All models performed better than the baseline method of predicting the last observed value, and the Decision Tree was found to be almost as accurate as the computationally heavier Random Forest model. When the future forecasting window was longer than a minute, the optimal sensor modalities included occupant activity data from passive infrared sensors in addition to CO2 concentration. Our findings suggest that machine learning can be applied in multimodal time-series data to find a simple, accurate, and resource-efficient forecasting model for proactive control of indoor environments.

Alzheimer’s disease progression detection model based on an early fusion of cost-effective multimodal data

Alzheimer’s disease (AD) is a severe neurodegenerative disease. The identification of patients at high risk of conversion from mild cognitive impairment to AD via earlier close monitoring, targeted investigations, and appropriate management is crucial. Recently, several machine learning (ML) algorithms have been used for AD progression detection. Most of these studies only utilized neuroimaging data from baseline visits. However, AD is a complex chronic disease, and usually, a medical expert will analyze the patient’s whole history when making a progression diagnosis. Furthermore, neuroimaging data are always either limited or not available, especially in developing countries, due to their cost. In this paper, we compare the performance of five widely used ML algorithms, namely, the support vector machine, random forest, k-nearest neighbor, logistic regression, and decision tree to predict AD progression with a prediction horizon of 2.5 years. We use 1029 subjects from the Alzheimer’s disease neuroimaging initiative (ADNI) database. In contrast to previous literature, our models are optimized using a collection of cost-effective time-series features including patient’s comorbidities, cognitive scores, medication history, and demographics. Medication and comorbidity text data are semantically prepared. Drug terms are collected and cleaned before encoding using the therapeutic chemical classification (ATC) ontology, and then semantically aggregated to the appropriate level of granularity using ATC to ensure a less sparse dataset. Our experiments assert that the early fusion of comorbidity and medication features with other features reveals significant predictive power with all models. The random forest model achieves the most accurate performance compared to other models. This study is the first of its kind to investigate the role of such multimodal time-series data on AD prediction.

Multimodal multitask deep learning model for Alzheimer’s disease progression detection based on time series data

Early prediction of Alzheimer’s disease (AD) is crucial for delaying its progression. As a chronic disease, ignoring the temporal dimension of AD data affects the performance of a progression detection and medically unacceptable. Besides, AD patients are represented by heterogeneous, yet complementary, multimodalities. Multitask modeling improves progression-detection performance, robustness, and stability. However, multimodal multitask modeling has not been evaluated using time series and deep learning paradigm, especially for AD progression detection. In this paper, we propose a robust ensemble deep learning model based on a stacked convolutional neural network (CNN) and a bidirectional long short-term memory (BiLSTM) network. This multimodal multitask model jointly predicts multiple variables based on the fusion of five types of multimodal time series data plus a set of background (BG) knowledge. Predicted variables include AD multiclass progression task, and four critical cognitive scores regression tasks. The proposed model extracts local and longitudinal features of each modality using a stacked CNN and BiLSTM network. Concurrently, local features are extracted from the BG data using a feed-forward neural network. Resultant features are fused to a deep network to detect common patterns which jointly used to predict the classification and regression tasks. To validate our model, we performed six experiments on five modalities from Alzheimer’s Disease Neuroimaging Initiative (ADNI) of 1536 subjects. The results of the proposed approach achieve state-of-the-art performance for both multiclass progression and regression tasks. Moreover, our approach can be generalized in other medial domains to analyze heterogeneous temporal data for predicting patient’s future status.

IRIS: A Modular Platform for Continuous Monitoring and Caretaker Notification in the Intensive Care Unit.

New approaches are needed to interpret large amounts of physiologic data continuously recorded in the ICU. We developed and prospectively validated a versatile platform (IRIS) for real-time ICU physiologic monitoring, clinical decision making, and caretaker notification. IRIS was implemented in the neurointensive care unit to stream multimodal time series data, including EEG, intracranial pressure (ICP), and brain tissue oxygenation (PbtO2), from ICU monitors to an analysis server. IRIS was applied for 364 patients undergoing continuous EEG, 26 patients undergoing burst suppression monitoring, and four patients undergoing intracranial pressure and brain tissue oxygen monitoring. Custom algorithms were used to identify periods of elevated ICP, compute burst suppression ratios (BSRs), and detect faulty or disconnected EEG electrodes. Hospital staff were notified of clinically relevant events using our secure API to route alerts through a password-protected smartphone application. Sustained increases in ICP and concordant decreases in PbtO2 were reliably detected using user-defined thresholds and alert throttling. BSR trends computed by the platform correlated highly with manual neurologist markings (r20.633-0.781; p < 0.0001). The platform identified EEG electrodes with poor signal quality with 95% positive predictive value, and reduced latency of technician response by 93%. This study validates a flexible real-time platform for monitoring and interpreting ICU data and notifying caretakers of actionable results, with potential to reduce the manual burden of continuous monitoring services on care providers. This work represents an important step toward facilitating translational medical data analytics to improve patient care and reduce health care costs.

MagicSox: An E-textile IoT system to quantify gait abnormalities

Abstract The global society is increasingly facing the challenges that reduce mobility, quality of life, and independence. Gait disorders are often both a result of, and predictor of further issues, tied to the 15 million stroke patients annually world-wide. These individuals face a number of gait abnormalities including drop foot that is a pathological condition, limiting patients' ability to lift the foot from the ground during the swing phase of walking. In this research work, we introduce a novel smart textile system, MagicSox that is woven with multiple sensors distributed over the surface of the foot. The overarching goal of MagicSox is to quantify the gait abnormalities in remote settings such as patients' homes so that clinicians and physical therapists can assess their patients on daily basis. The paper provides a detailed architecture of MagicSox that leverages the computing and communication capabilities of a modern Internet of Things (IoT) processor, the Intel Curie. We have developed an Android smart phone app that uses Bluetooth low energy (BLE) and automates the multi-sensor data collection from MagicSox. In terms of signal processing of wearable sensor data, we adopted multiplication of backward differences (MOBD) to analyze the multi-modal time series data to distinguish drop foot events from normal walking cycles. We pursued a usability study on 12 healthy participants who were asked to walk normally and also to simulate drop foot cycles. We developed support vector machine (SVM) classifiers to analyze the data. The classification resulted in the accuracy of drop foot detection varying from 73.38 % − 99.02 % . The promising results now encourage us to evaluate MagicSox on stroke patients in future studies.

Deep Dynamic Neural Networks for Multimodal Gesture Segmentation and Recognition.

This paper describes a novel method called Deep Dynamic Neural Networks (DDNN) for multimodal gesture recognition. A semi-supervised hierarchical dynamic framework based on a Hidden Markov Model (HMM) is proposed for simultaneous gesture segmentation and recognition where skeleton joint information, depth and RGB images, are the multimodal input observations. Unlike most traditional approaches that rely on the construction of complex handcrafted features, our approach learns high-level spatio-temporal representations using deep neural networks suited to the input modality: a Gaussian-Bernouilli Deep Belief Network (DBN) to handle skeletal dynamics, and a 3D Convolutional Neural Network (3DCNN) to manage and fuse batches of depth and RGB images. This is achieved through the modeling and learning of the emission probabilities of the HMM required to infer the gesture sequence. This purely data driven approach achieves a Jaccard index score of 0.81 in the ChaLearn LAP gesture spotting challenge. The performance is on par with a variety of state-of-the-art hand-tuned feature-based approaches and other learning-based methods, therefore opening the door to the use of deep learning techniques in order to further explore multimodal time series data.