Processing Of Multimodal Data Research Articles

Mirror Mirror on the Wall... An Unobtrusive Intelligent Multisensory Mirror for Well-Being Status Self-Assessment and Visualization

A person's well-being status is reflected by their face through a combination of facial expressions and physical signs. The SEMEOTICONS project translates the semeiotic code of the human face into measurements and computational descriptors that are automatically extracted from images, videos, and three-dimensional scans of the face. SEMEOTICONS developed a multisensory platform in the form of a smart mirror to identify signs related to cardio-metabolic risk. The aim was to enable users to self-monitor their well-being status over time and guide them to improve their lifestyle. Significant scientific and technological challenges have been addressed to build the multisensory mirror, from touchless data acquisition, to real-time processing and integration of multimodal data.

Time-Of-Flight Camera, Optical Tracker and Computed Tomography in Pairwise Data Registration.

PurposeA growing number of medical applications, including minimal invasive surgery, depends on multi-modal or multi-sensors data processing. Fast and accurate 3D scene analysis, comprising data registration, seems to be crucial for the development of computer aided diagnosis and therapy. The advancement of surface tracking system based on optical trackers already plays an important role in surgical procedures planning. However, new modalities, like the time-of-flight (ToF) sensors, widely explored in non-medical fields are powerful and have the potential to become a part of computer aided surgery set-up. Connection of different acquisition systems promises to provide a valuable support for operating room procedures. Therefore, the detailed analysis of the accuracy of such multi-sensors positioning systems is needed.MethodsWe present the system combining pre-operative CT series with intra-operative ToF-sensor and optical tracker point clouds. The methodology contains: optical sensor set-up and the ToF-camera calibration procedures, data pre-processing algorithms, and registration technique. The data pre-processing yields a surface, in case of CT, and point clouds for ToF-sensor and marker-driven optical tracker representation of an object of interest. An applied registration technique is based on Iterative Closest Point algorithm.ResultsThe experiments validate the registration of each pair of modalities/sensors involving phantoms of four various human organs in terms of Hausdorff distance and mean absolute distance metrics. The best surface alignment was obtained for CT and optical tracker combination, whereas the worst for experiments involving ToF-camera.ConclusionThe obtained accuracies encourage to further develop the multi-sensors systems. The presented substantive discussion concerning the system limitations and possible improvements mainly related to the depth information produced by the ToF-sensor is useful for computer aided surgery developers.

Kernel Manifold Alignment for Domain Adaptation.

The wealth of sensory data coming from different modalities has opened numerous opportunities for data analysis. The data are of increasing volume, complexity and dimensionality, thus calling for new methodological innovations towards multimodal data processing. However, multimodal architectures must rely on models able to adapt to changes in the data distribution. Differences in the density functions can be due to changes in acquisition conditions (pose, illumination), sensors characteristics (number of channels, resolution) or different views (e.g. street level vs. aerial views of a same building). We call these different acquisition modes domains, and refer to the adaptation problem as domain adaptation. In this paper, instead of adapting the trained models themselves, we alternatively focus on finding mappings of the data sources into a common, semantically meaningful, representation domain. This field of manifold alignment extends traditional techniques in statistics such as canonical correlation analysis (CCA) to deal with nonlinear adaptation and possibly non-corresponding data pairs between the domains. We introduce a kernel method for manifold alignment (KEMA) that can match an arbitrary number of data sources without needing corresponding pairs, just few labeled examples in all domains. KEMA has interesting properties: 1) it generalizes other manifold alignment methods, 2) it can align manifolds of very different complexities, performing a discriminative alignment preserving each manifold inner structure, 3) it can define a domain-specific metric to cope with multimodal specificities, 4) it can align data spaces of different dimensionality, 5) it is robust to strong nonlinear feature deformations, and 6) it is closed-form invertible, which allows transfer across-domains and data synthesis. To authors’ knowledge this is the first method addressing all these important issues at once. We also present a reduced-rank version of KEMA for computational efficiency, and discuss the generalization performance of KEMA under Rademacher principles of stability. Aligning multimodal data with KEMA reports outstanding benefits when used as a data pre-conditioner step in the standard data analysis processing chain. KEMA exhibits very good performance over competing methods in synthetic controlled examples, visual object recognition and recognition of facial expressions tasks. KEMA is especially well-suited to deal with high-dimensional problems, such as images and videos, and under complicated distortions, twists and warpings of the data manifolds. A fully functional toolbox is available at https://github.com/dtuia/KEMA.git.

Deep learning human actions from video via sparse filtering and locally competitive algorithms

Physiological and psychophysical evidence suggest that early visual cortex compresses the visual input on the basis of spatial and orientation-tuned filters. Recent computational advances have suggested that these neural response characteristics may reflect a `sparse coding' architecture--in which a small number of neurons need to be active for any given image--yielding critical structure latent in natural scenes. Here we present a novel neural network architecture combining a sparse filter model and locally competitive algorithms (LCAs), and demonstrate the network's ability to classify human actions from video. Sparse filtering is an unsupervised feature learning algorithm designed to optimize the sparsity of the feature distribution directly without having the need to model the data distribution. LCAs are defined by a system of differential equations where the initial conditions define an optimization problem and the dynamics converge to a sparse decomposition of the input vector. We applied this architecture to train a classifier on categories of motion in human action videos. Inputs to the network were small 3D patches taken from frame differences in the videos. Dictionaries were derived for each action class and then activation levels for each dictionary were assessed during reconstruction of a novel test patch. Overall, classification accuracy was at ? 97 %. We discuss how this sparse filtering approach provides a natural framework for multi-sensory and multimodal data processing including RGB video, RGBD video, hyper-spectral video, and stereo audio/video streams.

Detection of heart beats in multimodal data: a robust beat-to-beat interval estimation approach

The heart rate and its variability play a vital role in the continuous monitoring of patients, especially in the critical care unit. They are commonly derived automatically from the electrocardiogram as the interval between consecutive heart beat. While their identification by QRS-complexes is straightforward under ideal conditions, the exact localization can be a challenging task if the signal is severely contaminated with noise and artifacts. At the same time, other signals directly related to cardiac activity are often available. In this multi-sensor scenario, methods of multimodal sensor-fusion allow the exploitation of redundancies to increase the accuracy and robustness of beat detection.In this paper, an algorithm for the robust detection of heart beats in multimodal data is presented. Classic peak-detection is augmented by robust multi-channel, multimodal interval estimation to eliminate false detections and insert missing beats. This approach yielded a score of 90.70 and was thus ranked third place in the PhysioNet/Computing in Cardiology Challenge 2014: Robust Detection of Heart Beats in Muthmodal Data follow-up analysis.In the future, the robust beat-to-beat interval estimator may directly be used for the automated processing of multimodal patient data for applications such as diagnosis support and intelligent alarming.

The classification of multi-modal data with hidden conditional random field

The classification of multi-modal data has been an active research topic in recent years. It has been used in many applications where the processing of multi-modal data is involved. Motivated by the assumption that different modalities in multi-modal data share latent structure (topics), this paper attempts to learn the shared structure by exploiting the symbiosis of multiple-modality and therefore boost the classification of multi-modal data, we call it Multi-modal Hidden Conditional Random Field (M-HCRF). M-HCRF represents the intrinsical structure shared by different modalities as hidden variables in a undirected general graphical model. When learning the latent shared structure of the multi-modal data, M-HCRF can discover the interactions among the hidden structure and the supervised category information. The experimental results show the effectiveness of our proposed M-HCRF when applied to the classification of multi-modal data.

Analyse multimodale de l’agir professoral et degré de granularité de traitement. Réflexions méthodologiques

L’agir professoral étant par nature multimodal, son analyse requiert une démarche appropriée pour prendre en compte autant que possible le non verbal. Or, une analyse qui se veut multimodale interroge nécessairement la granularité du traitement des données, la finesse d’analyse des interactions et le nombre des phénomènes à considérer.À partir d’un exemple d’analyse tiré de notre corpus d’interactions didactiques, il s’agira de réfléchir au degré de finesse de traitement des données et aux garde-fous méthodologiques à poser pour conserver une compréhension globale de l’interaction ou pour éviter de surinterpréter les données.

Augmented Affective-Cognition for Usability Study of In-Vehicle System User Interface

Usability of in-vehicle systems has become increasingly important for ease of operations and safety of driving. The user interface (UI) of in-vehicle systems is a critical focus of usability study. This paper studies how to use advanced computational, physiology- and behavior-based tools and methodologies to determine affective/emotional states and behavior of an individual in real time and in turn how to adapt the human-vehicle interaction to meet users' cognitive needs based on the real-time assessment. Specifically, we set up a set of physiological sensors that are capable of collecting EEG, facial EMG, skin conductance response, and respiration data and a set of motion sensing and tracking equipment that is capable of capturing eye ball movement and objects which the user is interacting with. All hardware components and software are integrated into an augmented sensor platform that can perform as “one coherent system” to enable multimodal data processing and information inference for context-aware analysis of emotional states and cognitive behavior based on the rough set inference engine. Meanwhile subjective data are also recorded for comparison. A usability study of in-vehicle system UI is shown to demonstrate the potential of the proposed methodology.

Cardiac Interventional Guidance using Multimodal Data Processing and Visualisation: medInria as an Interoperability Platform

medInria is a free medical imaging software developed at Inria, which aims at providing clinicians with state-of-the-art algorithms dedicated to medical image processing and visualization. Efforts have been made to simplify the user interface, while keeping high-level algorithms. In this particular article, we will concentrate on its use in preoperative preparation for cardiac interventions, and how we handle the difficulties arising from the lack of standard format for data types such as meshes or fibers, the absence of a common programming interface for data processing algorithms, notably registration, and the issues of visualisation where display conventions would be beneficial.

Breast Image Registration and Deformation Modeling

Image-based examination of the breast facilitates the detection of breast diseases, particularly of present benign and malignant lesions. For computer-aided processing of serial and multimodal clinical data, both for visual correlation and quantitative analysis, automated image-registration methods are an indispensable tool. The wide range of modalities and the high variability of breast appearance have led to a large diversity of proposed approaches for tissue deformation modeling and image registration. In this article, we review current developments in breast image registration techniques, and comment on their clinical relevance, individual capabilities, and open challenges.

Spontaneous brain activity and EEG microstates. A novel EEG/fMRI analysis approach to explore resting-state networks

The brain is active even in the absence of explicit input or output as demonstrated from electrophysiological as well as imaging studies. Using a combined approach we measured spontaneous fluctuations in the blood oxygen level dependent (BOLD) signal along with electroencephalography (EEG) in eleven healthy subjects during relaxed wakefulness (eyes closed). In contrast to other studies which used the EEG frequency information to guide the functional MRI (fMRI) analysis, we opted for transient EEG events, which identify and quantify brain electric microstates as time epochs with quasi-stable field topography. We then used this microstate information as regressors for the BOLD fluctuations. Single trial EEGs were segmented with a specific module of the LORETA (low resolution electromagnetic tomography) software package in which microstates are represented as normalized vectors constituted by scalp electric potentials, i.e., the related 3-dimensional distribution of cortical current density in the brain. Using the occurrence and the duration of each microstate, we modeled the hemodynamic response function (HRF) which revealed BOLD activation in all subjects. The BOLD activation patterns resembled well known resting-state networks (RSNs) such as the default mode network. Furthermore we “cross validated” the data performing a BOLD independent component analysis (ICA) and computing the correlation between each ICs and the EEG microstates across all subjects. This study shows for the first time that the information contained within EEG microstates on a millisecond timescale is able to elicit BOLD activation patterns consistent with well known RSNs, opening new avenues for multimodal imaging data processing.

Learning Multimodal Dictionaries

Real-world phenomena involve complex interactions between multiple signal modalities. As a consequence, humans are used to integrate at each instant perceptions from all their senses in order to enrich their understanding of the surrounding world. This paradigm can be also extremely useful in many signal processing and computer vision problems involving mutually related signals. The simultaneous processing of multimodal data can, in fact, reveal information that is otherwise hidden when considering the signals independently. However, in natural multimodal signals, the statistical dependencies between modalities are in general not obvious. Learning fundamental multimodal patterns could offer deep insight into the structure of such signals. In this paper, we present a novel model of multimodal signals based on their sparse decomposition over a dictionary of multimodal structures. An algorithm for iteratively learning multimodal generating functions that can be shifted at all positions in the signal is proposed, as well. The learning is defined in such a way that it can be accomplished by iteratively solving a generalized eigenvector problem, which makes the algorithm fast, flexible, and free of user-defined parameters. The proposed algorithm is applied to audiovisual sequences and it is able to discover underlying structures in the data. The detection of such audio-video patterns in audiovisual clips allows to effectively localize the sound source on the video in presence of substantial acoustic and visual distractors, outperforming state-of-the-art audiovisual localization algorithms.

Evaluación de la cognición social y teoría de la mente en pacientes con enfermedad cerebelosa degenerativa aislada no dementes

To investigate whether the cerebellum could participate in social cognition (SC). General neuropsychological tests, executive tests (EF), social cognition tests, which assess the ability to infer other peoples mental states, and the Beck Depression Inventory were given to 10 non-demented patients with isolated cerebellar degenerative disease, and to 10 healthy controls matched for sex, age, and years of education. ANOVA and correlation coefficients were employed for the statistical analysis. Patients within the cerebellar group were significantly impaired (p<or=0.05) in EF test [Wisconsin Card Sorting Test (WSCT)] and belief questions (BQ) from Theory of Mind (ToM) tests. Performance in control questions (CQ) from ToM tests was similar for both groups. Lower scores in BQ correlated with a lower conceptual ability, the severity of apathy (NPI) and static ataxia. CQ correlated with measures of attention and free recall. The cerebellum may contribute in the control of social behavior through the processing of multimodal data, motor, cognitive and emotional.

A virtual reality patient simulation system for teaching emergency response skills to U.S. Navy medical providers.

Rapid and effective medical intervention in response to civil and military-related disasters is crucial for saving lives and limiting long-term disability. Inexperienced providers may suffer in performance when faced with limited supplies and the demands of stabilizing casualties not generally encountered in the comparatively resource-rich hospital setting. Head trauma and multiple injury cases are particularly complex to diagnose and treat, requiring the integration and processing of complex multimodal data. In this project, collaborators adapted and merged existing technologies to produce a flexible, modular patient simulation system with both three-dimensional virtual reality and two-dimensional flat screen user interfaces for teaching cognitive assessment and treatment skills. This experiential, problem-based training approach engages the user in a stress-filled, high fidelity world, providing multiple learning opportunities within a compressed period of time and without risk. The system simulates both the dynamic state of the patient and the results of user intervention, enabling trainees to watch the virtual patient deteriorate or stabilize as a result of their decision-making speed and accuracy. Systems can be deployed to the field enabling trainees to practice repeatedly until their skills are mastered and to maintain those skills once acquired. This paper describes the technologies and the process used to develop the trainers, the clinical algorithms, and the incorporation of teaching points. We also characterize aspects of the actual simulation exercise through the lens of the trainee.

NeuroNet: collaborative intraoperative guidance and control

Neurophysiological monitoring assesses CNS structure function relationships during surgery. NeuroNet supports remote performance of this task through real time multimodal data processing and multimedia network communication. The system is fully integrated, transparently combining the collection, processing, and presentation of real time data sources, including all physiological monitoring functions, with non real time functions and extensive online database information. Workstations are mounted in instrumentation racks and configured with appropriate electronics to support various data acquisition tasks including electroencephalograms (EEGs), electromyograms (EMGs), and multimodality evoked potentials. Multiple racks can be used in parallel on the same case if the number of variables to be monitored exceeds the capacity of a single tack. The data acquired on these systems is transparently accessible, in real time, across the network for both review and analysis.