Processing Of Multimodal Data Research Articles

MODELS OF COMPUTATION FOR DIGITAL TWINS DATA PROCESSING

Background. The digital twin is a virtual model of a physical object or process (physical twin) which fully reflects its characteristics in dynamics over a period of time. The concept of digital twin involves the representation, processing, manipulation of multimodal data which characterizes the physical twin. The digital twin data processing requires new models of computation in order to simplify synchronization and aggregation of multimodal data for both behaviour and appearance models of the digital twin.Objective. The purpose of the research presented in this paper is to develop models of computation for digital twins data processing as well as to fulfil practical implementation of multimodal data synchronization and aggregation for e-health applications.Methods. The basic computation model is based on the concept of a multi-image which is considered as a data mathematical model. The notion of a multi-image is defined in the algebraic system of aggregates. In the basic computation model the multi-image is a key abstraction, which is defined at two levels: at the level of input data formation and at the level of formation of multimodal data structure to be processed. The advanced computation model for digital twins includes three stages of synchronization and aggregation, a stage of data processing of both the behavioural model and the appearance model according to the specific task of studying the real-world object (physical twin), and a stage of reproducing the digital twin.Results. The proposed computation schemes make it easy to synchronize and aggregate heterogeneous data entering into a computer system from multiple sources. The results obtained in the experimental part on this research allow concluding that the computational complexity of multimodal data processing is reduced in comparison with the traditional approach.Conclusions. Two computation models are proposed for digital twin data processing: a basic computation model and a computation advanced model for digital twins. The key component of these models is multimodal data synchronization and aggregation procedure. A mathematical apparatus of the algebraic system of aggregates is used in the presented research to simplify synchronization and aggregation of multimodal data defined in time.

MULTIMODAL DATA PROCESSING BASED ON ALGEBRAIC SYSTEM OF AGGREGATES RELATIONS

Context. In many engineering tasks where the monitoring of changes in the characteristics of an observation object, subject, or process is required, it is necessary to process multimodal data recorded with respect to time moments when these characteristics are registered. In this paper, the author presents a new approach to solving the problem of multimodal data structures timewise processing, which allows to simplify the processing of such data by using the mathematical apparatus of an algebraic system of aggregates and thereby reduce the requirements to computing resources. The algebraic system of aggregates operates with such specific data structures as aggregates and multi-images. These complex data structures can be obtained as a result of data measuring, generating, recording, etc. The processing of such multimodal data can also require discrete intervals processing. Objective. The goal of the work is to formalise the relations between basic mathematical objects defined in the algebraic system of aggregates, such as elements, tuples and aggregates, as well as the data structures based on these mathematical objects, namely, discreet intervals and multi-images. Method. The research presented in this paper is based on both the algebraic system of aggregates and the concept of multi-image which enable multimodal data timewise processing. A carrier of the algebraic system of aggregates is an arbitrary set of specific structures – aggregates. An aggregate is a tuple of arbitrary tuples, elements of which belong to predefined sets. Aggregates can be processed by using logical, ordering, and arithmetical operations defined in the algebraic system of aggregates. A multi-image is a non-empty aggregate, the first tuple of which is a tuple of time values. Such tuple of time values represents a certain discrete interval. To process discrete intervals and multi-images, a set of relations is defined in the algebraic system of aggregates. This set includes relations between tuple elements, relations between tuples, and relations between aggregates. The relations between tuples enable arithmetical comparison, frequency comparison, and interval comparison. This mathematical apparatus can be used for both complex representation of object (process) multimodal characteristics and further timewise processing of data represented as multi-images. Results. The approach to discrete intervals and multi-images processing based on relations, which are defined in the algebraic system of aggregates, has been developed and presented in the paper. The author provides examples of the developed approach practical implementation. Conclusions. The results obtained in the research presented in this paper has shown that the relations defined in algebraic system of aggregates enable processing of complex data structures named multi-images in data modelling, prediction and other tasks. To allow data processing with respect to time scale, discrete intervals can be employed. A discrete interval is a tuple of time values. In the paper, the author shows how relations for discrete intervals comparison can be used for solving practical tasks. Besides, the author presents the software tools which can be used for practical implementation of the given theoretical approach by employing the domain-specific language ASAMPL.

Deep interaction: Wearable robot-assisted emotion communication for enhancing perception and expression ability of children with Autism Spectrum Disorders

Recent changes in both the social economy and people’s living conditions and their habits have had an influence on the incidence of Autism Spectrum Disorder (ASD), which has brought huge economic and mental burden to the society , and has become an urgent public health problem. The main symptom of autism in children is the presence of a social barrier, and one of the main reasons for that is a lack of emotional cognitive ability. However, the existing autism-treatment systems intended for children pay little attention to the emotion cognition disorder. Besides, too much importance has been given to the interaction with children, while limiting the timeliness and movability of these systems. With the aim to address this shortcoming, in this work, we focus on the emotion cognition disorder and design a feasible solution for enhancing perception and expression ability of children with ASD. First, the first-view emotional care system for children with ASD (First-ECS) is developed using a wearable robot as a system carrier and realizing the deep emotional interaction with children with autism from the first-view perspective. Emotion communication is used to meet high timeliness requirements for emotion transmission in the First-ECS. Next, the emotional interaction mechanism that is applicable to the line of sight and non-line of sight communication scenarios is introduced. Also, the emotion perception engine and emotion expression engine are designed. Subsequently, multimodal data collection and processing by a wearable affective robot are discussed. In addition, this paper introduces a multimodal data fusion method from the angle of emotion relevance and emotion computing model based on audio-visual data. Finally, a demo platform is built to verify the feasibility of the proposed system.

Модель подання мультимодальних даних для комплексного опису об’єктів спостереження

The paper is devoted to solving the problem of the complex presentation of multimodal data about an object of observation, the characteristics of which are measured and investigated taking into account the time stamps and the relationship between the data of various modalities. This problem is relevant for many engineering tasks. In particular, the comprehensive presentation of data on a physical object is important in the technology of digital twins, which has recently become more widely used. Based on the fact that a digital twin is determined by both a behavioral model and a visual model, the paper presents a data model based on the concept of a multimodal data element. This model makes possible a comprehensive and consistent presentation of data about the object of observation. The mathematical model of the complex presentation of multimodal data is based on the concepts of an aggregate and a multi-image defined in an algebraic system of aggregates. The article analyzes the relationship between the complex presentation and processing of multimodal data, which are determined in time, using the apparatus of the algebraic system of aggregates and the consideration of aggregated data as functions of several variables. Thus, the analysis of the data of the muxel model can be performed using logical operations, ordering operations, and relations defined in the algebraic system of aggregates as well as operations and approaches defined in other mathematical concepts. Since the muxel model presents large amounts of data, it is advisable to optimize the way how data is presented. This can be achieved by data compression, modification of the data structure, or a combination of the first and second approaches. In the first approach, it is advisable to apply muxel model data compression based on the RLE method. In the second approach, it is proposed to apply data structures similar to the Sparse Voxel Octree, which is used to reduce data volume of models in voxel graphics. The practical implementation of the processing of multimodal data presented using the muxel model can be performed by means of ASAMPL programming language.

Real-Time EEG–EMG Human–Machine Interface-Based Control System for a Lower-Limb Exoskeleton

This article presents a rehabilitation technique based on a lower-limb exoskeleton integrated with a human-machine interface (HMI). HMI is used to record and process multimodal signals collected using a foot motor imagery (MI)-based brain-machine interface (BMI) and multichannel electromyographic (EMG) signals recorded from leg muscles. Current solutions of HMI-equipped rehabilitation assistive technologies tested under laboratory conditions demonstrated a great deal of success, but faced several difficulties caused by the limited accuracy of detecting MI electroencephalography (EEG) and the reliability of online control when executing a movement by patients dressed in an exoskeleton. In the case of lower-limb representation, there is still the problem of reliably distinguishing leg movement intentions and differentiating them in BMI systems. Targeting the design of a rehabilitation technique replicating the natural mode of motor control in exoskeleton walking patients, we have shown how the combined use of multimodal signals can improve the accuracy, performance, and reliability of HMI. The system was tested on healthy subjects operating the exoskeleton under different conditions. The study also resulted in algorithms of multimodal HMI data collection, processing, and classification. The developed system can analyze up to 15 signals simultaneously in real-time during a movement. Foot MI is extracted from EEG signals (seven channels) using the event-related (de)synchronization effect. Supplemented by EMG signals reflecting motor intention, the control system can initiate and differentiate the movement of the right and left legs with a high degree of reliability. The classification and control system permits one to work online when the exoskeleton is executing a movement.

Multimodal Data Processing Framework for Smart City: A Positional-Attention Based Deep Learning Approach

In the past few years, edge computing has brought tremendous convenience to the development of smart cities, releasing computation pressure to edge compute nodes. However, a series of problems, such as the explosive growth of smart devices and limited spectrum resources, still greatly limit the application of edge computing. Different types of end devices generate and collect multimodal information, and substantial data is transmitted to upper nodes. Multimodal machine learning methods process data at edge nodes, and only high-level features are uploaded to the cloud in order to save bandwidth. In this article, we propose a novel multimodal data processing framework based on multiple attention mechanisms. Two distinct attention mechanisms are used to capture inter and intra-modality dependencies and align different modalities together. We conduct experiments on image captioning, a core research hotspot in multimodal machine learning. A unified hierarchical structure extracts features from images and natural language. Matching attention aligns visual and textual information. Besides, we propose a new attention mechanism, positional attention, which finds the relationship of each element within one sensory modality. The hierarchical structure realizes parallel computation in the training phase and speeds up the training of the model. Experiments and analysis demonstrate significant improvements over baselines, proving the effectiveness of our method.

Semantics and pragmatics of programming language ASAMPL

This paper presents semantics and practical implementation of the domain-specific programming language ASAMPL. This programming language has been developed to support the efficient processing of multimodal data processing, in particular, the processing of multimedia content which components are evidently defined in terms of time. The data processing concept employed in ASAMPL is based on the data structures, operations, and relations defined in the algebraic system of aggregates. The paper explains the compilation approach used for this programming language as well as it presents the test results and their discussion. Problems in programming 2020; 1: 74-83

ІНТЕЛЕКТУАЛЬНА КОЛОРИЗАЦІЯ ЗОБРАЖЕНЬ ЗА ДОПОМОГОЮ ГЕНЕРАТИВНИХ ЗМАГАЛЬНИХ МЕРЕЖ

Статтю присвячено застосуванню генеративних змагальних нейронних мереж на прикладі розв’язку задачі інтелектуальної колоризації зображень. В основі розробленої програмної системи для інтелектуальної колоризації лежить робота двох конкуруючих згорткових нейронних мереж: мережі-генератора та мережі-дискримінатора. На етапі підготовки даних пропонується виконувати перетворення зображень з колірної моделі RGB у модель YUV або LAB, що дозволяє отримати кращий результат колоризації. Також у статті розглянуто можливість переходу від оброблення зображень до оброблення мультимодальних даних та запропоновано подавати вхідні мультимодальні дані про об’єкти на зображені у вигляді мультимодальної багаторівневої онтологічної моделі.

Detecting Building Changes between Airborne Laser Scanning and Photogrammetric Data

Detecting topographic changes in an urban environment and keeping city-level point clouds up-to-date are important tasks for urban planning and monitoring. In practice, remote sensing data are often available only in different modalities for two epochs. Change detection between airborne laser scanning data and photogrammetric data is challenging due to the multi-modality of the input data and dense matching errors. This paper proposes a method to detect building changes between multimodal acquisitions. The multimodal inputs are converted and fed into a light-weighted pseudo-Siamese convolutional neural network (PSI-CNN) for change detection. Different network configurations and fusion strategies are compared. Our experiments on a large urban data set demonstrate the effectiveness of the proposed method. Our change map achieves a recall rate of 86.17%, a precision rate of 68.16%, and an F1-score of 76.13%. The comparison between Siamese architecture and feed-forward architecture brings many interesting findings and suggestions to the design of networks for multimodal data processing.

Multimodal GAN for Energy Efficiency and Cloud Classification in Internet of Things

Efficient processing of large-scale multimodal sensor data is a key issue for applying the Internet of Things (IoT). Accurate cloud classification is critical for weather and climate monitoring, which are parts of IoT applications. In this paper, we propose a novel generative deep model named multimodal generative adversarial network (Multimodal GAN) to improve both the energy efficiency and the cloud classification accuracy in IoT. The proposed Multimodal GAN is composed of a discriminator and a generator, each of which is devised to a two-stream network. The branches of two-stream structure correspond to the cloud visual information and the cloud scalar information, respectively. Therefore, the Multimodal GAN is capable of generating the cloud visual information and cloud scalar information simultaneously. Afterward, the training set is extended by the generated multimodal cloud samples, and the deep multimodal cloud classification model is trained by the extended training set. As a result, the classification model possesses high generalization ability and is less prone to be over-fitting. Moreover, the feature representations extracted from the classification model reflect the salient information of raw multimodal cloud data, and therefore they can be stored and transmitted in IoT. The effectiveness of the proposed method in energy efficiency and cloud classification is validated on the multimodal cloud dataset.

Signal Processing of Multimodal Mobile Lifelogging Data Towards Detecting Stress in Real-World Driving

Stress is a negative emotion that is part of everyday life. However, frequent episodes or prolonged periods of stress can be detrimental to long-term health. Nevertheless, developing self-awareness is an important aspect of fostering effective ways to self-regulate these experiences. Mobile lifelogging systems provide an ideal platform to support self-regulation of stress by raising awareness of negative emotional states via continuous recording of psychophysiological and behavioral data. However, obtaining meaningful information from large volumes of raw data represents a significant challenge because these data must be accurately quantified and processed before stress can be detected. This work describes a set of algorithms designed to process multiple streams of lifelogging data for stress detection in the context of real world driving. Two data collection exercises have been performed where multimodal data, including raw cardiovascular activity and driving information, were collected from 21 people during daily commuter journeys. Our approach enabled us to 1) pre-process raw physiological data to calculate valid measures of heart rate variability, a significant marker of stress, 2) identify/correct artefacts in the raw physiological data, and 3) provide a comparison between several classifiers for detecting stress. Results were positive and ensemble classification models provided a maximum accuracy of 86.9 percent for binary detection of stress in the real-world.

Logical Operations in Algebraic System of Aggregates for Multimodal Data Representation and Processing

Проблематика. Сфера застосувань програмного забезпечення, яке оперує мультимодальними даними, стає все ширшою. Для розроблення ефективних алгоритмів оброблення мультимодальних даних необхідне їх подання як комплексних структур. Існуючі підходи більшістю орієнтовані на незалежне подання даних різної природи, що знижує ефективність їх застосування для комплексного подання мультимодальних даних. Мета дослідження. Розроблення математичного забезпечення для комплексного подання та оброблення мультимо­дальних даних у комп’ютерних системах. Методика реалізації. Алгебрична система агрегатів розроблена для комплексного подання мультимодальних даних. У ній визначено логічні операції, операції впорядкування та арифметичні операції над агрегатами. Логічні операції можуть використовуватися для формування комплексного подання потрібних даних у формі агрегатів для подальшого їх аналізу. Передумовою такого подання є те, що дані мають належати різним модальностям та бути визначеними в часі, тобто являти собою послідовності. Результати дослідження. Запропоновано та описано логічні операції над агрегатами. Ці операції дають можливість отримувати різноманітні композиції мультимодальних даних, що робить можливим комплексне подання даних для всебічного опису об’єктів та процесів у різноманітних галузях, в тому числі в медицині. Розроблено та представлено концепцію мультиобразу, яка дає змогу комплексно описувати об’єкт, суб’єкт або процес на основі даних, отриманих під час спостереження, яке виконується з урахуванням часу. Висновки. Алгебрична система агрегатів є засобом для комплексного подання даних. Вона дає змогу оброблювати мультимодальні дані, використовуючи набір операції та відношень. Основною відмінністю між логічними операціями в алгебричній системі агрегатів та логічними операціями над множинами є те, що агрегат являє собою комплексне подання набору даних, представлених у вигляді послідовностей, де порядок слідування елементів кожної модальності є важливим і таким, що впливає на результат операції.

Collection and Analysis of Multimodal Data for SUDEP Biomarker Discovery.

This paper discusses the acquisition and processing of multimodal physiological data from patients with epilepsy in Epilepsy Monitoring Units for the discovery of risk factors for Sudden Expected Death in Epilepsy (SUDEP) that can be combined through integrative analysis for biomarker discovery.

Data processing approaches and strategies for non-destructive fruits quality inspection and authentication: a review

Fruit quality inspection and authentication instruments are the essential requirement at the different stages of fruit processing industries from harvesting to market. In recent years, various intelligent analytical methods such as electronic nose, gas chromatography and mass spectroscopy, UV–Vis–NIR spectroscopy, machine vision, hyperspectral imaging and many more have been evolved to access the fruit quality at different stages such as maturity judgement of an on-tree fruit, shelf life measurement of harvested fruit, other quality parameters measurement of various fruit products at processing industries etc. Information extracted from various analytical methods needs to be processed using different data processing approaches and strategies, which plays the major role to bring the intelligence in the analytical instruments. Although, highly promising results have been reported to process data acquired from similar type of sensory panel (gas sensor array in electronic nose) and single sensing technique (impedance measurement) but still there are several challenges to process data acquired from multiple sensing techniques fusion (similar or complementary in nature) to predict better informative results. Recently, there is a growing interest in the direction of multiple sensing techniques fusion to extract better information from fruit samples in a reliable manner and also in less time. This paper presents an extensive review of classical and modern data processing approaches and strategies that have been used for single and multiple non-destructive sensing methods in the area of fruit quality inspection and authentication. Various approaches and strategies for preprocessing, data fusion, feature extraction, model design, multi-modal data processing, training, testing and validation for single and multiple sensing techniques have been briefly explained in the presented review. The presented review also discusses the need, scope, and challenges of data processing methods for multiple sensing techniques fusion. Different commercially available handheld and lab level analytical instruments also have been reviewed based on their intelligence, complexity and quality parameters prediction.

PhysioLab - a multivariate physiological computing toolbox for ECG, EMG and EDA signals: a case of study of cardiorespiratory fitness assessment in the elderly population

The exponential increase of wearable health-tracking technologies offers new possibilities but also poses new challenges in signal processing to enable fitness monitoring through multimodal physiological recordings. Although there are several software tools used for post-processing in physiological computing applications, limitations in the analysis, incorporating signals from multiple sources, integrating contextual information and providing information visualization tools prevent a widespread use of this technology. To address these issues, we introduce PhysioLab, a multimodal processing Matlab tool for the data analysis of Electromyography (EMG), Electrocardiography (ECG) and Electrodermal Activity (EDA). The software is intended to facilitate the processing and comprehension of multimodal physiological data with the aim of assessing fitness in several domains. A unique feature of PhysioLab is that is informed by normative data grouped by age and sex, allowing contextualization of data based on users’ demographics. Besides signal processing, PhysioLab includes a novel approach to multivariable data visualization with the aim of simplifying interpretation by non-experts users. The system computes a set of ECG features based on heart rate variability analysis, EMG parameters to quantify force and fatigue levels, and galvanic skin level/responses from EDA signals. Furthermore, PhysioLab provides compatibility with data from multiple low-cost wearable sensors. We conducted an experiment with 17 community-dwelling older adults (64.5 ± 6.4) to assess the feasibility of the tool in characterizing cardiorespiratory profiles during physical activity. Correlation analyses and regression models showed significant interactions between physiology and fitness evaluations. Our results suggest novel ways that physiological parameters could be effectively used to complement traditional fitness assessment.

PhD Forum: Multimodal IoT and EMR based Smart Health Application for Asthma Management in Children.

According to a study done in 2014 by National Health Interview Survey around 6.3 million children in United States suffer from asthma [1]. Asthma remains one of the leading reasons for pediatric admissions to children's hospitals, and has a prevalence rate of approximately 10% in children and it leads to missed days from school and other societal costs. This occurs despite improved medications to control asthma symptoms. Asthma management is challenging as it involves understanding asthma causes and avoiding asthma triggers that are both multi-factorial and individualistic in nature. It is almost impossible for doctors to constantly monitor each patient's health and environmental triggers. According to a recent article, the IoT device market in health-care will increase to a worth of $117 billion by the year 2020 [2]. The monitoring segment of IoT devices have predicted to increase $15 billion in 2017 [5]. The sales of smart watches, fitness and health trackers, are expected to account for more than 70% of all wearables sale worldwide in 2016 [6]. According to IBM, the volume of health-care data has reached to 150 exabytes in 2017 [7]. The data generated from these consumer graded devices is increasing day by day. This data collection has exacerbated the problem of understanding the data and making sense of it.

Mirror Mirror on the Wall... An Unobtrusive Intelligent Multisensory Mirror for Well-Being Status Self-Assessment and Visualization

A person's well-being status is reflected by their face through a combination of facial expressions and physical signs. The SEMEOTICONS project translates the semeiotic code of the human face into measurements and computational descriptors that are automatically extracted from images, videos, and three-dimensional scans of the face. SEMEOTICONS developed a multisensory platform in the form of a smart mirror to identify signs related to cardio-metabolic risk. The aim was to enable users to self-monitor their well-being status over time and guide them to improve their lifestyle. Significant scientific and technological challenges have been addressed to build the multisensory mirror, from touchless data acquisition, to real-time processing and integration of multimodal data.

Time-Of-Flight Camera, Optical Tracker and Computed Tomography in Pairwise Data Registration.

PurposeA growing number of medical applications, including minimal invasive surgery, depends on multi-modal or multi-sensors data processing. Fast and accurate 3D scene analysis, comprising data registration, seems to be crucial for the development of computer aided diagnosis and therapy. The advancement of surface tracking system based on optical trackers already plays an important role in surgical procedures planning. However, new modalities, like the time-of-flight (ToF) sensors, widely explored in non-medical fields are powerful and have the potential to become a part of computer aided surgery set-up. Connection of different acquisition systems promises to provide a valuable support for operating room procedures. Therefore, the detailed analysis of the accuracy of such multi-sensors positioning systems is needed.MethodsWe present the system combining pre-operative CT series with intra-operative ToF-sensor and optical tracker point clouds. The methodology contains: optical sensor set-up and the ToF-camera calibration procedures, data pre-processing algorithms, and registration technique. The data pre-processing yields a surface, in case of CT, and point clouds for ToF-sensor and marker-driven optical tracker representation of an object of interest. An applied registration technique is based on Iterative Closest Point algorithm.ResultsThe experiments validate the registration of each pair of modalities/sensors involving phantoms of four various human organs in terms of Hausdorff distance and mean absolute distance metrics. The best surface alignment was obtained for CT and optical tracker combination, whereas the worst for experiments involving ToF-camera.ConclusionThe obtained accuracies encourage to further develop the multi-sensors systems. The presented substantive discussion concerning the system limitations and possible improvements mainly related to the depth information produced by the ToF-sensor is useful for computer aided surgery developers.

Kernel Manifold Alignment for Domain Adaptation.

The wealth of sensory data coming from different modalities has opened numerous opportunities for data analysis. The data are of increasing volume, complexity and dimensionality, thus calling for new methodological innovations towards multimodal data processing. However, multimodal architectures must rely on models able to adapt to changes in the data distribution. Differences in the density functions can be due to changes in acquisition conditions (pose, illumination), sensors characteristics (number of channels, resolution) or different views (e.g. street level vs. aerial views of a same building). We call these different acquisition modes domains, and refer to the adaptation problem as domain adaptation. In this paper, instead of adapting the trained models themselves, we alternatively focus on finding mappings of the data sources into a common, semantically meaningful, representation domain. This field of manifold alignment extends traditional techniques in statistics such as canonical correlation analysis (CCA) to deal with nonlinear adaptation and possibly non-corresponding data pairs between the domains. We introduce a kernel method for manifold alignment (KEMA) that can match an arbitrary number of data sources without needing corresponding pairs, just few labeled examples in all domains. KEMA has interesting properties: 1) it generalizes other manifold alignment methods, 2) it can align manifolds of very different complexities, performing a discriminative alignment preserving each manifold inner structure, 3) it can define a domain-specific metric to cope with multimodal specificities, 4) it can align data spaces of different dimensionality, 5) it is robust to strong nonlinear feature deformations, and 6) it is closed-form invertible, which allows transfer across-domains and data synthesis. To authors’ knowledge this is the first method addressing all these important issues at once. We also present a reduced-rank version of KEMA for computational efficiency, and discuss the generalization performance of KEMA under Rademacher principles of stability. Aligning multimodal data with KEMA reports outstanding benefits when used as a data pre-conditioner step in the standard data analysis processing chain. KEMA exhibits very good performance over competing methods in synthetic controlled examples, visual object recognition and recognition of facial expressions tasks. KEMA is especially well-suited to deal with high-dimensional problems, such as images and videos, and under complicated distortions, twists and warpings of the data manifolds. A fully functional toolbox is available at https://github.com/dtuia/KEMA.git.

Deep learning human actions from video via sparse filtering and locally competitive algorithms

Physiological and psychophysical evidence suggest that early visual cortex compresses the visual input on the basis of spatial and orientation-tuned filters. Recent computational advances have suggested that these neural response characteristics may reflect a `sparse coding' architecture--in which a small number of neurons need to be active for any given image--yielding critical structure latent in natural scenes. Here we present a novel neural network architecture combining a sparse filter model and locally competitive algorithms (LCAs), and demonstrate the network's ability to classify human actions from video. Sparse filtering is an unsupervised feature learning algorithm designed to optimize the sparsity of the feature distribution directly without having the need to model the data distribution. LCAs are defined by a system of differential equations where the initial conditions define an optimization problem and the dynamics converge to a sparse decomposition of the input vector. We applied this architecture to train a classifier on categories of motion in human action videos. Inputs to the network were small 3D patches taken from frame differences in the videos. Dictionaries were derived for each action class and then activation levels for each dictionary were assessed during reconstruction of a novel test patch. Overall, classification accuracy was at ? 97 %. We discuss how this sparse filtering approach provides a natural framework for multi-sensory and multimodal data processing including RGB video, RGBD video, hyper-spectral video, and stereo audio/video streams.