First-person Activity Recognition Research Articles

Holistic-Guided Disentangled Learning With Cross-Video Semantics Mining for Concurrent First-Person and Third-Person Activity Recognition.

The popularity of wearable devices has increased the demands for the research on first-person activity recognition. However, most of the current first-person activity datasets are built based on the assumption that only the human-object interaction (HOI) activities, performed by the camera-wearer, are captured in the field of view. Since humans live in complicated scenarios, in addition to the first-person activities, it is likely that third-person activities performed by other people also appear. Analyzing and recognizing these two types of activities simultaneously occurring in a scene is important for the camera-wearer to understand the surrounding environments. To facilitate the research on concurrent first- and third-person activity recognition (CFT-AR), we first created a new activity dataset, namely PolyU concurrent first- and third-person (CFT) Daily, which exhibits distinct properties and challenges, compared with previous activity datasets. Since temporal asynchronism and appearance gap usually exist between the first- and third-person activities, it is crucial to learn robust representations from all the activity-related spatio-temporal positions. Thus, we explore both holistic scene-level and local instance-level (person-level) features to provide comprehensive and discriminative patterns for recognizing both first- and third-person activities. On the one hand, the holistic scene-level features are extracted by a 3-D convolutional neural network, which is trained to mine shared and sample-unique semantics between video pairs, via two well-designed attention-based modules and a self-knowledge distillation (SKD) strategy. On the other hand, we further leverage the extracted holistic features to guide the learning of instance-level features in a disentangled fashion, which aims to discover both spatially conspicuous patterns and temporally varied, yet critical, cues. Experimental results on the PolyU CFT Daily dataset validate that our method achieves the state-of-the-art performance.

PDEs on graphs for semi-supervised learning applied to first-person activity recognition in body-worn video

This paper showcases the use of PDE-based graph methods for modern machine learning applications. We consider a case study of body-worn video classification because of the large volume of data and the lack of training data due to sensitivity of the information. Many modern artificial intelligence methods are turning to deep learning which typically requires a lot of training data to be effective. They can also suffer from issues of trust because the heavy use of training data can inadvertently provide information about details of the training images and could compromise privacy. Our alternate approach is a physics-based machine learning that uses classical ideas like optical flow for video analysis paired with linear mixture models such as non-negative matrix factorization along with PDE-based graph classification methods that parallel geometric equations from PDE such as motion by mean curvature. The upshot is a methodology that can work well on video with modest amounts of training data and that can also be used to compress the information about the video scene so that no personal information is contained in the compressed data, making it possible to provide a larger group of people access to these compressed data without compromising privacy. The compressed data retains information about the wearer of the camera while discarding information about people, objects, and places in the scene.

First-person activity recognition from micro-action representations using convolutional neural networks and object flow histograms

A novel first-person human activity recognition framework is proposed in this work. Our proposed methodology is inspired by the central role moving objects have in egocentric activity videos. Using a Deep Convolutional Neural Network we detect objects and develop discriminant object flow histograms in order to represent fine-grained micro-actions during short temporal windows. Our framework is based on the assumption that large scale activities are synthesized by fine-grained micro-actions. We gather all the micro-actions and perform Gaussian Mixture Model clusterization, so as to build a micro-action vocabulary that is later used in a Fisher encoding schema. Results show that our method can reach 60% recognition rate on the benchmark ADL dataset. The capabilities of the proposed framework are also showcased by profoundly evaluating for a great deal of hyper-parameters and comparing to other State-of-the-Art works.

First-Person Activity Recognition from Micro-Action Representations using Convolutional Neural Networks and Object Flow Histograms

First-Person Activity Recognition from Micro-Action Representations using Convolutional Neural Networks and Object Flow Histograms

A correlation based feature representation for first-person activity recognition

In this paper, a simple yet efficient activity recognition method for first-person video is introduced. The proposed method is appropriate for representation of high-dimensional features such as those extracted from convolutional neural networks (CNNs). The per-frame (per-segment) extracted features are considered as a set of time series, and inter and intra-time series relations are employed to represent the video descriptors. To find the inter-time relations, the series are grouped and the linear correlation between each pair of groups is calculated. The relations between them can represent the scene dynamics and local motions. The introduced grouping strategy helps to considerably reduce the computational cost. Furthermore, we split the series in temporal direction in order to preserve long term motions and better focus on each local time window. In order to extract the cyclic motion patterns, which can be considered as primary components of various activities, intra-time series correlations are exploited. The representation method results in highly discriminative features which can be linearly classified. The experiments confirm that our method outperforms the state-of-the-art methods on recognizing first-person activities on the two challenging first-person datasets.

PSO based combined kernel learning framework for recognition of first-person activity in a video

This paper presents human activity recognition problem from first-person view-point (ego-centric video). The task is to understand the activities of a person by an observer (wearable camera or robot) from real-time video data. An efficient human activity recognition system demands the choice of useful traits and the suitable kernels for those traits. In this work, we have proposed a combined kernel learning (CKL) framework using PSO as optimization algorithm for first-person activity recognition in a video. This framework does appropriate feature selection and combines those features from their respective kernels from the video data in a productive way. The proposed algorithm learns an optimal composite kernel from the combination of the basis kernel constructed from different motion-related features of the first-person video. To determine both basis kernel and their combination, this method can optimize a data-dependent kernel evaluation measure. The performance of the proposed CKL is evaluated by combining different types of motion features from the first-person video (JPL-interaction dataset). The result shows a comparatively better rate of accuracy than that of other state-of-the-art human activity recognition methods.

First-Person Activity Recognition: Feature, Temporal Structure, and Prediction

This paper discusses the problem of recognizing interaction-level human activities from a first-person viewpoint. The goal is to enable an observer (e.g., a robot or a wearable camera) to understand `what activity others are performing to it' from continuous video inputs. These include friendly interactions such as `a person hugging the observer' as well as hostile interactions like `punching the observer' or `throwing objects at the observer', whose videos involve a large amount of camera ego-motion caused by physical interactions. The paper investigates multi-channel kernels to integrate global and local motion information, and presents a new activity learning/recognition methodology that explicitly considers temporal structures displayed in first-person activity videos. Furthermore, we present a novel algorithm for early recognition (i.e., prediction) of activities from first-person videos, which allows us to infer ongoing activities at their early stage. In our experiments, we not only show classification results with segmented videos, but also confirm that our new approach is able to detect activities from continuous videos and perform early recognition reliably.

Multi-scale Conditional Random Fields for first-person activity recognition on elders and disabled patients

We propose a novel pervasive system to recognise human daily activities from a wearable device. The system is designed in a form of reading glasses, named ‘Smart Glasses’, integrating a 3-axis accelerometer and a first-person view camera. Our aim is to classify subject’s activities of daily living (ADLs) based on their vision and head motion data. This ego-activity recognition system not only allows caretakers to track on a specific person (such as disabled patient or elderly people), but also has the potential to remind/warn people with cognitive impairments of hazardous situations. We present the following contributions: a feature extraction method from accelerometer and video; a classification algorithm integrating both locomotive (body motions) and stationary activities (without or with small motions); a novel multi-scale dynamic graphical model for structured classification over time. In this paper, we collect, train and validate our system on two large datasets: 20 h of elder ADLs datasets and 40 h of patient ADLs datasets, containing 12 and 14 different activities separately. The results show that our method efficiently improves the system performance (F-Measure) over conventional classification approaches by an average of 20%–40% up to 84.45%, with an overall accuracy of 90.04% for elders. Furthermore, we also validate our method on 30 patients with different disabilities, achieving an overall accuracy up to 77.07%.