Complex Activity Recognition Research Articles

Multi Resident Complex Activity Recognition in Smart Home: A Literature Review

Multi Resident Complex Activity Recognition in Smart Home: A Literature Review

Utilization of Mobile Phone Sensors for Complex Human Activity Recognition

Activity recognition is a significant part of pervasive computing as it can be employed in a wide range of fields which include eldercare and healthcare. While previous efforts have proven to be successful in identifying simple human activities, the means for identifying complex human activities remains an on-going effort. It has been established that more often than not, in an actual circumstance, human activities are conducted in an intricate mode. The objectives of this study are (a) to examine the utilization of solely the sensors of a mobile phone to distinguish complex human activities and (b) to enhance the complex activities recognition capacity of mobile phones through the application of multiple or other forms of sensors. This endeavour reassesses earlier studies on mobile phone utilization for complex activity recognition with the emphasis on schemes directed at smart home applications. An overall configuration for a human activity recognition (HAR) scheme, as well as an analysis of the latest investigations related to the use of mobile phones for complex activity recognition is also included in this paper. We conclude with a discussion on the results obtained and the forwarding of our proposals.

Towards complex activity recognition using a Bayesian network-based probabilistic generative framework

A Bayesian network-based probabilistic generative framework is presented to address diversity and uncertainty in complex activity recognition.The framework introduces the Chinese restaurant process to explicitly characterize the unique configurations of a complex activity.An enhanced model is presented to characterize more temporal relational variabilities than the previous models over our framework.Our models significantly outperform the state-of-the-arts on three benchmark datasets with different challenges.Our approach is robust against the incomplete or incorrect observations of primitive events. Complex activity recognition is challenging since a complex activity can be performed in different ways, with each having its own configuration of primitive events and their temporal dependencies. To address such temporal relational variabilities in complex activity recognition, we propose a Bayesian network-based probabilistic generative framework that employs Allens interval relation network to represent local temporal dependencies in a generative way. By employing the Chinese restaurant process and introducing relation generation constraints, our framework can characterize these unique internal configurations of a particular complex activity as a joint distribution. Three concrete models are implemented based on our framework. Specifically, in this paper we improve two of our previous models and provide an enhanced model to handle temporal relational variabilities in complex activities more efficiently. Empirical evaluations on three benchmark datasets demonstrate the competitiveness of our framework. In particular, it is shown that our models are rather robust against errors caused by the low-level predictions from raw signals.

Semantic segmentation of real-time sensor data stream for complex activity recognition

Data segmentation plays a critical role in performing human activity recognition in the ambient assistant living systems. It is particularly important for complex activity recognition when the events occur in short bursts with attributes of multiple sub-tasks. Although substantial efforts have been made in segmenting the real-time sensor data stream such as static/dynamic window sizing approaches, little has been explored to exploit object semantic for discerning sensor data into multiple threads of activity of daily living. This paper proposes a semantic-based approach for segmenting sensor data series using ontologies to perform terminology box and assertion box reasoning, along with logical rules to infer whether the incoming sensor event is related to a given sequences of the activity. The proposed approach is illustrated using a use-case scenario which conducts semantic segmentation of a real-time sensor data stream to recognise an elderly persons complex activities.

Complex activity recognition with multi-modal multi-positional body sensing

Complex activity recognition with multi-modal multi-positional body sensing

Tracking a Subset of Skeleton Joints: An Effective Approach towards Complex Human Activity Recognition

We present a robust algorithm for complex human activity recognition for natural human-robot interaction. The algorithm is based on tracking the position of selected joints in human skeleton. For any given activity, only a few skeleton joints are involved in performing the activity, so a subset of joints contributing the most towards the activity is selected. Our approach of tracking a subset of skeleton joints (instead of tracking the whole skeleton) is computationally efficient and provides better recognition accuracy. We have developed both manual and automatic approaches for the selection of these joints. The position of the selected joints is tracked for the duration of the activity and is used to construct feature vectors for each activity. Once the feature vectors have been constructed, we use a Support Vector Machines (SVM) multiclass classifier for training and testing the algorithm. The algorithm has been tested on a purposely built dataset of depth videos recorded using Kinect camera. The dataset consists of 250 videos of 10 different activities being performed by different users. Experimental results show classification accuracy of 83% when tracking all skeleton joints, 95% when using manual selection of subset joints, and 89% when using automatic selection of subset joints.

Complex human activities recognition using interval temporal syntactic model

A novel method based on interval temporal syntactic model was proposed to recognize human activities in video flow. The method is composed of two parts: feature extract and activities recognition. Trajectory shape descriptor, speeded up robust features (SURF) and histograms of optical flow (HOF) were proposed to represent human activities, which provide more exhaustive information to describe human activities on shape, structure and motion. In the process of recognition, a probabilistic latent semantic analysis model (PLSA) was used to recognize sample activities at the first step. Then, an interval temporal syntactic model, which combines the syntactic model with the interval algebra to model the temporal dependencies of activities explicitly, was introduced to recognize the complex activities with a time relationship. Experiments results show the effectiveness of the proposed method in comparison with other state-of-the-art methods on the public databases for the recognition of complex activities.

Estimation of Walking rate in Complex activity recognition

Physical activity recognition using embedded sensors has enabled by many context-aware applications in different areas. In sequential acceleration data there is a natural dependence between observations of movement or behavior, a fact that has been largely ignored in most analyses. In this paper, investigate the role that smart devices, including smartphones, can play in identifying activities of daily living. Monitoring and precisely quantifying users’ physical activity with inertial measurement unit-based devices, for instance, has also proven to be important in health management of patients affected by chronic diseases, e.g. We show that their combination only improves the overall recognition performance when their individual performances are not very high, so that there is room for performance improvement. We show that the system can be used accurately to monitor both feet movement and use this result in many applications such as any playing. Time and frequency domain features of the signal were used to discriminate between activities, it demonstrates accuracy of 93% when employing a random forest analytical approach.

Recognising complex activities with histograms of relative tracklets

We propose a method for activity recognition from video and accelerometer data.Visual accelerometer localization and tracking establishes cross-modal relations.RETLETS encode relative motion between tracked objects and local visual features.Recognition using various feature combinations is evaluated on the 50 Salads dataset.The method using RETLETS outperforms the state-of-the-art on this dataset. Display Omitted One approach to the recognition of complex human activities is to use feature descriptors that encode visual interactions by describing properties of local visual features with respect to trajectories of tracked objects. We explore an example of such an approach in which dense tracklets are described relative to multiple reference trajectories, providing a rich representation of complex interactions between objects of which only a subset can be tracked. Specifically, we report experiments in which reference trajectories are provided by tracking inertial sensors in a food preparation scenario. Additionally, we provide baseline results for HOG, HOF and MBH, and combine these features with others for multi-modal recognition. The proposed histograms of relative tracklets (RETLETS) showed better activity recognition performance than dense tracklets, HOG, HOF, MBH, or their combination. Our comparative evaluation of features from accelerometers and video highlighted a performance gap between visual and accelerometer-based motion features and showed a substantial performance gain when combining features from these sensor modalities. A considerable further performance gain was observed in combination with RETLETS and reference tracklet features.

Hierarchical Complex Activity Representation and Recognition Using Topic Model and Classifier Level Fusion

Human activity recognition is an important area of ubiquitous computing. Most current researches in activity recognition mainly focus on simple activities, e.g., sitting, running, walking, and standing. Compared with simple activities, complex activities are more complicated with high-level semantics, e.g., working, commuting, and having a meal. This paper presents a hierarchical model to recognize complex activities as mixtures of simple activities and multiple actions. We generate the components of complex activities using a clustering algorithm, represent and recognize complex activities by applying a topic model on these components. It is a data-driven method that can retain effective information for representing and recognizing complex activities. In addition, acceleration and physiological signals are fused in classifier level to ensure the overall performance of complex activity recognition. The results of experiments show that our method has ability to represent and recognize complex activities effectively.

A hybrid rule and machine learning based generic alerting platform for smart environments.

Existing smart environment based alert solutions have adopted a relatively complex and tailored approach to supporting individuals. These solutions have involved sensor based monitoring, activity recognition and assistance provisioning. Traditionally they have suffered from a number of issues, rooted in scalability and performance, associated with complex activity recognition processes. This paper introduces a generic approach to realizing an alerting platform within a smart environment. The core concept of this approach is presented and placed within the context of related work. A description of the approach is provided, followed by an evaluation. This evaluation shows the approach offers reasonable accuracy, future work will increase accuracy.

Mining intricate temporal rules for recognizing complex activities of daily living under uncertainty

Daily living activity recognition can be exploited to benefit mobile and ubiquitous computing applications. Techniques so far are mature to recognize simple actions. Due to the characteristics of diversity and uncertainty in daily living applications, most existing complex activity recognition approaches have notable limitations. First, graphical model-based approaches still lack sufficient expressive power to model rich temporal relations among activities. Second, it would be rather difficult for graphical model-based approaches to build a unified model for achieving multiple types of tasks. Third, current semantic-based approaches often fail to capture uncertainties. Fourth, formulae in these semantic-based approaches are often manually encoded. Meanwhile, it is impractical to handcraft each formula accurately in daily living scenarios where temporal relations among activities are intricate. To address these issues, we present a probabilistic semantic-based framework that combines Markov logic network with 15 temporal and hierarchical relations to explicitly perform diverse inference tasks of daily living in a unified manner. Advanced pattern mining techniques are introduced to automatically learn the propositional logic rules of intricate relations as well as their weights. Experimental results show that by logical reasoning with the mined temporal dependencies under uncertainty, the proposed model leads to an improved performance, particularly when recognizing complex activities involving the incomplete or incorrect observations of atomic actions. HighlightsA probabilistic semantic-based framework is presented to explicitly perform diverse inference tasks.The framework combines Markov logic networks with 15 relations between activities.Advanced pattern mining techniques are introduced to learn intricate temporal rules.Performances can be improved by logical reasoning with the mined rules under uncertainty.Our approach is robust to the incomplete or incorrect observations of intervals.

Complex Activity Recognition Via Attribute Dynamics

The problem of modeling the dynamic structure of human activities is considered. Video is mapped to a semantic feature space, which encodes activity attribute probabilities over time. The binary dynamic system (BDS) model is proposed to jointly learn the distribution and dynamics of activities in this space. This is a non-linear dynamic system that combines binary observation variables and a hidden Gauss---Markov state process, extending both binary principal component analysis and the classical linear dynamic systems. A BDS learning algorithm, inspired by the popular dynamic texture, and a dissimilarity measure between BDSs, which generalizes the Binet---Cauchy kernel, are introduced. To enable the recognition of highly non-stationary activities, the BDS is embedded in a bag of words. An algorithm is introduced for learning a BDS codebook, enabling the use of the BDS as a visual word for attribute dynamics (WAD). Short-term video segments are then quantized with a WAD codebook, allowing the representation of video as a bag-of-words for attribute dynamics. Video sequences are finally encoded as vectors of locally aggregated descriptors, which summarize the first moments of video snippets on the BDS manifold. Experiments show that this representation achieves state-of-the-art performance on the tasks of complex activity recognition and event identification.

Recognizing Complex Activities by a Probabilistic Interval-Based Model

A key challenge in complex activity recognition is the fact that a complex activity can often be performed in several different ways, with each consisting of its own configuration of atomic actions and their temporal dependencies. This leads us to define an atomic activity-based probabilistic framework that employs Allen's interval relations to represent local temporal dependencies. The framework introduces a latent variable from the Chinese Restaurant Process to explicitly characterize these unique internal configurations of a particular complex activity as a variable number of tables.It can be analytically shown that the resulting interval network satisfies the transitivity property, and as a result, all local temporal dependencies can be retained and are globally consistent.Empirical evaluations on benchmark datasets suggest our approach significantly outperforms the state-of-the-art methods.

Complex activity recognition using time series pattern dictionary learned from ubiquitous sensors

Sensor-based human activity recognition has become an important research field within pervasive and ubiquitous computing. Techniques for recognizing atomic activities such as gestures or actions are mature for now, but complex activity recognition still remains a challenging issue. In this paper, we address the problem of complex activity recognition using time series extracted from multiple sensors. We first build a dictionary of time series patterns, called shapelets, to represent atomic activities, then present three shapelet-based models to recognize sequential, concurrent, and generic complex activities. We use the datasets collected from three different labs to evaluate our shapelet-based approach and the results show that our approach can handle complex activity recognition effectively. Our experimental results also show that the shapelet-based approach outperforms other competing approaches in terms of recognition accuracy and system usage.

Evaluation of Wellness Detection Techniques using Complex Activities Association for Smart Home Ambient

Wireless Sensor Network based smart homes have the potential to meet the growing challenges of independent living of elderly people in smart homes. However, wellness detection of elderly people in smart homes is still a challenging research domain. Many researchers have proposed several techniques; however, majority of these techniques does not provide a comprehensive solution because complex activities cannot be determined easily and comprehensive wellness is difficult to diagnose. In this study’s critical review, it has been observed that strong association lies among the vital wellness determination parameters. In this paper, an association rules based model is proposed for the simple and complex (overlapped) activities recognition and comprehensive wellness detection mechanism after analyzing existing techniques. It considers vital wellness detection parameters (temporal association of sub activity location and sub activity, time gaps between two adjacent activities, temporal association of inter and intra activities). Activity recognition and wellness detection will be performed on the basis of extracted temporal association rules and expert knowledgebase. Learning component is an important module of our proposed model to accommodate the changing trends in the frequent pattern behavior of an elderly person and recommend a caregiver/expert to adjust the expert knowledgebase according to the found abnormalities.

Sensor-based human activity recognition system with a multilayered model using time series shapelets

We exploit time series shapelets for complex human activity recognition.We present a multilayered activity model to represent four types of activities.We implement a prototype system based on smartphone for human activity recognition.Daily living and basketball play activity recognition are conducted for evaluation. Human activity recognition can be exploited to benefit ubiquitous applications using sensors. Current research on sensor-based activity recognition is mainly using data-driven or knowledge-driven approaches. In terms of complex activity recognition, most data-driven approaches suffer from portability, extensibility and interpretability problems, whilst knowledge-driven approaches are often weak in handling intricate temporal data. To address these issues, we exploit time series shapelets for complex human activity recognition. In this paper, we first describe the association between activity and time series transformed from sensor data. Then, we present a recursively defined multilayered activity model to represent four types of activities and employ a shapelet-based framework to recognize various activities represented in the model. A prototype system was implemented to evaluate our approach on two public datasets. We also conducted two real-world case studies for system evaluation: daily living activity recognition and basketball play activity recognition. The experimental results show that our approach is capable of handling complex activity effectively. The results are interpretable and accurate, and our approach is fast and energy-efficient in real-time.

Wearable Sensor-Based Behavioral Anomaly Detection in Smart Assisted Living Systems

Detecting behavioral anomalies in human daily life is important to developing smart assisted-living systems for elderly care. Based on data collected from wearable motion sensors and the associated locational context, this paper presents a coherent anomaly detection framework to effectively detect different behavioral anomalies in human daily life. Four types of anomalies, including spatial anomaly, timing anomaly, duration anomaly, and sequence anomaly, are detected using a probabilistic theoretical framework. This framework is based on complex activity recognition using dynamic Bayesian network modeling. The maximum-likelihood estimation algorithm and Laplace smoothing are used in learning the parameters in the anomaly detection model. We conducted experimental evaluation in a mock apartment environment, and the results verified the effectiveness of the proposed framework. We expect that this behavioral anomaly detection system can be integrated into future smart homes for elderly care.

An approach for complex activity recognition by key frames

A new method for complex activity recognition in videos by key frames was presented. The progressive bisection strategy (PBS) was employed to divide the complex activity into a series of simple activities and the key frames representing the simple activities were extracted by the self-splitting competitive learning (SSCL) algorithm. A new similarity criterion of complex activities was defined. Besides the regular visual factor, the order factor and the interference factor measuring the timing matching relationship of the simple activities and the discontinuous matching relationship of the simple activities respectively were considered. On these bases, the complex human activity recognition could be achieved by calculating their similarities. The recognition error was reduced compared with other methods when ignoring the recognition of simple activities. The proposed method was tested and evaluated on the self-built broadcast gymnastic database and the dancing database. The experimental results prove the superior efficiency.

A hierarchical feature graph matching method for recognition of complex human activities

Abstract Complex human activities in natural videos are often composed of several atomic-level actions organized hierarchically. We should not only consider the appearance variability of these action units, but also model the spatiotemporal relationships between them when recognizing such high-level complex activities. In this paper, we focus on the problem of recognition of complex human activities in an example-based video retrieval framework and propose a new method based on hierarchical feature-graph matching. A video depicting an activity is represented as a high-level feature graph (HLFG), and each node of the HLFG is a mid-level feature graph (MLFG) constructed on a local collection of spatiotemporal interest points. MLFG, the first level of our two-level graph structure, describes the local feature contents and spatiotemporal arrangements of interest points. HLFG, the second level, describes the appearance variability and spatiotemporal arrangements of atomic-level actions in a way. Final recognition is accomplished by matching the HLFGs of the query and test videos, and matching two HLFGs involves matching the MLFGs between them. We use an efficient spectral method to solve these two graph-matching problems. Our method does not require any preprocessing and gives reasonable results with even a small number of query examples. We evaluate our approach with one publicly available complex human activity dataset and achieve results comparable to other systems that have studied this problem.