Group Activity Recognition Research Articles

Group Activity Recognition by Using Effective Multiple Modality Relation Representation With Temporal-Spatial Attention

Group activity recognition has received a great deal of interest because of its broader applications in sports analysis, autonomous vehicles, CCTV surveillance systems and video summarization systems. Most existing methods typically use appearance features and they seldom consider underlying interaction information. In this work, a technology of novel group activity recognition is proposed based on multi-modal relation representation with temporal-spatial attention. First, we introduce an object relation module, which processes all objects in a scene simultaneously through an interaction between their appearance feature and geometry, thus allowing the modeling of their relations. Second, to extract effective motion features, an optical flow network is fine-tuned by using the action loss as the supervised signal. Then, we propose two types of inference models, opt-GRU and relation-GRU, which are used to encode the object relationship and motion representation effectively, and form the discriminative frame-level feature representation. Finally, an attention-based temporal aggregation layer is proposed to integrate frame-level features with different weights and form effective video-level representations. We have performed extensive experiments on two popular datasets, and both have achieved state-of-the-art performance. The datasets are the Volleyball dataset and the Collective Activity dataset, respectively.

Coherence Constrained Graph LSTM for Group Activity Recognition.

This work aims to address the group activity recognition problem by exploring human motion characteristics. Traditional methods hold that the motions of all persons contribute equally to the group activity, which suppresses the contributions of some relevant motions to the whole activity while overstating some irrelevant motions. To address this problem, we present a Spatio-Temporal Context Coherence (STCC) constraint and a Global Context Coherence (GCC) constraint to capture the relevant motions and quantify their contributions to the group activity, respectively. Based on this, we propose a novel Coherence Constrained Graph LSTM (CCG-LSTM) with STCC and GCC to effectively recognize group activity, by modeling the relevant motions of individuals while suppressing the irrelevant motions. Specifically, to capture the relevant motions, we build the CCG-LSTM with a temporal confidence gate and a spatial confidence gate to control the memory state updating in terms of the temporally previous state and the spatially neighboring states, respectively. In addition, an attention mechanism is employed to quantify the contribution of a certain motion by measuring the consistency between itself and the whole activity at each time step. Finally, we conduct experiments on two widely-used datasets to illustrate the effectiveness of the proposed CCG-LSTM compared with the state-of-the-art methods.

A framework for group activity detection and recognition using smartphone sensors and beacons

Understanding occupant activities in a building is essential for building management systems to provide occupants with comfort and intelligent indoor environment. However, current occupant activity recognition mainly focuses on individual activity. Group activity recognition indoors has gained little attention, but remains of paramount importance, such as working together, taking classes, and discussions. In this paper, we propose a framework for group activity detection and recognition (i.e., GADAR framework) using smartphone sensors and Bluetooth beacons data. This framework consists of the following four layers: user layer, data package layer, processing layer, and output layer. As individuals within the group show similarity in motion, audio, and proximity, such similarity values are calculated and clustered into groups using hierarchical clustering. The framework then extracts the role, motion, speaking and location features from the clustered groups to distinguish different group activities. Decision tree classifier was selected to recognize the group activity that the group is engaged in. An experiment was conducted to identify the following three common group activities: taking class, seminar, and discussion. The result shows that the proposed GADAR framework could provide more than 89% accuracy in group detection and 89% accuracy in recognizing group activity.

Learning Semantics-Preserving Attention and Contextual Interaction for Group Activity Recognition.

In this paper, we investigate the problem of group activity recognition by learning semantics-perserving attention and contextual interaction among different people. Conventional methods usually aggregate the features extracted from individual persons by pooling operations, which lack physical meaning and cannot fully explore the contextual information for group activity recognition. To address this, we develop a Semantics-Preserving Teacher-Student (SPTS) networks architecture. Our SPTS networks first learn a Teacher Network in the semantic domain that classifies the word of group activity based on the words of individual actions. Then we design a Student Network in the appearance domain that recognizes the group activity according to the input video. We enforce the Student Network to mimic the Teacher Network in the learning procedure. In this way, we allocate semantics-preserving attention to different people, which is more effective to seek the key people and discard the misleading people, while no extra labelled data are required. Moreover, a group of people inherently lie in a graphbased structure, where the people and their relationship can be regarded as the nodes and edges of a graph respectively. Based on this, we build two graph convolutional modules on both the Teacher Network and the Student Network to reason the dependency among different people. Furthermore, we extend our approach on action segmentation task based on its intermediate features. Experimental results on four datasets for group activity analysis clearly show the superior performance of our method in comparisons with the state-of-the-arts.

Group activity recognition with an interaction force based on low‐level features

Group activity recognition of humans is an important task for video surveillance systems. In complex situations, however, ambiguities usually arise as a result of chaotic movements in the scene, especially when it is solved based upon motion trajectory analysis involving object segmentation and tracking. In this work, we present an interaction force model (IFM) to recognize group activity using low‐level feature based on dense optical flow. The IFM used to define interaction among people consists of three principal forces: group motion force (GMF), interaction motion force (IMF), and reciprocal motion force (RMF), to model the behaviors of the activity. Our algorithm is evaluated using the well‐known NUS‐HGA and BEHAVE datasets. Results of our approach reveal good efficiency with high accuracy, regardless of the surrounding conditions, and is competitive with other state‐of‐the‐art methods. © 2019 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.

GARSAaaS: group activity recognition and situation analysis as a service

Human activity recognition using embedded mobile and embedded sensors is becoming increasingly important. Scaling up from individuals to groups, that is, group activity recognition, has attracted significant attention recently. This paper proposes a model and specification language for group activities called GroupSense-L, and a novel architecture called GARSAaaS (GARSA-as-a-Service) to provide services for mobile Group Activity Recognition and Situation Analysis (or GARSA) applications. We implemented and evaluated GARSAaaS which is an extension of a framework called GroupSense (Abkenar et al., 2016 IEEE 30th International Conference on Advanced Information Networking and Applications (AINA), 2016) where sensor data, collected using smartphone sensors, smartwatch sensors and embedded sensors in things, are aggregated via a protocol for these different devices to share information, as required for GARSA. We illustrate our approach via a scenario for providing services for bush walking leaders and bush walkers in a bushwalking group activity. We demonstrate the feasibility of our model and expressiveness of our proposed model.

StagNet: An Attentive Semantic RNN for Group Activity and Individual Action Recognition

In real life, group activity recognition plays a significant and fundamental role in a variety of applications, e.g. sports video analysis, abnormal behavior detection, and intelligent surveillance. In a complex dynamic scene, a crucial yet challenging issue is how to better model the spatio-temporal contextual information and inter-person relationship. In this paper, we present a novel attentive semantic recurrent neural network (RNN), namely, stagNet, for understanding group activities and individual actions in videos, by combining the spatio-temporal attention mechanism and semantic graph modeling. Specifically, a structured semantic graph is explicitly modeled to express the spatial contextual content of the whole scene, which is further incorporated with the temporal factor through structural-RNN. By virtue of the “factor sharing” and “message passing” mechanisms, our stagNet is capable of extracting discriminative and informative spatio-temporal representations and capturing inter-person relationships. Moreover, we adopt a spatio-temporal attention model to focus on key persons/frames for improved recognition performance. Besides, a body-region attention and a global-part feature pooling strategy are devised for individual action recognition. In experiments, four widely-used public datasets are adopted for performance evaluation, and the extensive results demonstrate the superiority and effectiveness of our method.

Jointly social grouping and identification in visual dynamics with causality-induced hierarchical Bayesian model

We concentrate on modeling the person-person interactions for group activity recognition. In order to solve the complexity and ambiguity problems caused by a large number of human objects, we propose a causality-induced hierarchical Bayesian model to tackle the interaction activity video, referring to the “what” interaction activities happen, “where” interaction atomic occurs in spatial, and “when” group interaction happens in temporal. In particular, Granger Causality has been characterized with multiple features to encode the interacting relationships between each individual in the group. Furthermore, to detect and identify the concurrent interactive simultaneously, we investigate the Relative Entropy as a metric to measure the reasonable motion dependency between two arbitrary individuals. Filtered by the causality dependency, causality motion features have been cast as the multiplicative probabilistic ingredients in Bayesian factors to formulate the compact learned latent interaction patterns aggregately that enable the power of discrimination. Experiments demonstrate our model outperforms state-of-the-art models.

Group Activity Recognition Method based on Camera in The Building

Group activities are an activity carried out together at the same time and place. In the office/university environment, they play a role in energy consumption. For this reason, the smart building needs group activity recognition (GAR) systems for electrical device control. Several studies on the GAR have been carried out in the outdoor environment, such as exercising, walking, or running. However, indoor group activities that share of energy consumption are found still rare, such as in meetings, seminars, and classroom activities. This study proposes a GAR method in the buildings using multi-image based on Camera through the sitting position of people and formed from the face identified from the image and visualized into the mapping. The simulation was carried out based on references from the scenario of the meeting, seminar, and class activity in the S307-room, Department of Electrical Engineering and Information Technology, Universitas Gadjah Mada. The Neural Network algorithm was used to the GAR. The performance was evaluated in various measures such as precision, accuracy, and recall. The result of the GAR accuracy of the learning phase was 93.33%. The results of the GAR accuracy of testing phase were 63% and the error of GAR were 37%, respectively.

Deep Fully Connected Model for Collective Activity Recognition

Group activity recognition is a challenging task because there is an exponentially large number of semantic and geometrical relationships among individuals. This makes it difficult to model these interactions and merge them as a whole for group activity classification. In this paper, we propose a deep fully-connected model for group recognition, first we use the spatial-temporal model based on convolution neural network (CNN) and long short-term memory networks (LSTM) network to capture the dynamic features of each person. Then, we use the fully-connected conditional random field (FCCRF) to learn the interactions between people. Finally, with FCCRF potential functions we re-fine the activity recognition predicted by the spatial-temporal model. The experimental results on collective activity data-set and collective activity extended data-set show that our model improves recognition accuracy over baseline methods and gets competitive results in comparison to the state-of-the-art models.

GroupSense

Human activity recognition using embedded mobile and embedded sensors is becoming increasingly important. Scaling up from individuals to groups, that is, Group Activity Recognition (GAR), has attracted significant attention recently. This article proposes a model and modeling language for GAR called GroupSense-L and a novel distributed middleware called GroupSense for mobile GAR. We implemented and tested GroupSense using smartphone sensors, smartwatch sensors, and embedded sensors in things, where we have a protocol for these different devices to exchange information required for GAR. A range of continuous group activities (from simple to fairly complex) illustrates our approach and demonstrates the feasibility of our model and richness of the proposed specialization. We then conclude with lessons learned for GAR and future work.

Deep neural network model for group activity recognition using contextual relationship

In this paper, we present contextual relationship-based learning model using deep neural network for recognizing the activities performed by a group of people in a video sequence. The proposed model comprises of the context learning using a bottom-up approach, learning from individual human actions to group level activity as well as learning from the scene information. We build deep convolutional neural network model to capture human action-pose feature for a given input video sequence. To capture group level temporal flow changes, aggregated action-pose feature of persons within the context area are fed to deep recurrent neural network, which provides spatio-temporal group descriptor. Together with this, we build a scene level convolutional neural network, to extract scene level feature which improves the performance of group activity recognition. The probabilistic inference model, as an additional layer in deep neural network, added to ensemble the models and provide a unified deep learning framework. Experimental results show the efficiency of the proposed model on standard benchmark collective activity dataset in group activity recognition. We also present the evaluated results by varying different learning parameters, optimizers, especially recurrent neural network models long short-term memory and gated recurrent unit on the benchmark collective activity dataset.

Discriminative context learning with gated recurrent unit for group activity recognition

In this study, we address the problem of similar local motions that create confusion within different group activities. To reduce the influences of motions, we propose a discriminative group context feature (DGCF) that considers prominent sub-events. Moreover, we adopt a gated recurrent unit (GRU) model that can learn temporal changes in a sequence. In real-world scenarios, people perform activities with different temporal lengths. The GRU model handles an arbitrary length of data for training with nonlinear hidden units in the network. However, when we use a deep neural network model, data scarcity causes overfitting problems. Data augmentation methods for images are ineffective for trajectory data augmentation. Thus, we also propose a method for trajectory augmentation. We evaluate the effectiveness of the proposed method on three datasets. In our experiments on each dataset, we show that the proposed method outperforms the competing state-of-the-art methods for group activity recognition.

A Comprehensive Study of Group Activity Recognition Methods in Video

Objectives: To provide comprehensive review of different group activity recognition methods, categorize them and provide path to new researcher in this domain. Methods/Statistical Analysis: Different methods of group activity recognition categorized and analyzed according to hand-crafted and learned feature descriptors. Pros and cons of each method are presented. Methods are analyzed in detailed by finding its local level features to global level feature descriptors used along with performance on benchmark dataset. Findings: Different models of group activity recognition are characterized as per the capabilities of the defined model considering individual pose of person, atomic activity of person, person-person interaction, person-group interaction, group-group interaction, uses of temporal information, and recognition of group activity frame wise or video wise. This comprehensive review provides brief information about group activity recognition methods and can be used as brief literature review to the researcher seeking the facts and findings in the field of computer vision in group activity recognition. Applications/Improvements: This reviews help in different applications of human activity analysis, mainly in group activity recognition and the models described here can be used in different applications such running or walking on pathways, waiting at public places, queuing in line in group and many more group activity applications for further enhancement.

Multi-Instance Classification by Max-Margin Training of Cardinality-Based Markov Networks.

We propose a probabilistic graphical framework for multi-instance learning (MIL) based on Markov networks. This framework can deal with different levels of labeling ambiguity (i.e., the portion of positive instances in a bag) in weakly supervised data by parameterizing cardinality potential functions. Consequently, it can be used to encode different cardinality-based multi-instance assumptions, ranging from the standard MIL assumption to more general assumptions. In addition, this framework can be efficiently used for both binary and multiclass classification. To this end, an efficient inference algorithm and a discriminative latent max-margin learning algorithm are introduced to train and test the proposed multi-instance Markov network models. We evaluate the performance of the proposed framework on binary and multi-class MIL benchmark datasets as well as two challenging computer vision tasks: cyclist helmet recognition and human group activity recognition. Experimental results verify that encoding the degree of ambiguity in data can improve classification performance.

Group Activity Recognition with Group Interaction Zone Based on Relative Distance Between Human Objects

In this paper, we address the problem of recognizing group activities of human objects based on their motion trajectory analysis. In order to resolve the complexity and ambiguity problems caused by a large number of human objects, we propose a Group Interaction Zone (GIZ) to detect meaningful groups in a scene to effectively handle noisy information. Two novel features, Group Interaction Energy (GIE) feature and Attraction and Repulsion Features, are proposed to better describe group activities within a GIZ. We demonstrate the performance of our method in two ways by (i) comparing the performance of the proposed method with the previous methods and (ii) analyzing the influence of the proposed features and GIZ-based meaningful group detection on group activity recognition using public datasets.

Identifying physical activity type in manual wheelchair users with spinal cord injury by means of accelerometers.

This was a cross-sectional study. The main objective of this study was to develop and test classification algorithms based on machine learning using accelerometers to identify the activity type performed by manual wheelchair users with spinal cord injury (SCI). The study was conducted in the Physical Therapy department and the Physical Education and Sports department of the University of Valencia. A total of 20 volunteers were asked to perform 10 physical activities, lying down, body transfers, moving items, mopping, working on a computer, watching TV, arm-ergometer exercises, passive propulsion, slow propulsion and fast propulsion, while fitted with four accelerometers placed on both wrists, chest and waist. The activities were grouped into five categories: sedentary, locomotion, housework, body transfers and moderate physical activity. Different machine learning algorithms were used to develop individual and group activity classifiers from the acceleration data for different combinations of number and position of the accelerometers. We found that although the accuracy of the classifiers for individual activities was moderate (55-72%), with higher values for a greater number of accelerometers, grouped activities were correctly classified in a high percentage of cases (83.2-93.6%). With only two accelerometers and the quadratic discriminant analysis algorithm we achieved a reasonably accurate group activity recognition system (>90%). Such a system with the minimum of intervention would be a valuable tool for studying physical activity in individuals with SCI.

강의실 환경에서의 집단 개념동작 인식 기법

As smart phones with built-in sensors are developed, research on recognition using wearable devices is increasing. Existing papers are mostly limited on research to personal activity recognition. In this paper, we propose a method to recognize conceptual group activity. Before doing recognition, we generate new data based on the analysis of the conceptual group activity in a classroom. The study focuses on three activities in the classroom environment: Taking Lesson, Doing Presentation and Discussing. With the proposed algorithm, the recognition rate is over 96%. Using this method in real time will make it easy to automatically analyze the activity and the purpose of the classrooms. Moreover, it can increase the utilization of the classroom through the data analysis. Further research will focus on group activity recognition in other environments and the design of an group activity recognition system.

A New Network-Based Algorithm for Human Activity Recognition in Videos

In this paper, a new network-transmission-based (NTB) algorithm is proposed for human activity recognition in videos. The proposed NTB algorithm models the entire scene as an error-free network. In this network, each node corresponds to a patch of the scene and each edge represents the activity correlation between the corresponding patches. Based on this network, we further model people in the scene as packages, while human activities can be modeled as the process of package transmission in the network. By analyzing these specific package transmission processes, various activities can be effectively detected. The implementation of our NTB algorithm into abnormal activity detection and group activity recognition are described in detail in this paper. Experimental results demonstrate the effectiveness of our proposed algorithm.

Qualitative Spatio-temporal Reasoning Based Group Activity Recognition

Human group activity recognition is still a challenging task in computer vision. However, most of the works focus on the feature extraction and the analysis of the motion trajectories for recognizing the multi-person or group activities in surveillance videos. A qualitative spatio-temporal relation based on Hidden Markov Model (HMM) method is proposed to classify human group activities. We first propose Unified QTCB relations to represent the relations of the group. And then Unified QTCB relation based on HMM is proposed for group activity classification. Experiments are successfully conducted on the human group activity video database, and the performance of our approach is evaluated and compared with some other methods. The results show that our approach is more suitable for recognizing group activities.