Fine-grained Action Recognition Research Articles

Object-Level Trajectories Based Fine-Grained Action Recognition in Visual IoT Applications

The emerging computer vision and deep learning technologies are being applied to the intelligent analysis of sports training videos. In this paper, a deep learning based fine-grained action recognition (FGAR) method is proposed to analyze soccer training videos. The proposed method was applied to indoor training equipment for evaluating whether a player has stopped a soccer ball successfully or not. First, the problem of FGAR is modeled as human-object (player-ball) interactions. The object-level trajectories are proposed as a new descriptor to identify fine-grained sports videos. The proposed descriptor can take advantage of high-level semantic and human-object interaction motions. Second, a cascaded scheme of deep networks based on the object-level trajectories is proposed to realize FGAR. The cascaded network is constructed by concatenating a detector network with a classifier network (a long-short-term-memory (LSTM)-based network). The cascaded scheme takes the advantage of the high efficiency of the detector on object detection and the outstanding performance of the LSTM-based network on processing time series. The experimental results show that the proposed method can achieve an accuracy of 93.24%.

Activity Recognition Using Gazed Text and Viewpoint Information for User Support Systems

The development of information technology has added many conveniences to our lives. On the other hand, however, we have to deal with various kinds of information, which can be a difficult task for elderly people or those who are not familiar with information devices. A technology to recognize each person’s activity and providing appropriate support based on that activity could be useful for such people. In this paper, we propose a novel fine-grained activity recognition method for user support systems that focuses on identifying the text at which a user is gazing, based on the idea that the content of the text is related to the activity of the user. It is necessary to keep in mind that the meaning of the text depends on its location. To tackle this problem, we propose the simultaneous use of a wearable device and fixed camera. To obtain the global location of the text, we perform image matching using the local features of the images obtained by these two devices. Then, we generate a feature vector based on this information and the content of the text. To show the effectiveness of the proposed approach, we performed activity recognition experiments with six subjects in a laboratory environment.

Temporal and Fine-Grained Pedestrian Action Recognition on Driving Recorder Database.

The paper presents an emerging issue of fine-grained pedestrian action recognition that induces an advanced pre-crush safety to estimate a pedestrian intention in advance. The fine-grained pedestrian actions include visually slight differences (e.g., walking straight and crossing), which are difficult to distinguish from each other. It is believed that the fine-grained action recognition induces a pedestrian intention estimation for a helpful advanced driver-assistance systems (ADAS). The following difficulties have been studied to achieve a fine-grained and accurate pedestrian action recognition: (i) In order to analyze the fine-grained motion of a pedestrian appearance in the vehicle-mounted drive recorder, a method to describe subtle change of motion characteristics occurring in a short time is necessary; (ii) even when the background moves greatly due to the driving of the vehicle, it is necessary to detect changes in subtle motion of the pedestrian; (iii) the collection of large-scale fine-grained actions is very difficult, and therefore a relatively small database should be focused. We find out how to learn an effective recognition model with only a small-scale database. Here, we have thoroughly evaluated several types of configurations to explore an effective approach in fine-grained pedestrian action recognition without a large-scale database. Moreover, two different datasets have been collected in order to raise the issue. Finally, our proposal attained 91.01% on National Traffic Science and Environment Laboratory database (NTSEL) and 53.23% on the near-miss driving recorder database (NDRDB). The paper has improved +8.28% and +6.53% from baseline two-stream fusion convnets.

Fine-grained action recognition of boxing punches from depth imagery

Competitive sporting environments demand reliable statistics on an athlete’s performance to measure an athlete’s actions during competition, and to differentiate between the fine-grained actions performed. This is especially true for combat sports such as boxing where the variations observed between the main punching actions are subtle, making automatic classification of movements extremely difficult. This paper presents a robust framework for the automatic classification of a boxer’s punches. Overhead depth imagery is employed to alleviate challenges associated with occlusions, and robust body-part tracking is developed for the noisy time-of-flight sensors. Punch recognition is addressed through multi-class Support Vector Machine (SVM) and Random Forest classifiers using combinations of features. A coarse-to-fine hierarchical SVM classifier is presented in this paper based on prior knowledge of boxing punches. This framework has been applied to boxing image sequences taken at the Australian Institute of Sport with 14 elite boxers. Results demonstrate the effectiveness of the action recognition method, with the hierarchical SVM classifier yielding a 97.3% accuracy improving on the recent state-of-the-art action recognition systems.

Rank Pooling for Action Recognition.

We propose a function-based temporal pooling method that captures the latent structure of the video sequence data - e.g., how frame-level features evolve over time in a video. We show how the parameters of a function that has been fit to the video data can serve as a robust new video representation. As a specific example, we learn a pooling function via ranking machines. By learning to rank the frame-level features of a video in chronological order, we obtain a new representation that captures the video-wide temporal dynamics of a video, suitable for action recognition. Other than ranking functions, we explore different parametric models that could also explain the temporal changes in videos. The proposed functional pooling methods, and rank pooling in particular, is easy to interpret and implement, fast to compute and effective in recognizing a wide variety of actions. We evaluate our method on various benchmarks for generic action, fine-grained action and gesture recognition. Results show that rank pooling brings an absolute improvement of 7-10 average pooling baseline. At the same time, rank pooling is compatible with and complementary to several appearance and local motion based methods and features, such as improved trajectories and deep learning features.

Multimodal Wearable Sensing for Fine-Grained Activity Recognition in Healthcare

State-of-the-art in-home activity recognition schemes with wearable devices are mostly capable of detecting coarse-grained activities (sitting, standing, walking, or lying down), but can't distinguish complex activities (sitting on the floor versus the sofa or bed). Such schemes often aren't effective for emerging critical healthcare applications -- for example, in remote monitoring of patients with Alzheimer's disease, bulimia, or anorexia -- because they require a more comprehensive, contextual, and fine-grained recognition of complex daily user activities. Here, a novel approach for in-home, fine-grained activity recognition uses multimodal wearable sensors on multiple body positions, along with lightly deployed Bluetooth beacons in the environment. In particular, this solution exploits measuring user's ambient environment and location context with wearable sensing and Bluetooth beacons, along with user movement captured with accelerometer and gyroscope sensors. The proposed algorithm is a two-level supervised classifier with both levels running on a server. In the first level, multisensor data from wearables on each body position are collected and analyzed using the proposed modified conditional random field (CRF)-based supervised activity classifier. The classified activity state from each of the wearables data are then fused for deciding the user's final activity state. Preliminary experimental results are presented on the classification of 19 complex daily activities of a user at home.

Marginal filtering in large state spaces

We describe the marginal filter for activity recognition using symbolic models.The marginal filter allows fine-grained activity recognition using wearable sensors.We identify and discuss advantages over particle filters for symbolic models. Recognising everyday activities including information about the context requires to handle large state spaces. The usage of wearable sensors like six degree of freedom accelerometers increases complexity even more. Common approaches are unable to maintain an accurate belief state within such complex domains. We show how marginal filtering can overcome limitations of standard particle filtering and efficiently infer the context of actions. Symbolic models of human behaviour are used to recognise activities in two different settings with different state space sizes. Based on these scenarios we compare the marginal filter to the standard particle filter. An evaluation shows that the marginal filter performs comparably in small state spaces but outperforms the particle filter in large state spaces.

A Knowledge-Driven Approach to Activity Recognition in Smart Homes

This paper introduces a knowledge-driven approach to real-time, continuous activity recognition based on multisensor data streams in smart homes. The approach goes beyond the traditional data-centric methods for activity recognition in three ways. First, it makes extensive use of domain knowledge in the life cycle of activity recognition. Second, it uses ontologies for explicit context and activity modeling and representation. Third and finally, it exploits semantic reasoning and classification for activity inferencing, thus enabling both coarse-grained and fine-grained activity recognition. In this paper, we analyze the characteristics of smart homes and Activities of Daily Living (ADL) upon which we built both context and ADL ontologies. We present a generic system architecture for the proposed knowledge-driven approach and describe the underlying ontology-based recognition process. Special emphasis is placed on semantic subsumption reasoning algorithms for activity recognition. The proposed approach has been implemented in a function-rich software system, which was deployed in a smart home research laboratory. We evaluated the proposed approach and the developed system through extensive experiments involving a number of various ADL use scenarios. An average activity recognition rate of 94.44 percent was achieved and the average recognition runtime per recognition operation was measured as 2.5 seconds.