Motion History Image Research Articles

An efficient vision based elderly care monitoring framework using fall detection

Fall is one of the major reason of elderly people living alone for getting injured. Numerous elderly individuals are living alone in their homes. In case if the elderly tumble down, it might be troublesome for them to ask for help. Due to this problem there is a dire requirement of an efficient home monitoring framework. An effective fall detection system can help provide rapid help to the patient and improve the chances of survival of the patient. The principle goal of this paper is to design a vision-based fall detection framework for the elderly people. Fall identification framework at home is progressively vital and dependable observation framework is a need to mitigate the outcomes of fall. In this paper, we look forward to detecting a fall and normal walking of an individual using background subtraction, Motion History Image, feature extractor like Zernike Moments and then using a learning model. The proposed framework is able to detect fall and normal walking with an accuracy of 92.86% and 90.82%.

Comparison of Motion History Image and Approximated Ellipse Method in Human Fall Detection System

This paper compares two different method in human fall detection system namely motion history image and approximated ellipse. Research has been done in small studio with 4 CCTV camera as video data recorder, whereas video data are processed using MATLAB software. The experiment was carried out using three object’s fall direction and two type of falling movement. The fall direction is consist of front, side, and back fall. Whereas the falling movement is consist of direct and indirect fall movement. Meanwhile, the object’s initial position is standing and size of captured object is constant. The result is motion history image has accuracy 74.26% for direct falling movement, and 75.69% for indirect falling movement. Whereas approximated ellipse has accuracy 56.85% for direct falling movement, and 61.81% for indirect falling movement. Therefore, motion history image is better than approximated ellipse in human fall detection system.

3D Features for human action recognition with semi‐supervised learning

Human action recognition (HAR) is a very challenging task because of intra-class variations and complex backgrounds. Here, a motion history image (MHI)-based interest point refinement is proposed to remove the noisy interest points. Histogram of oriented gradient (HOG) and histogram of optical flow (HOF) techniques are extended from spatial to spatio-temporal domain to preserve the temporal information. These local features are used to build the trees for the random forest technique. During tree building, a semi-supervised learning is proposed for better splitting of data points at each node. For recognition of an action, mutual information is estimated for all the extracted interest points to each of the trained class by passing them through the random forest. The proposed method is evaluated on KTH, Weizmann, and UCF Sports standard datasets. The experimental results indicate that the proposed technique provides better performance compared to earlier reported techniques.

3D human action analysis and recognition through GLAC descriptor on 2D motion and static posture images

In this paper, we present an approach for identification of actions within depth action videos. First, we process the video to get motion history images (MHIs) and static history images (SHIs) corresponding to an action video based on the use of 3D Motion Trail Model (3DMTM). We then characterize the action video by extracting the Gradient Local Auto-Correlations (GLAC) features from the SHIs and the MHIs. The two sets of features i.e., GLAC features from MHIs and GLAC features from SHIs are concatenated to obtain a representation vector for action. Finally, we perform the classification on all the action samples by using the l2-regularized Collaborative Representation Classifier (l2-CRC) to recognize different human actions in an effective way. We perform evaluation of the proposed method on three action datasets, MSR-Action3D, DHA and UTD-MHAD. Through experimental results, we observe that the proposed method performs superior to other approaches.

Replay and key-events detection for sports video summarization using confined elliptical local ternary patterns and extreme learning machine

Sports broadcasters generate enormous amount of video content viewed all over the world. To capture the user interests in the rebroadcasted content, the sports videos are summarized that need the manual inspection and analysis. However, the huge repository and long duration of videos make manual analysis and summarization a laborious and time-consuming job. To overcome this problem, efforts have been made for automatic video summarization. In this paper, a novel framework to summarize sports videos is presented. It has been observed that the replays within a sports video represent key-events and these events can be used for video summarization. It has been noted that replays are usually sandwiched between start and stop of gradual-transitions. A thresholding-based approach is used to identify gradual transition effect (i.e. fade-in, fade-out) in sports video. The Gaussian mixture model (GMM) is then applied to key-event candidates to extract silhouettes and generate motion history image (MHI) for each key-event. The MHIs are processed using Confined Elliptical Local Ternary Patterns (CE-LTPs) for feature extraction. Extreme learning machine (ELM) classifier is used to learn the underlying model for events. A trained ELM-based classifier is then used for key-event detection. The output of classifier is then used for key-event labeling, replay detection, and complete game summarization. Performance of the proposed framework is evaluated on a dataset consisting of 20 videos of four different sports. Experimental results indicate the effectiveness of the proposed framework in terms of replays and key-events detection from selected dataset.

Combination of Motion History Image and Approximated Ellipse Method for Human Fall Detection System

Combination of Motion History Image and Approximated Ellipse Method for Human Fall Detection System

Learning spatiotemporal representations for human fall detection in surveillance video

In this paper, a computer vision based framework is proposed that detects falls from surveillance videos. Firstly, we employ background subtraction and rank pooling to model spatial and temporal representations in videos, respectively. We then introduce a novel three-stream Convolutional Neural Networks as an event classifier. Silhouettes and their motion history images serve as input to the first two streams, while dynamic images whose temporal duration is equal to motion history images, are used as input to the third stream. Finally, we apply voting on the results of event classification to perform multi-camera fall detection. The main novelty of our method against the conventional ones is that high-quality spatiotemporal representations in different levels are learned to take full advantage of the appearance and motion information. Extensive experiments have been conducted on two widely used fall datasets. The results have shown to demonstrate the effectiveness of the proposed method.

Vision-Based Flying Targets Detection via Spatiotemporal Context Fusion

Deriving from the imperative necessities for developing Sense and Avoid (SAA) capability of Unmanned Aerial Vehicle (UAV), a newly designed flying targets detection algorithm is presented in this paper for enhancing the UAV environment perception ability. Since spatiotemporal context is crucial for insuring the effectiveness of flying targets detection, the algorithm is constructed on the basis of spatiotemporal context fusion. The algorithm proposed in this paper contains three parts, namely the spatial context extraction, temporal context extraction and spatiotemporal context fusion. 1) In order to extract spatial context, dense sampling method is firstly applied to obtain dense image grids, then spatial context is generated via pre-learned conditional random field (CRF) model using a layered structure: dense image patches, bottom feature descriptors, sparse codes, and predicted CRF labels. 2) In order to extract temporal context, the forward and back motion history image (FBMHI) is firstly computed for detecting motion cues, and the adaptive foreground and background isolation is further adopted for acquiring the temporal probability map. 3) The presence probability map of flying targets is finally obtained by spatiotemporal context fusion, and flying targets is therefore picked out by analyzing fused presence probability map. A set of videos containing different drone models are selected for evaluation, and the comparisons against other algorithms demonstrate superiority of the proposed algorithm.

Video facial emotion recognition based on local enhanced motion history image and CNN-CTSLSTM networks

This paper focuses on the issue of recognition of facial emotion expressions in video sequences and proposes an integrated framework of two networks: a local network, and a global network, which are based on local enhanced motion history image (LEMHI) and CNN-LSTM cascaded networks respectively. In the local network, frames from unrecognized video are aggregated into a single frame by a novel method, LEMHI. This approach improves MHI by using detected human facial landmarks as attention areas to boost local value in difference image calculation, so that the action of crucial facial unit can be captured effectively. Then this single frame will be fed into a CNN network for prediction. On the other hand, an improved CNN-LSTM model is used as a global feature extractor and classifier for video facial emotion recognition in the global network. Finally, a random search weighted summation strategy is conducted as late-fusion fashion to final predication. Our work also offers an insight into networks and visible feature maps from each layer of CNN to decipher which portions of the face influence the networks’ predictions. Experiments on the AFEW, CK+ and MMI datasets using subject-independent validation scheme demonstrate that the integrated framework of two networks achieves a better performance than using individual network separately. Compared with state-of-the-arts methods, the proposed framework demonstrates a superior performance.

Assessment of microscopic repair dynamics in self-healing polymer by modeling laser speckle images

Self-healing in polymers plays an important role, since such polymers can spontaneously repair any damage introduced into them, and this in turn can enhance the durability and reliability of functional materials. Employment of the laser speckle imaging (LSI) technique can picture the microscopic repair dynamics of the healing process in real time. To assess the repair dynamics during the healing process, one needs to acquire a large amount of LSI data, and because of this the visual inspection of healing dynamics using such a data set can be a lengthy and tedious task. In order to support LSI processing, we demonstrate the application of a view-based temporal template method, the motion history image (MHI), to easily study the microscopic mechanisms of the self-healing process of polymers.The micromechanical dynamics of the healing process of a cut in both self-healing polymers as well as a non-healing version of a polymer was assessed using the MHI algorithm. The results of MHI are obtained much more easily than those of a visual inspection of all the LSI frames acquired during the healing process of the polymer.The MHI algorithm produced a visual modeling of the LSI images recorded during the healing process. The MHI algorithm could become an alternative tool to generate online results on the healing dynamics in polymers and other functional materials.

In-air hand gesture signature recognition system based on 3-dimensional imagery

A traditional online handwritten signature recognition system requires direct contact to acquisition device and usually will leave a traceable print on the surface. This made a signature possible and vulnerable to certain attempts of tracking and imitated. Looking into this shortfall, this paper proposes a novel approach to recognise an individual based on his/ her in-air hand motion while signing his/her signature. In this study, a low-cost acquisition device – Microsoft Kinect sensor is adopted to capture an image sequence of hand gesture signature. Palm region is first located and segmented through a predictive palm segmentation algorithm, which are then combined to generate a volume data. The volume data is condensed and reduced into a motion representation image by means of Motion History Image (MHI), which produces rich motion and temporal information. Several features are extracted from the MHI for empirical evaluation. Two classical recognition modes – identification and verification, are testified with an in-house database (HGS database). The proposed system achieves 90.4% identification accuracy and 3.22% equal error rate in verification mode. The experimental results substantiated the potential of the proposed system.

Modeling of laser speckles of heterogeneous dynamics in drying and aging paint dispersions using a view-based temporal template method

Assessments of the microscopic mechanism underlying the phenomenon of drying and aging of various paint dispersions are important for the requirements of sustainable and high-quality paints. The present work demonstrates the application of the motion history image (MHI) algorithm to visualize the microscopic mechanisms during drying and aging of different paints dispersions. We assessed the micromechanical dynamics of the drying process on laser speckle imaging data after generating the MHI maps. The processes such as cracking and delamination of paint on a glass substrate and the flow of an imbibition front on a porous substrate were analyzed using the MHI analysis, which identifies the locations where the phenomenon evolves with time.

A Review of Computational Approaches for Human Behavior Detection

Computer vision techniques capable of detecting human behavior are gaining interest. Several researchers have provided their review on behavior detection, however most of the reviews are focused on activity recognition only, and reviews on gesture and facial expression recognition are very few. Therefore, all of them lack to cover complete human behavior analysis. In this study, we provide a comprehensive review of human behavior detection approaches. The framework of this review is based on activity, gesture and facial expression recognition since these are the most important cues for behavior detection. These three areas are further classified in existing computational approaches. One can easily recognize from this review that hidden Markov model is widely exploited for activity recognition while motion history image is still a developing area. Haar-like features can be a valid alternative for gesture recognition. For facial expression recognition, local binary patterns feature is a very popular choice. We have reviewed behavior detection techniques, mostly developed after year 2009. The explicit advantages of this review are: (1) it provides a deep analysis of computational approaches for activity, gesture and facial expression recognition. (2) It includes both types of techniques that include single human as well as multiple human activities. (3) It considers techniques developed in the last decade only pertaining to information about the most recent techniques. (4) It provides a brief description of popular datasets used for activity, gesture and facial expression recognition. (5) It discusses open issues to provide an insight for future also.

Combining CNN streams of RGB-D and skeletal data for human activity recognition

Inspired by the success of deep learning methods, for human activity recognition based on individual vision cues, this paper presents a ConvNets based approach for activity recognition by combining multiple vision cues. Moreover, a new method of creating skeleton images, from skeleton joint sequences, representing motion information is presented in this paper. Motion representation images, namely, Motion History Image (MHI), Depth Motion Maps (DMMs) and skeleton images are constructed from RGB, depth and skeletal data of RGB-D sensor. These images are then separately trained on ConvNets and respective softmax scores are fused at the decision level. The combination of these distinct vision cues, leads to complete utilization of data, available from RGB-D sensor. To evaluate the effectiveness of the proposed 5-CNNs approach, we conduct our experiments on three well known and challenging RGB-D datasets, CAD-60, SBU Kinect interaction and UTD-MHAD. Results show that the proposed approach of combining multiple cues by means of decision level fusion is competitive with other state of the art methods.

Application of qualitative biospeckle methods for the identification of scar region in a green orange

This study demonstrates the feasibility of a view-based method, the motion history image (MHI) to map biospeckle activity around the scar region in a green orange fruit. The comparison of MHI with the routine intensity-based methods validated the effectiveness of the proposed method. The results show that MHI can be implementated as an alternative online image processing tool in the biospeckle analysis.

PMHI: Proposals From Motion History Images for Temporal Segmentation of Long Uncut Videos

This letter proposes a method for the generation of temporal action proposals for the segmentation of long uncut video sequences. The presence of consecutive multiple actions in video sequences makes the temporal segmentation a challenging problem due to the unconstrained nature of actions in space and time. To address this issue, we exploit the nonaction segments present between the actual human actions in uncut videos. From the long uncut video, we compute the energy of consecutive nonoverlapping motion history images (MHIs), which provides spatiotemporal information of motion. Our proposals from MHIs (PMHI) are based on clustering the MHIs into actions and nonaction segments by detecting minima from the energy of MHIs. PMHI efficiently segments the long uncut videos into a small number of nonoverlapping temporal action proposals. The strength of PMHI is that it is unsupervised, which alleviates the requirement for any training data. Our temporal action proposal method outperforms the existing proposal methods on the Multi-view Human Action video (MuHAVi)-uncut and Computer Vision and Pattern recognition (CVPR) 2012 Change Detection datasets with an average recall rate of 86.1% and 86.0%, respectively.

Human action recognition based on scene semantics

Like outdoors, indoor security is also a critical problem and human action recognition in indoor area is still a hot topic. Most studies on human action recognition ignored the semantic information of a scene, whereas indoors contains varieties of semantics. Meanwhile, the depth sensor with color and depth data is more suitable for extracting the semantics context in human actions. Hence, this paper proposed an indoor action recognition method using Kinect based on the semantics of a scene. First, we proposed a trajectory clustering algorithm for a three-dimensional (3D) scene by combining the different characteristics of people such as the spatial location, movement direction, and speed. Based on the clustering results and scene context, it concludes a region of interest (ROI) extraction method for indoors, and dynamic time warping (DTW) is used to study the abnormal action sequences. Finally, the color and depth-data-based 3D motion history image (3D–MHI) features and the semantics context of the scene were combined to recognize human action. In the experiment, two datasets were tested and the results demonstrate that our semantics-based method performs better than other methods.

Human Action Recognition Based on Integrating Body Pose, Part Shape, and Motion

Human action recognition is a challenging problem, especially in the presence of multiple actors in the scene and/or viewpoint variations. In this paper, three modalities, namely, 3-D skeletons, body part images, and motion history image (MHI), are integrated into a hybrid deep learning architecture for human action recognition. The three modalities capture the main aspects of an action: body pose, part shape, and body motion. Although the 3-D skeleton modality captures the actor’s pose, it lacks information about the shape of the body parts as well as the shape of manipulated objects. This is the reason for including both the body-part images and the MHI as additional modalities. The deployed architecture combines convolution neural networks (CNNs), long short-term memory (LSTM), and a fine-tuned pre-trained architecture into a hybrid one. It is called MCLP: m ulti-modal C NN + L STM + VGG16 p re-trained on ImageNet. The MCLP consists of three sub-models: CL1D (for CNN1D + LSTM), CL2D (for CNN2D + LSTM), and CMHI (CNN2D for MHI), which simultaneously extract the spatial and temporal patterns in the three modalities. The decisions of these three sub-models are fused by a late multiply fusion module, which proved to yield better accuracy than averaging or maximizing fusion methods. The proposed combined model and its sub-models have been evaluated both individually and collectively on four public data sets: UTkinect Action3D, SBU Interaction, Florence3-D Action, and NTU RGB+D. Our recognition rates outperform the state-of-the-art rates on all the evaluated data sets.

Human action recognition: a construction of codebook by discriminative features selection approach

Human activity recognition (HAR) has significance in the domain of pattern recognition. HAR handles the complexity of human physical changes and heterogeneous formats of same human actions performed under dissimilar subjects. This research contributes a unique method focusing on the changes in human movement. The purpose is to identify and categorise human actions from video sequences. The interest points (IPs) are extracted from the subject video and motion history images (MHIs) are constructed and analysed after image segmentation. Discriminative features (DFs) are selected and the visual vocabulary is learned from the extracted DF (EDF). The EDF are then quantised by using visual vocabulary and images are represented based upon frequencies of visual words (VW). VW are formed from the EDF and then, a histogram of VW is developed based on the feature vectors extracted from MHI. These feature vectors are used for training support vector machine (SVM) for the classification of actions into various categories. Benchmark datasets like KTH, Weizmann and HMDB51 are used for evaluation and comparison with existing action recognition approaches depicts the better performance of adopted strategy.

Human action recognition: a construction of codebook by discriminative features selection approach

Human activity recognition (HAR) has significance in the domain of pattern recognition. HAR handles the complexity of human physical changes and heterogeneous formats of same human actions performed under dissimilar subjects. This research contributes a unique method focusing on the changes in human movement. The purpose is to identify and categorise human actions from video sequences. The interest points (IPs) are extracted from the subject video and motion history images (MHIs) are constructed and analysed after image segmentation. Discriminative features (DFs) are selected and the visual vocabulary is learned from the extracted DF (EDF). The EDF are then quantised by using visual vocabulary and images are represented based upon frequencies of visual words (VW). VW are formed from the EDF and then, a histogram of VW is developed based on the feature vectors extracted from MHI. These feature vectors are used for training support vector machine (SVM) for the classification of actions into various categories. Benchmark datasets like KTH, Weizmann and HMDB51 are used for evaluation and comparison with existing action recognition approaches depicts the better performance of adopted strategy.