Vision-based Human Activity Research Articles

Vision-based Human Activity Recognition Using Local Phase Quantization

Human activity recognition (HAR) has been the most active and interesting area of research in recent years due to its wide range of applications in the field, such as healthcare, security and surveillance, robotics, gaming, entertainment, etc. However, recognizing vision-based human activity is still a challenging as input sequences may have cluttered background, illumination conditions, occlusions, degradation of video quality, blurring, etc. In the literature, several state-of-the-art methods have been trained and tested on different datasets but have yet to perform adequately to a certain extent. Moreover, extracting potential features and combining appropriate methods is one of the most challenging tasks in realistic video. This paper proposes an efficient frequency-based blur-invariance local phase quantization feature extractor and multiclass SVM classifier that overcomes these challenges. The feature is invariant towards camera motion, misfocused optics, movements in the scene, and environmental conditions. The proposed feature vector is then fed to the classifier to recognize human activities. The experiment has conducted on two publicly available datasets, UCF101 and HMDB51, and has achieved 99.79% and 98.67% accuracies, respectively. The approach has also outperformed the existing state-of-the-art approaches in terms of computational cost without compromising the accuracy of HAR.

Weighted voting ensemble of hybrid CNN-LSTM Models for vision-based human activity recognition

Weighted voting ensemble of hybrid CNN-LSTM Models for vision-based human activity recognition

Hybrid HAR-CNN Model: A Hybrid Convolutional Neural Network Model for Predicting and Recognizing the Human Activity Recognition

Human activity recognition (HAR) is an active research area in computer vision from past several years and research is still continuing in this field due to the unavailability of perfect recognition system. The human activity recognition system it covers e-health, patient monitoring, assistive daily living activities, video surveillance, security and behaviour analysis, and sports analysis. Many researchers have suggested techniques that use visual perception to detect human activities. Researchers will need to address problems including light variations in human activity detection, interclass similarity between scenes, the surroundings and recording setting, and temporal variation in order to construct an efficient vision-based human activity recognition system. However, a significant drawback of many deep learning models is their inability to achieve satisfactory results in real-world scenarios due to the conflicts mentioned above. To address this challenge, we developed a hybrid HAR-CNN classifier aimed at enhancing the learning outcomes of Deep CNNs by combining two models: Improved CNN and VGG-19. Using the KTH dataset, we collected 6,000 images for training, validation, and testing of our proposed technique. Our research findings indicate that the Hybrid HAR-CNN model, which combines Improved CNN with VGG-19 Net, outperforms individual deep learning models such as Improved CNN and VGG-19 Net.

Hybrid optimized multimodal spatiotemporal feature fusion for vision-based sports activity recognition

Vision-based Human Activity Recognition (HAR) is a challenging research task in sports. This paper aims to track the player’s movements and recognize the different types of sports activities in videos. The proposed work aims in developing Hybrid Optimized Multimodal SpatioTemporal Feature Fusion (HOM-STFF) model using skeletal information for vision-based sports activity recognition. The proposed HOM-STFF model presents a deep multimodal feature fusion approach that combines the features that are generated from the multichannel-1DCNN and 2D-CNN network model using a concatenative feature fusion process. The fused features are fed into the 2-GRU model that generates temporal features for activity recognition. Nature-inspired Bald Eagle Search Optimizer (BESO) is applied to optimize the network weights during training. Finally, performance of the classification model is evaluated and compared for identifying different activities in sports videos. Experimentation was carried out with the three vision-based sports datasets namely, Sports Videos in the Wild (SVW), UCF50 sports action and Self-build dataset, which achieved accuracy rate of 0.9813, 0.9506 and 0.9733, respectively. The results indicate that the proposed HOM-STFF model outperforms the other state-of-the-art methods in terms of activity detection capability.

Time-weighted motion history image for human activity classification in sports

Vision-based human activity classification has remarkable potential for various applications in the sports context (e.g., motion analysis for performance enhancement, active sensing for athletes, etc.). Recently, learning-based human activity classifications have been widely researched. However, in sports scenes in which more detailed and player-specific classifications are required, this is a quite challenging task; in many cases, only a limited number of datasets are available, unlike daily movements such as walking or climbing stairs. Therefore, this paper proposes a time-weighted motion history image, an effective image sequence representation for learning-based human activity classification. Unlike conventional MHI based on the assumption that “the newer frame is more important,” our method generates importance-aware representation so that the predictor can “see” the frames that contribute to analyzing the specific human activity. Experimental results have shown the superiority of our method.

Human Activity Recognition Using Cascaded Dual Attention CNN and Bi-Directional GRU Framework.

Vision-based human activity recognition (HAR) has emerged as one of the essential research areas in video analytics. Over the last decade, numerous advanced deep learning algorithms have been introduced to recognize complex human actions from video streams. These deep learning algorithms have shown impressive performance for the video analytics task. However, these newly introduced methods either exclusively focus on model performance or the effectiveness of these models in terms of computational efficiency, resulting in a biased trade-off between robustness and computational efficiency in their proposed methods to deal with challenging HAR problem. To enhance both the accuracy and computational efficiency, this paper presents a computationally efficient yet generic spatial-temporal cascaded framework that exploits the deep discriminative spatial and temporal features for HAR. For efficient representation of human actions, we propose an efficient dual attentional convolutional neural network (DA-CNN) architecture that leverages a unified channel-spatial attention mechanism to extract human-centric salient features in video frames. The dual channel-spatial attention layers together with the convolutional layers learn to be more selective in the spatial receptive fields having objects within the feature maps. The extracted discriminative salient features are then forwarded to a stacked bi-directional gated recurrent unit (Bi-GRU) for long-term temporal modeling and recognition of human actions using both forward and backward pass gradient learning. Extensive experiments are conducted on three publicly available human action datasets, where the obtained results verify the effectiveness of our proposed framework (DA-CNN+Bi-GRU) over the state-of-the-art methods in terms of model accuracy and inference runtime across each dataset. Experimental results show that the DA-CNN+Bi-GRU framework attains an improvement in execution time up to 167× in terms of frames per second as compared to most of the contemporary action-recognition methods.

Human Activity Recognition Using Deep Learning : A Survey

With the use of deep learning algorithms from artificial intelligence (AI), several types of research have been conducted on video data. Object localization, behaviour analysis, scene understanding, scene labelling, human activity recognition (HAR), and event recognition make up the majority of them. Among all of them, HAR is one of the most difficult jobs and key areas of research in video data processing. HAR can be used in a variety of fields, including robotics, human-computer interaction, video surveillance, and human behaviour categorization. This research seeks to compare deep learning approaches on several benchmark video datasets for vision-based human activity detection. We suggest a brand-new taxonomy for dividing up the literature into CNN- and RNN-based methods. We further categorise these approaches into four subgroups and show several methodologies, their effectiveness, and experimental datasets. To illustrate the development of HAR techniques, a brief comparison is also provided with the handcrafted feature-based approach and its merger with deep learning. Finally, we go over potential future research areas and some unresolved issues with recognising human activities. This survey's goal is to present the most recent developments in HAR techniques for vision-based deep learning using the most recent literature analysis.

I3D Light - A Simple Motion Information Stream for I3D

Vision-based Human Activity Recognition (HAR) aims to recognize human activities based on the analysis of video data, and has extensive applications in modern industry and human life. Inflated 3D (I3D) is a deep learning architecture that is commonly used for action recognition by using two-stream video data: RGB stream and optical flow stream. I3D achieved great success on a variety of action recognition benchmarks. However, the use of optical flow incurs high computational cost, making the approach unsuitable for real-time applications. We propose an alternative simple motion information extractor to replace the optical flow branch and reduce the computational cost. It is a modified I3D that uses 128 frames of 112x112 images as input. The low spatial resolution and long temporal range of the proposed I3D RGB stream can reduce the spatial information and enhance the motion information. Experiments show that this simple motion stream can increase the accuracy of the original I3D spatial stream by 4.09% on the Kinetics 400 dataset.

Human activity recognition based on an amalgamation of CEV & SGM features

The method of marking video clips with action symbols is known as vision-based human activity recognition. Robust solutions to this problem have a variety of practical implementations. Due to differences in motion performance, recording environments, and inter-personal differences, the challenge is difficult. We specifically resolve these problems in this study work, and we solve imitations of state-of-the-art research. Projected human activity recognition is based on an amalgamation of CEV & SGM features. The proposed solution outperforms current models and produces state-of-the-art outcomes as compared to the best effectiveness of the control, according to experimental results on the datasets.

Internet-of-Things-Based Suspicious Activity Recognition Using Multimodalities of Computer Vision for Smart City Security

Automatic human activity recognition is one of the milestones of smart city surveillance projects. Human activity detection and recognition aim to identify the activities based on the observations that are being performed by the subject. Hence, vision-based human activity recognition systems have a wide scope in video surveillance, health care systems, and human-computer interaction. Currently, the world is moving towards a smart and safe city concept. Automatic human activity recognition is the major challenge of smart city surveillance. The proposed research work employed fine-tuned YOLO-v4 for activity detection, whereas for classification purposes, 3D-CNN has been implemented. Besides the classification, the presented research model also leverages human-object interaction with the help of intersection over union (IOU). An Internet of Things (IoT) based architecture is implemented to take efficient and real-time decisions. The dataset of exploit classes has been taken from the UCF-Crime dataset for activity recognition. At the same time, the dataset extracted from MS-COCO for suspicious object detection is involved in human-object interaction. This research is also applied to human activity detection and recognition in the university premises for real-time suspicious activity detection and automatic alerts. The experiments have exhibited that the proposed multimodal approach achieves remarkable activity detection and recognition accuracy.

A Review of Deep Learning-based Human Activity Recognition on Benchmark Video Datasets

ABSTRACT Different types of research have been done on video data using Artificial Intelligence (AI) deep learning techniques. Most of them are behavior analysis, scene understanding, scene labeling, human activity recognition (HAR), object localization, and event recognition. Among all these, HAR is one of the challenging tasks and thrust areas of video data processing research. HAR is applicable in different areas, such as video surveillance systems, human-computer interaction, human behavior characterization, and robotics. This paper aims to present a comparative review of vision-based human activity recognition with the main focus on deep learning techniques on various benchmark video datasets comprehensively. We propose a new taxonomy for categorizing the literature as CNN and RNN-based approaches. We further divide these approaches into four sub-categories and present various methodologies with their experimental datasets and efficiency. A short comparison is also made with the handcrafted feature-based approach and its fusion with deep learning to show the evolution of HAR methods. Finally, we discuss future research directions and some open challenges on human activity recognition. The objective of this survey is to give the current progress of vision-based deep learning HAR methods with the up-to-date study of literature.

A Novel Central Camera Calibration Method Recording Point-to-Point Distortion for Vision-Based Human Activity Recognition.

The camera is the main sensor of vison-based human activity recognition, and its high-precision calibration of distortion is an important prerequisite of the task. Current studies have shown that multi-parameter model methods achieve higher accuracy than traditional methods in the process of camera calibration. However, these methods need hundreds or even thousands of images to optimize the camera model, which limits their practical use. Here, we propose a novel point-to-point camera distortion calibration method that requires only dozens of images to get a dense distortion rectification map. We have designed an objective function based on deformation between the original images and the projection of reference images, which can eliminate the effect of distortion when optimizing camera parameters. Dense features between the original images and the projection of the reference images are calculated by digital image correlation (DIC). Experiments indicate that our method obtains a comparable result with the multi-parameter model method using a large number of pictures, and contributes a 28.5% improvement to the reprojection error over the polynomial distortion model.

Weighted Voting Ensemble of Different Cnn-Lstm Models for Vision-Based Human Activity Recognition Using the Stair Actions Dataset

Weighted Voting Ensemble of Different Cnn-Lstm Models for Vision-Based Human Activity Recognition Using the Stair Actions Dataset

Semi-Supervised Image-to-Video Adaptation for Video Action Recognition

Given a sequence of images i.e. video, the task given a sequence of images i.e. video, the task of action recognition is to identify the most same action among the action sequences learned by the system. Such human action recognition is based on evidence gathered from videos. It has a lot of applications including surveillance, video indexing, biometrics, telehealth, and human-computer interaction. Vision-based human activity recognition is plagued by numerous challenges thanks to reading changes, occlusion, variation in execution rate, camera motion, and background clutter. In this survey, we provide an overview and report of the existing methods based on their ability to handle these challenges as well as how these methods can be generalized and their ability to detect abnormal actions. Such systematic classification can facilitate researchers to spot the acceptable ways on the market to deal with every one of the challenges visaged and their limitations. In addition to this, we also identify the public datasets and the challenges posed by them. From this survey, we have a tendency to draw conclusions relating to however well a challenge has been resolved, and that we determine potential analysis areas that need more work.

Vision-based human activity recognition for reducing building energy demand

Occupancy behaviour in buildings can impact the energy performance and the operation of heating, ventilation and air-conditioning systems. To ensure building operations become optimised, it is vital to develop solutions that can monitor the utilisation of indoor spaces and provide occupants’ actual thermal comfort requirements. This study presents the analysis of the application of a vision-based deep learning approach for human activity detection and recognition in buildings. A convolutional neural network was employed to enable the detection and classification of occupancy activities. The model was deployed to a camera that enabled real-time detections, giving an average detection accuracy of 98.65%. Data on the number of occupants performing each of the selected activities were collected, and deep learning–influenced profile was generated. Building energy simulation and various scenario-based cases were used to assess the impact of such an approach on the building energy demand and provide insights into how the proposed detection method can enable heating, ventilation and air-conditioning systems to respond to occupancy’s dynamic changes. Results indicated that the deep learning approach could reduce the over- or under-estimation of occupancy heat gains. It is envisioned that the approach can be coupled with heating, ventilation and air-conditioning controls to adjust the setpoint based on the building space’s actual requirements, which could provide more comfortable environments and minimise unnecessary building energy loads. Practical application Occupancy behaviour has been identified as an important issue impacting the energy demand of building and heating, ventilation and air-conditioning systems. This study proposes a vision-based deep learning approach to capture, detect and recognise in real-time the occupancy patterns and activities within an office space environment. Initial building energy simulation analysis of the application of such an approach within buildings was performed. The proposed approach is envisioned to enable heating, ventilation and air-conditioning systems to adapt and make a timely response based on occupancy’s dynamic changes. The results presented here show the practicality of such an approach that could be integrated with heating, ventilation and air-conditioning systems for various building spaces and environments.

Vision-based human activity recognition: a survey

Human activity recognition (HAR) systems attempt to automatically identify and analyze human activities using acquired information from various types of sensors. Although several extensive review papers have already been published in the general HAR topics, the growing technologies in the field as well as the multi-disciplinary nature of HAR prompt the need for constant updates in the field. In this respect, this paper attempts to review and summarize the progress of HAR systems from the computer vision perspective. Indeed, most computer vision applications such as human computer interaction, virtual reality, security, video surveillance and home monitoring are highly correlated to HAR tasks. This establishes new trend and milestone in the development cycle of HAR systems. Therefore, the current survey aims to provide the reader with an up to date analysis of vision-based HAR related literature and recent progress in the field. At the same time, it will highlight the main challenges and future directions.

Sensor-based and vision-based human activity recognition: A comprehensive survey

Human activity recognition (HAR) technology that analyzes data acquired from various types of sensing devices, including vision sensors and embedded sensors, has motivated the development of various context-aware applications in emerging domains, e.g., the Internet of Things (IoT) and healthcare. Even though a considerable number of HAR surveys and review articles have been conducted previously, the major/overall HAR subject has been ignored, and these studies only focus on particular HAR topics. Therefore, a comprehensive review paper that covers major subjects in HAR is imperative. This survey analyzes the latest state-of-the-art research in HAR in recent years, introduces a classification of HAR methodologies, and shows advantages and weaknesses for methods in each category. Specifically, HAR methods are classified into two main groups, which are sensor-based HAR and vision-based HAR, based on the generated data type. After that, each group is divided into subgroups that perform different procedures, including the data collection, pre-processing methods, feature engineering, and the training process. Moreover, an extensive review regarding the utilization of deep learning in HAR is also conducted. Finally, this paper discusses various challenges in the current HAR topic and offers suggestions for future research.

Computer Vision-based Human Activity Recognition for Elderly Home Care

Video-based monitoring of elderly people at home receives more attention in recent days. In this paper, we propose a novel approach to develop smart monitoring system for elderly people using computer vision techniques. Gaussian Mixture Model (GMM) based algorithm is used for background and foreground separation inorder to track the activities of human object. The minimum bounding box of the human object is traced and features like major axis length, minor axis length and orientation angle are extracted. The proposed approach is evaluated on the video sequences of fall dataset.

Human activity recognition using 2D skeleton data and supervised machine learning

Vision-based human activity recognition (HAR) finds its application in many fields such as video surveillance, robot navigation, telecare and ambient intelligence. Most of the latest researches in the field of automated HAR based on skeleton data use depth devices such as Kinect to obtain three-dimensional (3D) skeleton information directly from the camera. Although these researches achieve high accuracy but are strictly device dependent and cannot be used for videos other than from specific cameras. Current work focuses on the use of only 2D skeletal data, extracted from videos obtained through any standard camera, for activity recognition. Appearance and motion features were extracted using 2D positions of human skeletal joints through OpenPose library. The approach was trained and tested on publically available datasets. Supervised machine learning was implemented for recognising four activity classes including sit, stand, walk and fall. Performance of five techniques including K-nearest neighbours (KNNs), support vector machine, Naive Bayes, linear discriminant and feed-forward back-propagation neural network was compared to find the best classifier for the proposed method. All techniques performed well with best results obtained through the KNN classifier.

Video benchmarks of human action datasets: a review

Vision-based Human activity recognition is becoming a trendy area of research due to its wide application such as security and surveillance, human–computer interactions, patients monitoring system, and robotics. In the past two decades, there are several publically available human action, and activity datasets are reported based on modalities, view, actors, actions, and applications. The objective of this survey paper is to outline the different types of video datasets and highlights their merits and demerits under practical considerations. Based on the available information inside the dataset we can categorise these datasets into RGB (Red, Green, and Blue) and RGB-D(depth). The most prominent challenges involved in these datasets are occlusions, illumination variation, view variation, annotation, and fusion of modalities. The key specification of these datasets is discussed such as resolutions, frame rate, actions/actors, background, and application domain. We have also presented the state-of-the-art algorithms in a tabular form that give the best performance on such datasets. In comparison with earlier surveys, our works give a better presentation of datasets on the well-organised comparison, challenges, and latest evaluation technique on existing datasets.