Depth Silhouettes Research Articles

People Detection with Depth Silhouettes and Convolutional Neural Networks on a Mobile Robot

People Detection with Depth Silhouettes and Convolutional Neural Networks on a Mobile Robot

Depth Maps-Based Human Segmentation and Action Recognition Using Full-Body Plus Body Color Cues Via Recognizer Engine

Assessment of human behavior during performance of daily routine actions at indoor areas plays a significant role in healthcare services and smart homes for elderly and disabled people. During this consideration, initially, depth images are captured using depth camera and segment human silhouettes due to color and intensity variation. Features considered spatiotemporal properties and obtained from the human body color joints and depth silhouettes information. Joint displacement and specific-motion features are obtained from human body color joints and side-frame differentiation features are processed based on depth data to improve classification performance. Lastly, recognizer engine is used to recognize different activities. Unlike conventional results that were evaluated using a single dataset, our experimental results have shown state-of-the-art accuracy of 88.9% and 66.70% over two challenging depth datasets. The proposed system should be serviceable with major contributions in different consumer application systems such as smart homes, video surveillance and health monitoring systems.

Recognition of Symbolic Gestures Using Depth Information

Symbolic gestures are the hand postures with some conventionalized meanings. They are static gestures that one can perform in a very complex environment containing variations in rotation and scale without using voice. The gestures may be produced in different illumination conditions or occluding background scenarios. Any hand gesture recognition system should find enough discriminative features, such as hand-finger contextual information. However, in existing approaches, depth information of hand fingers that represents finger shapes is utilized in limited capacity to extract discriminative features of fingers. Nevertheless, if we consider finger bending information (i.e., a finger that overlaps palm), extracted from depth map, and use them as local features, static gestures varying ever so slightly can become distinguishable. Our work here corroborated this idea and we have generated depth silhouettes with variation in contrast to achieve more discriminative keypoints. This approach, in turn, improved the recognition accuracy up to 96.84%. We have applied Scale-Invariant Feature Transform (SIFT) algorithm which takes the generated depth silhouettes as input and produces robust feature descriptors as output. These features (after converting into unified dimensional feature vectors) are fed into a multiclass Support Vector Machine (SVM) classifier to measure the accuracy. We have tested our results with a standard dataset containing 10 symbolic gesture representing 10 numeric symbols (0-9). After that we have verified and compared our results among depth images, binary images, and images consisting of the hand-finger edge information generated from the same dataset. Our results show higher accuracy while applying SIFT features on depth images. Recognizing numeric symbols accurately performed through hand gestures has a huge impact on different Human-Computer Interaction (HCI) applications including augmented reality, virtual reality, and other fields.

A Depth Video-based Human Detection and Activity Recognition using Multi-features and Embedded Hidden Markov Models for Health Care Monitoring Systems

Increase in number of elderly people who are living independently needs especial care in the form of healthcare monitoring systems. Recent advancements in depth video technologies have made human activity recognition (HAR) realizable for elderly healthcare applications. In this paper, a depth video-based novel method for HAR is presented using robust multi-features and embedded Hidden Markov Models (HMMs) to recognize daily life activities of elderly people living alone in indoor environment such as smart homes. In the proposed HAR framework, initially, depth maps are analyzed by temporal motion identification method to segment human silhouettes from noisy background and compute depth silhouette area for each activity to track human movements in a scene. Several representative features, including invariant, multi-view differentiation and spatiotemporal body joints features were fused together to explore gradient orientation change, intensity differentiation, temporal variation and local motion of specific body parts. Then, these features are processed by the dynamics of their respective class and learned, modeled, trained and recognized with specific embedded HMM having active feature values. Furthermore, we construct a new online human activity dataset by a depth sensor to evaluate the proposed features. Our experiments on three depth datasets demonstrated that the proposed multi-features are efficient and robust over the state of the art features for human action and activity recognition.

Robust human activity recognition from depth video using spatiotemporal multi-fused features

The recently developed depth imaging technologies have provided new directions for human activity recognition (HAR) without attaching optical markers or any other motion sensors to human body parts. In this paper, we propose novel multi-fused features for online human activity recognition (HAR) system that recognizes human activities from continuous sequences of depth map. The proposed online HAR system segments human depth silhouettes using temporal human motion information as well as it obtains human skeleton joints using spatiotemporal human body information. Then, it extracts the spatiotemporal multi-fused features that concatenate four skeleton joint features and one body shape feature. Skeleton joint features include the torso-based distance feature (DT), the key joint-based distance feature (DK), the spatiotemporal magnitude feature (M) and the spatiotemporal directional angle feature (θ). The body shape feature called HOG-DDS represents the projections of the depth differential silhouettes (DDS) between two consecutive frames onto three orthogonal planes by the histogram of oriented gradients (HOG) format. The size of the proposed spatiotemporal multi-fused feature is reduced by a code vector in the code book which is generated by vector quantization method. Then, it trains the hidden Markov model (HMM) with the code vectors of the multi-fused features and recognizes the segmented human activity by the forward spotting scheme using the trained HMM-based human activity classifiers. The experimental results on three challenging depth video datasets such as IM-DailyDepthActivity, MSRAction3D and MSRDailyActivity3D demonstrate that the proposed online HAR method using the proposed multi-fused features outperforms the state-of-the-art HAR methods in terms of recognition accuracy.

Human Depth Sensors-Based Activity Recognition Using Spatiotemporal Features and Hidden Markov Model for Smart Environments

Nowadays, advancements in depth imaging technologies have made human activity recognition (HAR) reliable without attaching optical markers or any other motion sensors to human body parts. This study presents a depth imaging-based HAR system to monitor and recognize human activities. In this work, we proposed spatiotemporal features approach to detect, track, and recognize human silhouettes using a sequence of RGB-D images. Under our proposed HAR framework, the required procedure includes detection of human depth silhouettes from the raw depth image sequence, removing background noise, and tracking of human silhouettes using frame differentiation constraints of human motion information. These depth silhouettes extract the spatiotemporal features based on depth sequential history, motion identification, optical flow, and joints information. Then, these features are processed by principal component analysis for dimension reduction and better feature representation. Finally, these optimal features are trained and they recognized activity using hidden Markov model. During experimental results, we demonstrate our proposed approach on three challenging depth videos datasets including IM-DailyDepthActivity, MSRAction3D, and MSRDailyActivity3D. All experimental results show the superiority of the proposed approach over the state-of-the-art methods.

A Hybrid Feature Extraction Approach for Human Detection, Tracking and Activity Recognition Using Depth Sensors

This paper presents spatiotemporal hybrid features, human tracking, and activity recognition into a single framework from video sequences captured by a RGB-D sensor. Initially, we received a sequence of depth maps to extract human silhouettes from the noisy background and track them using temporal human motion information from each frame. Then, hybrid features as optical flow motion features and distance parameters features are extracted from the depth silhouette region and used in an augmented form to work as patiotemporal features. In order to represent each activity in a better way, the augmented features are being clustered and symbolized by self-organization maps. Finally, these features are then processed by hidden Markov models to train and recognize human activities based on transition and emission probabilities values. The experimental results show the superiority of the proposed method over the state-of-the-art methods using two challenging depth images datasets.

3-D human pose recovery using nonrigid point set registration and body part tracking of depth data

In this paper, we present a novel approach for recovering a 3-D pose from a single human body depth silhouette using nonrigid point set registration and body part tracking. In our method, a human body depth silhouette is presented as a set of 3-D points and matched to another set of 3-D points using point correspondences. To recognize and maintain body part labels, we initialize the first set of points to corresponding human body parts, resulting in a body part-labeled map. Then, we transform the points to a sequential set of points based on point correspondences determined by nonrigid point set registration. After point registration, we utilize the information from tracked body part labels and registered points to create a human skeleton model. A 3-D human pose gets recovered by mapping joint information from the skeleton model to a 3-D synthetic human model. Quantitative and qualitative evaluation results on synthetic and real data show that complex human poses can be recovered more reliably with lower errors compared to other conventional techniques for 3-D pose recovery.

Hand number gesture recognition using recognized hand parts in depth images

In this paper, we present a novel approach of recognizing hand number gestures using the recognized hand parts in a depth image. Our proposed approach is divided into two stages: (i) hand parts recognition by random forests (RFs) and (ii) rule-based hand number gestures recognition. In the first stage, we create a database (DB) of synthetic hand depth silhouettes and their corresponding hand parts-labeled maps and then train RFs with the DB. Via the trained RFs, we recognize or label the hand parts in a depth silhouette. In the second stage, based on the information of the recognized or labeled hand parts, hand number gestures are recognized according to our derived rules. In our experiments, we quantitatively and qualitatively evaluated our hand parts recognition system with synthetic and real data. Then, we tested our hand number gesture recognition system with real data. Our results show the average recognition rate of 97.80 % over the ten hand number gestures from five different subjects.

A depth video sensor-based life-logging human activity recognition system for elderly care in smart indoor environments.

Recent advancements in depth video sensors technologies have made human activity recognition (HAR) realizable for elderly monitoring applications. Although conventional HAR utilizes RGB video sensors, HAR could be greatly improved with depth video sensors which produce depth or distance information. In this paper, a depth-based life logging HAR system is designed to recognize the daily activities of elderly people and turn these environments into an intelligent living space. Initially, a depth imaging sensor is used to capture depth silhouettes. Based on these silhouettes, human skeletons with joint information are produced which are further used for activity recognition and generating their life logs. The life-logging system is divided into two processes. Firstly, the training system includes data collection using a depth camera, feature extraction and training for each activity via Hidden Markov Models. Secondly, after training, the recognition engine starts to recognize the learned activities and produces life logs. The system was evaluated using life logging features against principal component and independent component features and achieved satisfactory recognition rates against the conventional approaches. Experiments conducted on the smart indoor activity datasets and the MSRDailyActivity3D dataset show promising results. The proposed system is directly applicable to any elderly monitoring system, such as monitoring healthcare problems for elderly people, or examining the indoor activities of people at home, office or hospital.

Real-time 3D human pose recovery from a single depth image using principal direction analysis

In this paper, we present a novel approach to recover a 3D human pose in real-time from a single depth image using principal direction analysis (PDA). Human body parts are first recognized from a human depth silhouette via trained random forests (RFs). PDA is applied to each recognized body part, which is presented as a set of points in 3D, to estimate its principal direction. Finally, a 3D human pose is recovered by mapping the principal direction to each body part of a 3D synthetic human model. We perform both quantitative and qualitative evaluations of our proposed 3D human pose recovering methodology. We show that our proposed approach has a low average reconstruction error of 7.07 degrees for four key joint angles and performs more reliably on a sequence of unconstrained poses than conventional methods. In addition, our methodology runs at a speed of 20 FPS on a standard PC, indicating that our system is suitable for real-time applications. Our 3D pose recovery methodology is applicable to applications ranging from human computer interactions to human activity recognition.

Depth video-based gait recognition for smart home using local directional pattern features and hidden Markov model

Gait recognition at smart home is considered as a primary function of the smart system nowadays. The significance of gait recognition is high especially for the elderly as gait is one of the basic activities to promote and preserve their health. In this work, a novel method was proposed for human gait recognition by processing depth videos from a depth camera. The gait recognition method utilizes local directional patterns (LDPs) for local feature extraction from depth silhouettes and hidden Markov models (HMMs) for recognition. The LDP features were first extracted from the depth silhouettes of a human body from each frame of a video containing human gait. The dimension of the LDP features was reduced by principal component analysis. Then, each HMM was trained using the LDP features. Finally, the recognition was done with a maximum likelihood calculation of the trained HMMs of different gaits. We focused on training and recognizing two kinds of gaits here, namely, normal and abnormal. The proposed approach shows superior recognition performance over other traditional methods of gait recognition.

Hand Gesture Recognition and Interface via a Depth Imaging Sensor for Smart Home Appliances

This paper presents a novel hand gesture interface system via a depth imaging sensor for appliances control in smart home environments. To control appliances with hand gestures, we recognize the hand parts in a depth hand silhouette and generate control commands. In our methodologies, we first create a database (DB) of synthetic hand depth silhouettes and their corresponding hand parts-labelled maps and then train a random forests (RFs) classifier with the DB. Via the trained RFs, we recognize the hand parts in a depth silhouette. Then based on the information of the recognized hand parts, control commands are finally generated according to our implemented control interface. By testing our system on real hand gestures, we have obtained an average recognition rate of 98.50% from five different subjects. With the presented interface system, users can control smart home appliances such as TV, fan, lighting, doors and change channels, temperature, and volume by just hand gestures.

A Spatiotemporal Robust Approach for Human Activity Recognition

Nowadays, human activity recognition is considered to be one of the fundamental topics in computer vision research areas, including human-robot interaction. In this work, a novel method is proposed utilizing the depth and optical flow motion information of human silhouettes from video for human activity recognition. The recognition method utilizes enhanced independent component analysis (EICA) on depth silhouettes, optical flow motion features, and hidden Markov models (HMMs) for recognition. The local features are extracted from the collection of the depth silhouettes exhibiting various human activities. Optical flow-based motion features are also extracted from the depth silhouette area and used in an augmented form to form the spatiotemporal features. Next, the augmented features are enhanced by generalized discriminant analysis (GDA) for better activity representation. These features are then fed into HMMs to model human activities and recognize them. The experimental results show the superiority of the proposed approach over the conventional ones.

An Indoor Human Activity Recognition System for Smart Home Using Local Binary Pattern Features with Hidden Markov Models

Smart home technologies are getting considerable attentions nowadays for better care of the residents, especially the elderly. One of the key technologies is the human activity recognition (HAR) system which automatically recognizes various indoor activities of a resident and reacts upon the needs of the resident, known as a proactive system. In this work, we propose a novel HAR system that utilizes depth imaging. Our HAR system utilizes local binary patterns (LBP) as local activity features from depth silhouettes and recognizes human activities via Hidden Markov Model (HMM). In our methodology, first LBP features were extracted from depth human body silhouettes from each frame of a video containing human activity. Then, principal component analysis (PCA) and linear discriminant analysis (LDA) were performed over the LBP features to obtain condensed features. Applying these features, each activity HMM was trained. Finally, HAR was performed with the trained HMMs. Our approach shows superior recognition performance over the traditional silhouette feature-based approaches. The system should be practical to be used for smart homes.

Human Activity Recognition via Recognized Body Parts of Human Depth Silhouettes for Residents Monitoring Services at Smart Home

Human activity recognition (HAR) is an emerging methodology essential for smart homes with practical applications such as personal lifecare and healthcare services for the elderly and disabled people. In this work, we present a novel HAR methodology utilizing the recognized body parts of human depth silhouettes and Hidden Markov Models (HMMs). We first create a database of synthetic depth silhouettes and their corresponding body parts labelled silhouettes of various human activities to train random forests (RFs). With the trained RFs, a set of 23 body parts are recognized from incoming depth silhouettes, yielding a set of centroids from the identified body parts. From the dynamics of these centroids, motion parameters are computed: a set of magnitude and directional angle features. Finally, the spatio-temporal dynamics of these motion features of various activities are used to train HMMs. We have performed HAR with the trained HMMs for six typical home activities and obtained the mean recognition rate of 97.16%. The presented HAR methodology should be useful for residents monitoring services at smart homes.

Depth video-based human activity recognition system using translation and scaling invariant features for life logging at smart home

Video-based human activity recognition systems have potential contributions to various applications such as smart homes and healthcare services. In this work, we present a novel depth video-based translation and scaling invariant human activity recognition (HAR) system utilizing R transformation of depth silhouettes. To perform HAR in indoor settings, an invariant HAR method is critical to freely perform activities anywhere in a camera view without translation and scaling problems of human body silhouettes. We obtain such invariant features via R transformation on depth silhouettes. Furthermore, in R transforming depth silhouettes, shape information of human body reflected in depth values is encoded into the features. In R transformation, 2D feature maps are computed first through Radon transform of each depth silhouette followed by computing 1D feature profile through R transform to get the translation and scaling invariant features. Then, we apply Principle Component Analysis (PCA) for dimension reduction and Linear Discriminant Analysis (LDA) to make the features more prominent, compact and robust. Finally, Hidden Markov Models (HMMs) are used to train and recognize different human activities. Our proposed system shows superior recognition rate over the conventional approaches, reaching up to the mean recognition rate of 93.16% for six typical human activities whereas the conventional PC and IC-based depth silhouettes achieved only 74.83% and 86.33% ,while binary silhouettes-based R transformation approach achieved 67.08% respectively. Our experimental results show that the proposed method is robust, reliable, and efficient in recognizing the daily human activities.

Human activity recognition based on independent depth silhouette components and optical flow features

In this work, we propose a novel method that utilises video-based depth and optical flow information of human body movement for human activity recognition. The recognition method utilises independent components (ICs) of depth silhouettes and optical flow-based motion features from a series of depth images and discrete hidden Markov models (HMMs) for recognition. The IC features are extracted from a collection of depth silhouettes containing various human activities, over which linear discriminant analysis (LDA) is performed for better classification. Furthermore, to improve the recognition performance, optical flow-based motion features extracted from the consecutive depth silhouettes are utilised in an augmented form. In addition, discrete HMMs are employed to recognise and model the time-sequential features of human activities. Our results show that the depth silhouette feature-based approach provides better human activity recognition than most generally used binary silhouette-based approaches, and that the augmented depth IC and optical flow features provide additional improvements.

Recognition of Human Home Activities via Depth Silhouettes and ℜ Transformation for Smart Homes

This paper presents a novel human home activity recognition (HAR) system designed for smart homes that utilize depth silhouettes and ℜ transformation to continuously recognize the daily activities of the elderly and disabled in an indoor environment for better lifecare and e-healthcare services. Previously, ℜ transformation has been applied only on binary silhouettes that provide only the shape information of human activities. In this work, ℜ transformation was utilized on depth silhouettes such that the depth information of human body parts could be used in HAR in addition to the shape information. In ℜ transformation, 2D directional projection maps are computed via Radon transform, and then 1D feature profiles, which are translation and scaling invariants. Then, by applying the principle component analysis and linear discriminant analysis, the prominent activity features would be extracted. Finally, Hidden Markov Models would be used to train and recognize daily home activities. The results showed a mean recognition rate of 96.55% over ten typical home activities, whereas the same system utilizing binary silhouettes could achieve only 85.75%. The proposed methodology should be useful in designing and developing a compact HAR system that can be practically used in a smart environment including smart home for the care of the elderly, infirmed or disabled people.

Video-based Indoor Human Gait Recognition Using Depth Imaging and Hidden Markov Model: A Smart System for Smart Home

Smart homes that are capable of home healthcare and e-Health services are receiving much attention due to their potential for better care of the elderly and disabled in an indoor environment. Recently the Center for Sustainable Healthy Buildings at Kyung Hee University has developed a novel indoor human activity recognition methodology based on depth imaging of a user’s activities. This system utilizes Independent Component Analysis to extract spatiotemporal features from a series of depth silhouettes of various activities. To recognise the activities from the spatiotemporal features, trained Hidden Markov Models of the activities would be used. In this study, this technique has been extended to recognise human gaits (including normal and abnormal). Since this system could be of great significance for the caring of the elderly, to promote and preserve their health and independence, the gait recognition system would be considered a primary function of the smart system for smart homes. The indoor gait recognition system is trained to detect abnormal gait patterns and generate warnings. The system works in real-time and is aimed to be installed at smart homes. This paper provides the information for further development of the system for their application in the future.