Human Silhouette Research Articles

Human movement monitoring system for classification of strength exercises and verification of their execution technique

Human movement analysis is critical to optimizing sports training and influencing exercise intensity and effectiveness. In the age of modern technology, more and more advanced systems are emerging to support coaches and expand the range of analysis performed. This article aims to verify that artificial intelligence, together with machine learning algorithms, can accurately classify exercises in a dynamic gym environment and effectively assess the correctness of their performance. For the initial analysis of movement, the Google MediaPipe Pose model was used, which was responsible for detecting the human silhouette and determining the coordinates of the position of critical joints. Based on these coordinates, the angles between each joint were calculated, and then their sequences were further analyzed. The sequences were analyzed using the following three algorithms: support vector machine (SVM), dense neural network, and LSTM recurrent network. As a result, the system based on recurrent LSTM networks achieved the best prediction efficiency of approximately 98%, enabling accurate exercise classification. Subsequently, verification of the activities' correctness was also carried out, and the system, based on recursive LSTM networks, again achieved the best efficiency, this time equal to 96% on average for all exercises. On this basis, it was concluded that the discussed approach enables practical analysis of human movement, which can significantly improve training methods and facilitate coaching work.

Localizing somatic symptoms associated with childhood maltreatment

Childhood maltreatment has been linked to adult somatic symptoms, although this has rarely been examined in daily life. Furthermore, the localization of somatization associated with childhood maltreatment and its subtypes is unknown. This large-scale experience sampling study used body maps to examine the relationships between childhood maltreatment, its subtypes, and the intensity and location of negative somatic sensations in daily life. Participants (N = 2,234; 33% female and 67% male) were part of MyBPLab 2.0, a study conducted using a bespoke mobile phone application. Four categories of childhood maltreatment (emotional abuse, emotional neglect, physical abuse, and physical neglect) were measured using the Childhood Trauma Questionnaire. Using gender-matched human silhouettes, participants indicated the location and intensity of feelings of negative activation in the body. Childhood maltreatment generally and its four measured subtypes were all positively associated with heightened negative activation on both the front and back body maps. For females, total childhood maltreatment was associated with negative activation in the abdomen and lower back, while for males, the association was localized to the lower back. Similarly, each of the four subscales had localized associations with negative activation in the abdomen and lower back in females and lower back in males, except for emotional abuse, which was also associated with negative activation in the abdomen in males. These associations likely reflect increased somatization in individuals exposed to childhood maltreatment, suggesting a role for psychotherapeutic interventions in alleviating associated distress.

3D far-field Lidar sensing and computational modeling for human identification.

3D sensors offer depth sensing that may be used for task-specific data processing and computational modeling. Many existing methods for human identification using 3D depth sensors primarily focus on Kinect data, where the range is very limited. This work considers a 3D long-range Lidar sensor for far-field imaging of human subjects in 3D Lidar full motion video (FMV) of "walking" action. 3D Lidar FMV data for human subjects are used to develop computational modeling for automated human silhouette and skeleton extraction followed by subject identification. We propose a matrix completion algorithm to handle missing data in 3D FMV due to self-occlusion and occlusion from other subjects for 3D skeleton extraction. We further study the effect of noise in the 3D low resolution far-field Lidar data in human silhouette extraction performance of the model. Moreover, this work addresses challenges associated with far-field 3D Lidar including learning with a limited amount of data and low resolution. Moreover, we evaluate the proposed computational algorithm using a gallery of 10 subjects for human identification and show that our method is competitive with the state-of-the-art OpenPose and V2VPose skeleton extraction models using the same dataset for human identification.

GaitAMR: Cross-view gait recognition via aggregated multi-feature representation

Gait recognition is an emerging long-distance biometric technology applied in many fields, including video surveillance. The most recent gait recognition methods treat human silhouettes as global or local regions to extract the gait properties. However, the global approach may cause the fine-grained differences of limbs to be ignored, whereas the local approach focuses only on the details of body parts and cannot consider the correlation between adjacent regions. Moreover, as a multi-view task, view changes have a significant impact on the integrity of the silhouette, which necessitates considering the disturbances brought about by the view itself. To address these problems, this paper proposes a novel gait recognition framework, namely, gait aggregation multi-feature representation (GaitAMR), to extract the most discriminative subject features. In GaitAMR, we propose a holistic and partial temporal aggregation strategy, that extracts body movement descriptors, both globally and locally. Besides, we use the optimal view features as supplementary information for spatiotemporal features, and thus enhance the view stability in the recognition process. By effectively aggregating feature representations from different domains, our method enhances the discrimination of gait patterns between subjects. Experimental results on public gait datasets show that GaitAMR improves gait recognition in occlusion conditions, outperforming state-of-the-art methods.

Model-based and model-free deep features fusion for high performed human gait recognition.

In the last decade, the need for a non-contact biometric model for recognizing candidates has increased, especially after the pandemic of COVID-19 appeared and spread worldwide. This paper presents a novel deep convolutional neural network (CNN) model that guarantees quick, safe, and precise human authentication via their poses and walking style. The concatenated fusion between the proposed CNN and a fully connected model has been formulated, utilized, and tested. The proposed CNN extracts the human features from two main sources: (1) human silhouette images according to model-free and (2) human joints, limbs, and static joint distances according to a model-based via a novel, fully connected deep-layer structure. The most commonly used dataset, CASIA gait families, has been utilized and tested. Numerous performance metrics have been evaluated to measure the system quality, including accuracy, specificity, sensitivity, false negative rate, and training time. Experimental results reveal that the proposed model can enhance recognition performance in a superior manner compared with the latest state-of-the-art studies. Moreover, the suggested system introduces a robust real-time authentication with any covariate conditions, scoring 99.8% and 99.6% accuracy in identifying casia (B) and casia (A) datasets, respectively.

No Sex Differences in the Attentional Bias for the Right Side of Human Bodies

Ambiguous silhouettes representing human individuals which perform unimanual actions are interpreted more often as right-handed. Such a preference might reflect a perceptual frequency effect, due to the fact that most social interactions occur with right-handers. As a consequence, observers would preferentially attend to the region in which others’ dominant hand usually falls, thus increasing the efficiency in monitoring both aggressive and communicative acts. Given that men can be more dangerous compared with women, the right-hand bias should be larger when observing male rather than female individuals, and given that aggressive interactions involve men more frequently than women, it should be larger in male rather than female observers. However, previous studies did not specifically test whether: (i) male—compared with female—observers pay more attention to the right side of others (regardless of the observed individuals’ sex), or (ii) observers (regardless of their sex) pay more attention to the right side of male—compared with female—individuals. Therefore, in the present study we used ambiguous human silhouettes rotating about their vertical axis with one arm extended in order to determine whether the rightward bias is larger for male rather than female figures and/or in male rather than female participants. According to our data, the bias toward the right side of human bodies was not significantly associated with either the figure’s or the participant’s sex.

An Elliptical Modeling Supported System for Human Action Deep Recognition Over Aerial Surveillance

The advancement of computer vision technology has led to the development of sophisticated algorithms capable of accurately recognizing human actions from red-green-blue videos recorded by drone cameras. Hence, possessing an exceptional potential, human action recognition also faces many challenges including, tendency of humans to perform the same action in different ways, limited camera angles, and field of view. In this research article, a system has been proposed to tackle the forementioned challenges by using red-green-blue videos as input while the videos were recorded by drone cameras. First of all, the video was split into its constituent frames and then gamma correction was applied on each frame to obtain an optimized version of the image. Then the Felzenszwalb’s algorithm performed the segmentation to segment out human from the input image and human silhouette was generated. Utilizing the silhouette, skeleton was extracted to spot thirteen body key points. The key points were then used to perform elliptical modeling to estimate the individual boundaries of the body parts while the elliptical modeling was governed by the Gaussian mixture model-expectation maximization algorithm. The elliptical models of the body parts were utilized to spot fiducial points that if tracked, could provide very useful information about the performed action. Some other features that were extracted for this study include, the 3d point cloud feature vector, relative distance and velocity of the key-points, and their mutual angles. The features were then forwarded for optimization under a quadratic discriminant analysis and finally, a convolutional neural network was trained to perform the action classification. Three benchmark datasets including, the Drone-Action dataset, the UAV-Human dataset, and the Okutama-Action dataset were used for a comprehensive experimentation. The system outperformed the state-of-the-art approaches by securing accuracies of 80.03%, 48.60%, and 78.01% over the Drone-Action dataset, the UAV-Human dataset, and the Okutama-Action dataset respectively.

RFMask: A Simple Baseline for Human Silhouette Segmentation With Radio Signals

Human silhouette segmentation, which is originally defined in computer vision, has achieved promising results for understanding human activities. However, the physical limitation makes existing systems based on optical cameras suffer from severe performance degradation under low illumination, smoke, and/or opaque obstruction conditions. To overcome such limitations, in this paper, we propose to utilize the radio signals, which can traverse obstacles and are unaffected by the lighting conditions to achieve silhouette segmentation. The proposed RFMask framework is composed of three modules. It first transforms RF signals captured by millimeter wave radar on two planes into spatial domain and suppress interference with the signal processing module. Then, it locates human reflections on RF frames and extract features from surrounding signals with human detection module. Finally, the extracted features from RF frames are aggregated with an attention based mask generation module. To verify our proposed framework, we collect a dataset containing <b>804,760</b> radio frames and <b>402,380</b> camera frames with human activities under various scenes. Experimental results show that the proposed framework can achieve impressive human silhouette segmentation even under the challenging scenarios (such as low light and occlusion scenarios) where traditional optical-camera-based methods fail. To the best of our knowledge, this is the first investigation towards segmenting human silhouette based on millimeter wave signals. We hope that our work can serve as a baseline and inspire further research that perform vision tasks with radio signals. The dataset and codes will be made in public.

Automatic Anomaly Monitoring in Public Surveillance Areas

With the dramatic increase in video surveillance applications and public safety measures, the need for an accurate and effective system for abnormal/suspicious activity classification also increases. Although it has multiple applications, the problem is very challenging. In this paper, a novel approach for detecting normal/abnormal activity has been proposed. We used the Gaussian Mixture Model (GMM) and Kalman filter to detect and track the objects, respectively. After that, we performed shadow removal to segment an object and its shadow. After object segmentation we performed occlusion detection method to detect occlusion between multiple human silhouettes and we implemented a novel method for region shrinking to isolate occluded humans. Fuzzy c-mean is utilized to verify human silhouettes and motion based features including velocity and optical flow are extracted for each identified silhouettes. Gray Wolf Optimizer (GWO) is used to optimize feature set followed by abnormal event classification that is performed using the XG-Boost classifier. This system is applicable in any surveillance application used for event detection or anomaly detection. Performance of proposed system is evaluated using University of Minnesota (UMN) dataset and UBI (University of Beira Interior)-Fight dataset, each having different type of anomaly. The mean accuracy for the UMN and UBI-Fight datasets is 90.14% and 76.9% respectively. These results are more accurate as compared to other existing methods.

Tracking and Analysis of Pedestrian’s Behavior in Public Places

Crowd management becomes a global concern due to increased population in urban areas. Better management of pedestrians leads to improved use of public places. Behavior of pedestrian’s is a major factor of crowd management in public places. There are multiple applications available in this area but the challenge is open due to complexity of crowd and depends on the environment. In this paper, we have proposed a new method for pedestrian’s behavior detection. Kalman filter has been used to detect pedestrian’s using movement based approach. Next, we have performed occlusion detection and removal using region shrinking method to isolate occluded humans. Human verification is performed on each human silhouette and wavelet analysis and particle gradient motion are extracted for each silhouettes. Gray Wolf Optimizer (GWO) has been utilized to optimize feature set and then behavior classification has been performed using the Extreme Gradient (XG) Boost classifier. Performance has been evaluated using pedestrian’s data from avenue and UBI-Fight datasets, where both have different environment. The mean achieved accuracies are 91.3% and 85.14% over the Avenue and UBI-Fight datasets, respectively. These results are more accurate as compared to other existing methods.

More than Shapes: The Silhouette Effect in Advertising

Even though the use of silhouettes in advertising is prevalent, knowledge about their impact on consumers is surprisingly limited. In this research, I propose that the use of human silhouettes, when compared to images of attractive human models, enhances advertising effectiveness by reducing the extent to which consumers experience self-threat when viewing an ad referent. In addition, I predict this effect holds only for appearance-related products and strengthens as consumers’ level of appearance-related self-esteem decreases. Six studies reported in this research provide converging evidence in support of these expectations.

A comprehensive study on codebook-based feature fusion for gait recognition

Vision-based person identification using gait is one of the important and challenging tasks in the fields of computer vision and machine learning. It has received significant research efforts in the past two decades due to its several benefits. It is non-invasive and can be performed at a distance without active collaboration from users. The identification can be performed from low-resolution videos using simple instrumentation. The conventional gait recognition approaches usually operate on the sequence of extracted human silhouettes. They derive several gait-related features from the segmented binary energy maps of the walker. However, such processes are sensitive to variations in the silhouette shapes, thus limiting their efficacy. Codebook-based feature encoding techniques have been proven to be effective and reported state-of-the-art recognition results on several visual datasets such as action recognition, image, and video classification, etc. The whole process usually follows the pipeline of pattern recognition which mainly consists of five steps: (i) local feature extraction, (ii) feature pre-processing, (iii) codebook computation, (iv) feature encoding, and (v) classification. Each step in the pipeline plays a crucial role in recognition accuracy. Since the visual gait sequences comprise different walking patterns of the subjects due to variations in their static appearance and motion dynamics, several features are extracted to encode this information. Finally, they are fused to recognize the identity of the subject. This paper presents a comprehensive study of codebook-based approaches, explains all the steps in the encoding of visual gait sequences, and uncovers some good practices to obtain state-of-the-art recognition results. In particular, we investigated two different local features to encode the static appearance and motion information of the walker, and twelve kinds of feature encoding methods. An extensive evaluation of these encoding methods is carried out on a large benchmark CASIA-B gait database and their performance comparison is presented.

Semantic segmentation-based system for fall detection and post-fall posture classification

Fall is one of the most critical issues faced by elders in their daily life. The consequences of falls range from fatal injuries, severe injuries to no injuries. Therefore, an effective system for detecting falls and treating post-fall injuries is important. Unlike wearable sensors, camera-based systems seem more comfortable and flexible for daily life monitoring. However, monitoring human activities in real settings using cameras is not only challenging but also pose privacy issues. To mitigate this problem, we propose a surveillance camera-based framework for fall detection and post-fall classification where the human silhouette is extracted and used instead of raw images. The human silhouette is obtained using a pixel-level classification based on a Multi-Scale Skip Connection Segmentation Network (MSSkip), which is shown to achieve state-of-the-art performance on the validation set from PASCAL VOC 2012 dataset for the class Person with an IoU of 90%. The temporal and spatial variations of the human poses are fed to a Convolutional Long Short Term Memory (ConvLSTM) network to detect whether or not a fall has occurred. The proposed fall detection method achieved an F1-score of 97.68% on the UP Fall dataset, and state-of-art performance on the UR Fall detection database. For post-fall posture classification, the Xception network is shown to achieve an F1-score of 97.85% on our customized post-fall dataset.

Toward Clothing Personalized Customization Multi-Perspective Silhouettes 3D Human Body Rapid Reconstruction

The clothing personalized online customization is required to design around the user shape parameters, and the traditional anthropometry method obtains the user shape parameter with a low speed and a high error. To tackle this issue, a rapid 3D human body reconstruction method is proposed based on multi-perspective silhouettes for clothing personalized customization. The multilevel dilated convolution semantic network (MDS-Net) is leveraged to extract the global and local features in the human silhouettes to implement the semantic segmentation. The torso parameter extraction network (TPE-Net) is leveraged to extract the shape and pose parameters of the multi-perspective human body segmentation maps. The principal component analysis (PCA) is leveraged to extract the semantic features in the latent space of the 3D human body model, and the semantic features are mapped to the shape and posture parameters output by the torso parameter extraction network, thus, 3D human body model is reconstrued. This method was verified in a PyTorch environment with four datasets. The experiments demonstrated that the mIoU of the multilevel dilated convolution semantic network on the test set is 0.881, which can achieve overall segmentation and local detail preservation. The torso parameter extraction network has an accuracy of 0.74 in the shape parameter prediction on the test set, the prediction of joint offset distance is proportional to the index in the kinematic tree. The entire 3D human body reconstruction method is verified by the real cases.

MiShape

We propose MiShape, a millimeter-wave (mmWave) wireless signal based imaging system that generates high-resolution human silhouettes and predicts 3D locations of body joints. The system can capture human motions in real-time under low light and low-visibility conditions. Unlike existing vision-based motion capture systems, MiShape is privacy non-invasive and can generalize to a wide range of motion tracking applications at-home. To overcome the challenges with low-resolution, specularity, and aliasing in images from Commercial-Off-The-Shelf (COTS) mmWave systems, MiShape designs deep learning models based on conditional Generative Adversarial Networks and incorporates the rules of human biomechanics. We have customized MiShape for gait monitoring, but the model is well adaptive to any tracking applications with limited fine-tuning samples. We experimentally evaluate MiShape with real data collected from a COTS mmWave system for 10 volunteers, with diverse ages, gender, height, and somatotype, performing different poses. Our experimental results demonstrate that MiShape delivers high-resolution silhouettes and accurate body poses on par with an existing vision-based system, and unlocks the potential of mmWave systems, such as 5G home wireless routers, for privacy-noninvasive healthcare applications.

Gerçek Zamanlı İnsan Davranışı Anlamaya Doğru: Optimal-Altı Bir Şekil Tanımlayıcı

In this study, two novel shape descriptors are proposed to be used in human behavior understanding problem. First is optimal shape descriptor, which has high performance but works very slow due to high algorithmic complexity. Second is suboptimal shape descriptor, performance of which is very close to optimal one, but works much more faster. Optimal means using minimum data to represent maximum knowledge. Algorithms are run on Weizmann dataset and results are shown both as figure and video link. Classification was performed using 12 statistical features extracted from the data sets' human silhouettes. An accuracy rating of 92 percent was obtained by using Euclidean distance in classification.

Dance Art Scene Classification Based on Convolutional Neural Networks

Digital multimedia resources have become an important part of people’s daily cultural life. Automatic scene classification of a large number of dance art videos is the basis for scene semantic based video content retrieval. In order to improve the accuracy of scene classification, the videos are identified using a deep convolutional neural network based on differential evolution for dance art videos. First, the Canny operator is used in YCbCr colour space to detect the human silhouette in the key frames of the video. Then, the AdaBoost algorithm based on cascade structure is used to implement human target tracking and labelling, and the construction and updating of weak classifiers are analysed. Next, a differential evolution algorithm is used to optimise the structural parameters of the convolutional neural network, and an adaptive strategy is adopted for the scaling factor of the differential evolution algorithm to improve the optimisation solution accuracy. Finally, the improved deep convolutional neural network is used to train the classification of the labelled videos in order to obtain stable scene classification results. The experimental results show that by reasonably setting the crossover rate of differential evolution and the convolutional kernel size of the convolutional neural network, high scene classification performance can be obtained. The high accuracy and low root-mean-square error validate the applicability of the proposed method in dance art scene classification.

PoseMapGait: A model-based gait recognition method with pose estimation maps and graph convolutional networks

Gait recognition is a particularly effective way to avoid the spread of COVID-19 while people are under surveillance. Because of its advantages of non-contact and long-distance identification. One category of gait recognition methods is appearance-based, which usually extracts human silhouettes as the initial input feature and achieves high recognition rates. However, the silhouette-based feature is easily affected by the view, clothing, bag, and other external variations. Another category is based on model-based, one popular model-based feature is extracted from human skeletons. The skeleton-based feature is robust to many variations because it is less sensitive to human shape. However, the performance of skeleton-based methods suffers from recognition accuracy loss due to limited input information. In this paper, instead of relying on coordinates from skeletons, we exploit that pose estimation maps, the byproduct of pose estimation. It not only preserves richer cues of the human body compared with the skeleton-based feature, but also keeps the advantage of being less sensitive to human shape compared with the silhouette-based feature. Specifically, the evolution of pose estimation maps is decomposed as one heatmaps evolution feature (extracted by gaitMap-CNN) and one pose evolution feature (extracted by gaitPose-GCN), which denote the invariant features of whole body structure and body pose joints for gait recognition, respectively. Our method is evaluated on two large datasets, CASIA-B and the CMU Motion of Body (MoBo) dataset. The proposed method achieves the new state-of-the-art performance compared with recent advanced model-based methods.

Extrinsic Behavior Prediction of Pedestrians via Maximum Entropy Markov Model and Graph-Based Features Mining

With the change of technology and innovation of the current era, retrieving data and data processing becomes a more challenging task for researchers. In particular, several types of sensors and cameras are used to collect multimedia data from various resources and domains, which have been used in different domains and platforms to analyze things such as educational and communicational setups, emergency services, and surveillance systems. In this paper, we propose a robust method to predict human behavior from indoor and outdoor crowd environments. While taking the crowd-based data as input, some preprocessing steps for noise reduction are performed. Then, human silhouettes are extracted that eventually help in the identification of human beings. After that, crowd analysis and crowd clustering are applied for more accurate and clear predictions. This step is followed by features extraction in which the deep flow, force interaction matrix and force flow features are extracted. Moreover, we applied the graph mining technique for data optimization, while the maximum entropy Markov model is applied for classification and predictions. The evaluation of the proposed system showed 87% of mean accuracy and 13% of error rate for the avenue dataset, while 89.50% of mean accuracy rate and 10.50% of error rate for the University of Minnesota (UMN) dataset. In addition, it showed a 90.50 mean accuracy rate and 9.50% of error rate for the A Day on Campus (ADOC) dataset. Therefore, these results showed a better accuracy rate and low error rate compared to state-of-the-art methods.

A Graph-Based Approach to Recognizing Complex Human Object Interactions in Sequential Data

The critical task of recognizing human–object interactions (HOI) finds its application in the domains of surveillance, security, healthcare, assisted living, rehabilitation, sports, and online learning. This has led to the development of various HOI recognition systems in the recent past. Thus, the purpose of this study is to develop a novel graph-based solution for this purpose. In particular, the proposed system takes sequential data as input and recognizes the HOI interaction being performed in it. That is, first of all, the system pre-processes the input data by adjusting the contrast and smoothing the incoming image frames. Then, it locates the human and object through image segmentation. Based on this, 12 key body parts are identified from the extracted human silhouette through a graph-based image skeletonization technique called image foresting transform (IFT). Then, three types of features are extracted: full-body feature, point-based features, and scene features. The next step involves optimizing the different features using isometric mapping (ISOMAP). Lastly, the optimized feature vector is fed to a graph convolution network (GCN) which performs the HOI classification. The performance of the proposed system was validated using three benchmark datasets, namely, Olympic Sports, MSR Daily Activity 3D, and D3D-HOI. The results showed that this model outperforms the existing state-of-the-art models by achieving a mean accuracy of 94.1% with the Olympic Sports, 93.2% with the MSR Daily Activity 3D, and 89.6% with the D3D-HOI datasets.