Gait Energy Image Research Articles

A novel lameness detection method for dairy cows based on temporal gait and spatial post features

Lameness is one of the common diseases in dairy farms, which seriously affects the health and welfare of dairy cows. The existing lameness detection methods based on computer vision have poor ability to extract spatiotemporal features, which reduces the lameness detection accuracy in dairy cows. In this paper, a spatiotemporal energy network (SEN) is proposed to detect lameness in dairy cows. It compresses the cow walking video into history energy image (HEI) and gait energy image (GEI), and then accurately extracts the temporal gait and spatial pose features of dairy cows. First, the YOLOv8n algorithm is used to remove background information from the video and segment the cow’s body. Second, the body contour sequence of the dairy cow is stacked based on the time series, and then different body parts of dairy cows are stacked on the channel to form the HEI. The HEI preserves the movement trajectory of the dairy cows. The first lameness classification model is obtained by training the MobileNetv2 network with HEI dataset. Third, the body contour sequence of the dairy cow is cropped according to the center of mass and body proportion and then stacked into an image, which is named GEI. The second lameness classification model is obtained by training the MobileNetv2 network with GEI dataset. Finally, the prediction results of the two models are fused according to the weight ratio to detect lameness in dairy cows. To verify the lameness detection effect of this method, 264 samples were randomly selected from 330 samples as the training set and the remaining 66 samples as the test set. The proposed method obtained competitive lameness detection results with classification accuracy of 96.22 %, sensitivity of 96.35 % and specificity of 98.14 %. By combining HEI and GEI, the spatiotemporal features of the cow walking are extracted more comprehensively. This method provides guidance for the lameness detection of dairy cows and reference for the gait analysis of animals.

Different gait combinations based on multi-modal deep CNN architectures

Gait recognition is the process of identifying a person from a distance based on their walking patterns. However, the recognition rate drops significantly under cross-view angle and appearance-based variations. In this study, the effectiveness of the most well-known gait representations in solving this problem is investigated based on deep learning. For this purpose, a comprehensive performance evaluation is performed by combining different modalities, including silhouettes, optical flows, and concatenated image of the Gait Energy Image (GEI) head and leg region, with GEI itself. This evaluation is carried out across different multimodal deep convolutional neural network (CNN) architectures, namely fine-tuned EfficientNet-B0, MobileNet-V1, and ConvNeXt-base models. These models are trained separately on GEIs, silhouettes, optical flows, and concatenated image of GEI head and leg regions, and then extracted GEI features are fused in pairs with other extracted modality features to find the most effective gait combination. Experimental results on the two different datasets CASIA-B and Outdoor-Gait show that the concatenated image of GEI head and leg regions significantly increased the recognition rate of the networks compared to other modalities. Moreover, this modality demonstrates greater robustness under varied carrying (BG) and clothing (CL) conditions compared to optical flows (OF) and silhouettes (SF). Codes available at https://github.com/busrakckugurlu/Different-gait-combinations-based-on-multi-modal-deep-CNN-architectures.git

GAIT analysis based on GENDER detection using pre-trained models and tune parameters

In past several decades, gait biometrics has emerged as a viable alternative to traditional identification methods, offering advancements in surveillance, monitoring, and analysis techniques. However, determining gender based on gait remains a challenge, particularly in computer vision applications. This study proposes a robust and adaptable approach to address this issue by leveraging gait analysis. There is a growing need for datasets tailored to gait analysis and recognition to facilitate the extraction of relevant data. While most existing research relies on image-based gait datasets, this study utilizes the OULP-Age dataset from OU-ISIR, representing gait through gait energy images (GEIs). The methodology involves feature extraction from GEIs using pre-trained models, followed by classification with the XGBoost classifier. Gender prediction is enhanced through parameter fine-tuning of the XGBoost classifier. Comparative analysis of 11 pre-trained models for feature extraction reveals that DenseNet models, combined with optimized XGBoost parameters, demonstrate promising results for gender prediction. This study contributes to advancing gender prediction based on gait analysis and underscores the efficacy of integrating deep learning models with traditional classifiers for improved accuracy and reliability.

Deep multi-convolutional stacked capsule network fostered human gait recognition from enhanced gait energy image

Deep multi-convolutional stacked capsule network fostered human gait recognition from enhanced gait energy image

Gender Recognition Based on Gradual and Ensemble Learning from Multi-View Gait Energy Images and Poses.

Image-based gender classification is very useful in many applications, such as intelligent surveillance, micromarketing, etc. One common approach is to adopt a machine learning algorithm to recognize the gender class of the captured subject based on spatio-temporal gait features extracted from the image. The image input can be generated from the video of the walking cycle, e.g., gait energy image (GEI). Recognition accuracy depends on the similarity of intra-class GEIs, as well as the dissimilarity of inter-class GEIs. However, we observe that, at some viewing angles, the GEIs of both gender classes are very similar. Moreover, the GEI does not exhibit a clear appearance of posture. We postulate that distinctive postures of the walking cycle can provide additional and valuable information for gender classification. This paper proposes a gender classification framework that exploits multiple inputs of the GEI and the characteristic poses of the walking cycle. The proposed framework is a cascade network that is capable of gradually learning the gait features from images acquired in multiple views. The cascade network contains a feature extractor and gender classifier. The multi-stream feature extractor network is trained to extract features from the multiple input images. Features are then fed to the classifier network, which is trained with ensemble learning. We evaluate and compare the performance of our proposed framework with state-of-the-art gait-based gender classification methods on benchmark datasets. The proposed framework outperforms other methods that only utilize a single input of the GEI or pose.

Machine learning models and methods for human gait recognition

The paper explores the challenge of human identification through gait recognition within biometric identification systems. Itoutlines the essential criteria for human biometric features, discusses primary biometric characteristics, and their application in biometric identification systems. The paper also examines the feasibility of utilizing gait as a biometric identifier, emphasizing its advantages, such as not requiring the upfront provision of personal biometric information and specialized equipment.The authors conduct an analysis of existing scientific literature in the field of gait recognition, categorizing gait recognition methodsinto template-based and non-template-based approaches. Throughout their research, they identify the key issues and challenges that researchers face in this domain, along with the prevailing trends in human gait recognition within biometric identification systems.Additionally, the paper introduces a method for person identification based on gait, utilizing the Histogram of Oriented Gradients and the Sum Variance Haralick texture features. It involves transforming input video into a series of images depicting the gait silhouette, creating a Gait Energy Image (GEI) by combining these gait silhouettes throughout a gait cycle, and translating the GEI into the Gait Gradient Magnitude Image (GGMI). The subsequent step involves extracting recommended gait characteristics from the GGMIs of participants included in a dataset.To preprocess the collected characteristics, Principal Component Analysis (PCA) is applied, reducing the dimensions that may negatively impact classification robustness, thereby enhancing overall performance. In the final step, a K-Nearest Neighbors (KNN) classifier is employed to categorize the characteristics obtained from a specific dataset.The proposed novel feature vector in the paper demonstrates increased reliability and effectively captures spatial variations in gait patterns. Notably, it reduces the dimensionality of the feature vector from 3780×1 to 63×1, resulting in decreased computational complexity in the gait recognition system. Experimental evaluations on the CASIA A and CASIA B datasets reveal that the proposed approach outperforms other HOG-based methods in most scenarios, with the exception of situations involving frontal images.

Robust Gait Recognition Based on Deep CNNs With Camera and Radar Sensor Fusion

In recent years, gait recognition has emerged as an important and promising solution for human identification. Generally, gait recognition is based on a single type of sensor, such as a camera or a radar. However, data of a single modality may only capture inadequate gait features of a person, such as camera data lacking the intuitive micro-motion pattern information and radar data lacking the information about gait appearance, making gait-based human identification system vulnerable to complex covariate conditions, e.g. cross-view and cross-walking-condition. To build a robust and reliable gait-based human identification system, in this study, we propose a multi-sensor gait recognition framework with deep convolutional neural networks (CNNs) by fusing camera gait energy images (GEIs) and radar time-Doppler spectrograms. To learn the fine-grained gait appearance features, we propose a body-part spatial attention (BPSA) module to obtain more discriminative body part representations of GEIs. To learn the gait micro-motion pattern, we propose a long-short temporal relation modeling (LSTRM) module to obtain the local and global micro-motion representation of time-Doppler spectrograms. Finally, we fuse the discriminative body part representation and the micro-motion pattern at the multi-scale feature space to obtain richer and more robust gait features for human identification. We provide an extensive empirical evaluation in terms of various complex covariate conditions, namely, cross-view and cross-walking-condition. Experiments on 121 subjects with eight views and three walking conditions of camera and radar data show our proposed method is more robust and accurate.

A multi-view human gait recognition using hybrid whale and gray wolf optimization algorithm with a random forest classifier

Gait recognition has become one of the furthest promising behavioral biometric techniques for identifying individuals. Gait recognition models are capable of identifying humans at a distance based on their walking manner without their permission or interference. However, it is usually noted that the performance of a gait recognition approach will drop drastically in the presence of covariates such as carrying conditions, clothing conditions, and variations in the view angle. Therefore, it is necessary to develop a robust gait recognition system in order to identify the most significant gait features. In this paper, we introduced a recently developed hybrid whale and gray wolf optimization algorithm (WGWOA) for determining the optimal subset of gait features by combining the advantages of whale optimization and gray wolf optimization techniques. Moreover, we employed principal component analysis (PCA) to extract the essential gait features from the gradient gait energy image (GGEI) and random forest (RF) approach to classify the optimal gait features. The proposed method has been assessed on the publicly available largest multi-view CASIA-B and OU-MVLP benchmark datasets. Experimental results indicate that the proposed model achieved an accuracy of 99.25%, 98.39%, and 97.97% under normal, carrying a bag and wearing coat walking conditions, respectively on the CASIA-B dataset. Furthermore, the proposed model achieved an accuracy of 97.63% under normal conditions on the OU-MVLP dataset. The comparative results also confirm that the proposed algorithm is superior to the contemporary approaches.

Gait-CNN-ViT: Multi-Model Gait Recognition with Convolutional Neural Networks and Vision Transformer.

Gait recognition, the task of identifying an individual based on their unique walking style, can be difficult because walking styles can be influenced by external factors such as clothing, viewing angle, and carrying conditions. To address these challenges, this paper proposes a multi-model gait recognition system that integrates Convolutional Neural Networks (CNNs) and Vision Transformer. The first step in the process is to obtain a gait energy image, which is achieved by applying an averaging technique to a gait cycle. The gait energy image is then fed into three different models, DenseNet-201, VGG-16, and a Vision Transformer. These models are pre-trained and fine-tuned to encode the salient gait features that are specific to an individual's walking style. Each model provides prediction scores for the classes based on the encoded features, and these scores are then summed and averaged to produce the final class label. The performance of this multi-model gait recognition system was evaluated on three datasets, CASIA-B, OU-ISIR dataset D, and OU-ISIR Large Population dataset. The experimental results showed substantial improvement compared to existing methods on all three datasets. The integration of CNNs and ViT allows the system to learn both the pre-defined and distinct features, providing a robust solution for gait recognition even under the influence of covariates.

A Deep-Learning Approach for Identifying a Drunk Person Using Gait Recognition

Various accidents caused by alcohol consumption have recently increased in prevalence and have become a huge social problem. There have been efforts to identify drunk individuals using mobile devices; however, it is difficult to apply this method to a large number of people. A promising approach that does not involve wearing any sensors or subject cooperation is a markerless, vision-based method that only requires a camera to classify a drunk gait. Herein, we first propose a markerless, vision-based method to determine whether a human is drunk or not based on his or her gait pattern. We employed a convolutional neural network to analyze gait patterns with image augmentation depending on gait energy images. Gait images captured through a camera allow a complex neural network to detect the human body shape accurately. It is necessary for removing the background behind human shape in the gait image because it disrupts the detection algorithm. A process of conversion into gait energy images and augmenting image data is then applied to the dataset of the gait images. A total of 20 participants participated in the experiment. They were required to walk along a line both with and without wearing the Drunk Busters Goggles, which were intended to collect sober and drunk gait images. Validation accuracy for the recognition of a drunk state in 20 persons was approximately 74.94% under optimal conditions. If the present approach fulfills its promise, we can prevent safety accidents due to alcohol, thus decreasing its burden on industries and society.

Gait Recognition Using Convolutional Neural Network

Biometrics are the body measurements and calculations related to individuals. Biometrics validation is used as a form of identification of individual. Gait recognition system is one of the most advanced technology that people have been working on for a while now that takes center stage in the field of biometrics. Compared to the other types of existing systems of biometric recognition such as fingerprint detection, iris-scanning systems etc., Gait Recognition system ensures no human intervention. This paper focuses on recognition based on a person’s gait. Every person has a distinct gait pattern that is unique to every other person. To train the model CASIA-B dataset has been used. The dataset includes 124 subjects where each sample has undergone Gait Energy Image extraction. Samples with clothing and baggage have been included which changes the silhouette of the person. Therefore, the model has been trained for a wider application where people wear different type of clothing and carry-ons. A Convolutional Neural Network consisting of 8 layers has been trained which performs well on both samples of dataset and an accuracy of 95.45% was obtained on dataset not involving layers of clothing and accuracy of 91.80% was obtained for the sample with clothing and baggage.

VCOACH: A Virtual Coaching System Based on Visual Streaming

A virtual coaching system is key for monitoring the performance during exercise regimes and rendering timely feedback to avoid potential physical harm. Ideally, the core technologies behind such a system imply exercise recognition/assessment for performance evaluation and subject identification for retrieval/update of personal profiles. In this work, we develop techniques based on learning models for inferences regarding exercise recognition, prediction of the subject's biometrics, exercise evaluation, and simultaneous recognition and evaluation. We address four indoor exercises: squats, push-ups, shoulder-press, and lunges. Different sorts of exercise evaluation are provided: (1) the overall-binary assessment, with a detailed assessment rendering the exact mistakes performed, and (2) the score-based evaluation giving a metric in the range of 0-10 for each aspect of the exercise. Thermal and RGB imagery are examined with various descriptors, such as raw images, Gait Energy Image (GEI), and skeletal pose, with a comparison between their performance. The GEI descriptor extracted from RGB imagery is introduced for exercise recognition and evaluation tasks achieving remarkable results. The GEI descriptor is further utilized for subject recognition and verification from the performed exercise showing the ability of subject discrimination from the way of exercising. Consequently, we introduce new kinds of biometrics based on indoor exercising inspired by the notion that individuals are recognizable from the fingerprint motion dynamics in exercise regimes. To the best of our knowledge, our work is the first to use GEI in such directions and show its remarkable effectiveness.

MODELS AND METHODS FOR GAIT-BASED PERSON IDENTIFICATION WHILE WEARING DIFFERENT OUTFIT

Abstract. This study presents a potential methodology for the identification of individuals based on their gait patterns, even when they are wearing different attire. The approach described in this study utilizes modified local patterns that have been specifically designed to be robust to rotation. These patterns enable the extraction of more discerning characteristics, particularly in scenarios where individuals may be wearing diverse clothing. The procedure was outlined in a comprehensive manner, including explanations for each step of the process. The experiments provided confirmation of the notable efficiency of gait-based person identification in scenarios where individuals wear varying attire. Keywords: Surveillance systems, Human identification, Silhouette extraction, Gait Energy Image, classification, Local Binary Pattern, Gait recognition.

MODELS AND METHODS FOR GAIT-BASED PERSON IDENTIFICATION WHILE WEARING DIFFERENT OUTFIT

Abstract. This study presents a potential methodology for the identification of individuals based on their gait patterns, even when they are wearing different attire. The approach described in this study utilizes modified local patterns that have been specifically designed to be robust to rotation. These patterns enable the extraction of more discerning characteristics, particularly in scenarios where individuals may be wearing diverse clothing. The procedure was outlined in a comprehensive manner, including explanations for each step of the process. The experiments provided confirmation of the notable efficiency of gait-based person identification in scenarios where individuals wear varying attire. Keywords: Surveillance systems, Human identification, Silhouette extraction, Gait Energy Image, classification, Local Binary Pattern, Gait recognition.

Batch Hard Contrastive Loss and Its Application to Cross-View Gait Recognition

Biometric person authentication comprises two tasks: the identification task (i.e., one-to-many matching) and the verification task (i.e., one-to-one matching). In this paper, we propose a loss function called batch hard contrastive loss (BHCn) for the deep learning-based verification task. For this purpose, we consider batch mining techniques developed in the identification task and translate them to the verification task. More specifically, inspired by batch mining triplet losses to learn a relative distance for the identification task, we propose BHCn to learn an absolute distance that better represents verification in general. Our method preserves the identity-agnostic nature of the contrastive loss by selecting the hardest pair of samples for each pair of identities in a batch instead of selecting the hardest pair for each sample. We validate the effectiveness of the proposed method in cross-view gait recognition using three networks: a lightweight input, structure, and output network we call GEI + CNN (Gait Energy Image Convolutional Neural Network) as well as the widely used GaitSet and GaitGL, which have sophisticated inputs, structures, and outputs. We trained these networks with the publicly available silhouette-based datasets, the OU-ISIR Gait Database Multi-View Large Population (OU-MVLP) dataset and the Institute of Automation Chinese Academy of Sciences Gait Database Multiview (CASIA-B) dataset. Experimental results show that the proposed BHCn outperforms other loss functions, such as a triplet loss with batch mining as well as the conventional contrastive loss.

Attention‐aware spatio‐temporal learning for multi‐view gait‐based age estimation and gender classification

AbstractRecently, gait‐based age and gender recognition have attracted considerable attention in the fields of advertisement marketing and surveillance retrieval due to the unique advantage that gaits can be perceived at a long distance. Intuitively, age and gender can be recognised by observing people's static shape (e.g. different hairstyles between males and females) and dynamic motion (e.g. different walking velocities between the elderly and youth). However, most of the existing gait‐based age and gender recognition methods are based on Gait Energy Image (GEI), which loses the capability of explicitly modelling temporal dynamic information and is not robust to the multi‐view recognition that inevitably happens in a real application. Therefore, in this study, an Attention‐aware Spatio‐Temporal Learning (ASTL) framework is proposed, which employs a silhouette sequence as input to learn essential and invariable spatial‐temporal gait representations. More specifically, a Multi‐Scale Temporal Aggregation (MSTA) module provides an effective scheme for dynamic gait description by exploring and aggregating multi‐scale temporal interval information, which is a core supplement to spatial representation. Then, a Multiple Attention Aggregation (MAA) module is designed to help the network focus on the most discriminatory information along temporal, spatial and channel dimensions. Finally, a Multimodal Collaborative Learning (MCL) block gives full play to the advantages of different modal features through a multimodal cooperative learning strategy. The mean absolute error (MAE) for the age estimation and the correct classification rate (CCR) for the gender classification on OU‐MVLP achieve 6.68 years and 97%, respectively, demonstrating the superiority of the method. Ablation experiments and visualisation results also prove the effectiveness of the three individual modules in their framework.

A Multi-Modal Gait Analysis-Based Detection System of the Risk of Depression.

Currently, depression has become a common mental disorder, especially among postgraduates. It is reported that postgraduates have a higher risk of depression than the general public, and they are more sensitive to contact with others. Thus, a non-contact and effective method for detecting people at risk of depression becomes an urgent demand. In order to make the recognition of depression more reliable and convenient, we propose a multi-modal gait analysis-based depression detection method that combines skeleton modality and silhouette modality. Firstly, we propose a skeleton feature set to describe depression and train a Long Short-Term Memory (LSTM) model to conduct sequence strategy. Secondly, we generate Gait Energy Image (GEI) as silhouette features from RGB videos, and design two Convolutional Neural Network (CNN) models with a new loss function to extract silhouette features from front and side perspectives. Then, we construct a multi-modal fusion model consisting of fusing silhouettes from the front and side views at the feature level and the classification results of different modalities at the decision level. The proposed multi-modal model achieved accuracy at 85.45% in the dataset consisting of 200 postgraduate students (including 86 depressive ones), 5.17% higher than the best single-mode model. The multi-modal method also shows improved generalization by reducing the gender differences. Furthermore, we design a vivid 3D visualization of the gait skeletons, and our results imply that gait is a potent biometric for depression detection.

Gait-ViT: Gait Recognition with Vision Transformer.

Identifying an individual based on their physical/behavioral characteristics is known as biometric recognition. Gait is one of the most reliable biometrics due to its advantages, such as being perceivable at a long distance and difficult to replicate. The existing works mostly leverage Convolutional Neural Networks for gait recognition. The Convolutional Neural Networks perform well in image recognition tasks; however, they lack the attention mechanism to emphasize more on the significant regions of the image. The attention mechanism encodes information in the image patches, which facilitates the model to learn the substantial features in the specific regions. In light of this, this work employs the Vision Transformer (ViT) with an attention mechanism for gait recognition, referred to as Gait-ViT. In the proposed Gait-ViT, the gait energy image is first obtained by averaging the series of images over the gait cycle. The images are then split into patches and transformed into sequences by flattening and patch embedding. Position embedding, along with patch embedding, are applied on the sequence of patches to restore the positional information of the patches. Subsequently, the sequence of vectors is fed to the Transformer encoder to produce the final gait representation. As for the classification, the first element of the sequence is sent to the multi-layer perceptron to predict the class label. The proposed method obtained 99.93% on CASIA-B, 100% on OU-ISIR D and 99.51% on OU-LP, which exhibit the ability of the Vision Transformer model to outperform the state-of-the-art methods.

Robust and Lightweight System for Gait-based Age Estimation towards Viewing Angle Variations

Abstract: Gait based age recognition is a very challenging task, as it involves multiple hurdles such as change in viewpoint of the person. The proposed system handles this problem by performing a sequence of tasks such as GEI formation from silhouette, applying DCT on GEI and extracting the features and finally using MLP for age estimation. The proposed system proves its effectiveness comparing the performance with state of art methods -conventional methods and deep learning based methods. The performance of the system is estimated on OU-MVLP and OULP-Age datasets. The experimental results show the robustness of the system against viewing angle variations. Background: In computer vision applications, gait-based age estimation across several cameras is critical, especially when following the same person in various viewpoints. Objective: To implement the system which adopts lightweight approach for gait-based age estimation. Method: The proposed system uses a combination of the discrete cosine transform (DCT) and multi-layer perceptron (MLP) on gait energy image (GEI) to perform age estimation. Result: The performance of the system is extensively evaluated on the OU-MVLP and OULP-Age datasets. Conclusion: The proposed system attains best mean absolute error (MAE) of 5.05 (in years) for the OU-MVLP dataset, and 5.65 for the OULP dataset.

A Deep Learning Approach for Biometric Security in Video Surveillance System Using Gait

Video surveillance systems and biometrics inclusion play a significant part in various applications like a criminal investigation, medical rehabilitation, virtual reality, etc. Human Gait is a popular biometric where the individual is differentiated by unique limb actions and special ground reaction force. The unique movement of limb actions is called gait, and a record of 2-D aggregate floor response force through one walking cycle is called Cumulative Foot Pressure Images (CFPI). Both gait and cumulative foot pressure images can be acquired simultaneously of the same person during walking under a surveillance system for human identification. Accurate gait recognition is highly impactful for most applications and a major challenge for researchers due to various external factors like different shoes, mood, clothes, injuries etc., affects the individual gait. The novel system addresses the accuracy issue and proposes two models using the Deep Convolution Neural Network (DCNN) architecture on a large standard database, CASIA-D, containing gait pose and CFPI images of the same person. First, the model of the DCNN is trained using unique Gait Energy Image (GEI) features which reduce the computational time compared to other types of feature sets. The second model is prepared using the CFPI features. Experimentation has been carried out to evaluate the performance of these models with different optimization methods and activation functions and has proven that the DCNN model is far superior in building an accurate gait recognition system on large standard datasets.