Gait Recognition Tasks Research Articles

Gait recognition using deep learning with handling defective data from multiple wearable sensors

Gait recognition based on multiple wearable sensors has received widespread attention in recent years. Collecting data from various types of multimodal sensors allows for a more comprehensive capture of gait information. This effectively enhances gait recognition performance. However, the increase in the number of sensors also leads to the possibility of generating defective data during the process of data collection and processing, particularly data redundancy and data anomaly. Traditional solutions often involve complex data analysis and selection, which may result in important information being filtered out and high computational demands. To address these issues, this paper proposes a new method that combines the powerful feature extraction capabilities of convolutional neural networks. To address the issue of data redundancy, the Channel Multiscale-aware (CMSA) Module is designed. It extracts information with different receptive fields from different channels of the same feature map to directly reduce similarities in information. To address the issue of data anomaly, the Feature Combination based on Local Channel Attention (FCLCA) Module is designed to select channel segments least affected by data anomaly from the feature map. Attention mechanisms are incorporated into both the CMSA and FCLCA modules to facilitate the model in adapting and learning more crucial features. Additionally, improvements are made to their pooling structures. After numerous experiments, it was demonstrated that our model achieved superior performance in gait recognition tasks.

Gait‐based human recognition based on millimetre wave multiple input multiple output radar point cloud constructed using velocity‐depth‐time

AbstractGait recognition is to recognise different individuals based on their faint differences of gait characteristics, which is different from and more challengeable than the recognition of human activities based on relatively bigger differences between different motions. Existing millimetre‐wave Multiple Input Multiple Output radar point cloud data contains time‐varying three‐dimensional spatial positions, velocity, and intensity information. How to enhance the accuracy of gait recognition by effectively utilising the available radar point cloud data has become an attractive research topic in recent years. A velocity‐depth‐time (VDT) based point cloud construction method for millimetre‐wave Multiple Input Multiple Output radar is proposed for gait recognition application, which can not only alleviate the sparsity problem of mmWave point cloud but also make the constructed point cloud to exhibit temporal structural features of micro‐motions, and therefore enable the successful application of PointNet++ to mmWave‐MIMO point cloud gait recognition. New point clouds are constructed by the proposed method using public gait recognition datasets of 10 and 20 individuals from mmWave‐MIMO radar, which are used to conduct gait recognition experiments using PointNet++. The results show that the point clouds constructed based on VDT are more conducive to the gait recognition task. Even using the classic PointNet++ model, which is not specially designed for radar point clouds, high recognition accuracy can be achieved for gait recognition tasks. The recognition accuracies are improved by 11% and 12% in this work for datasets of 10 and 20 individuals, respectively, compared with the 84% and 80% achieved by the traditional method using the same dataset and the same PointNet++ model, while the accuracies are improved by 5% and 12%, respectively, compared with the 90% and 80% achieved by the original dataset thesis method corresponding to 10‐individual and 20‐individual datasets.

Unsupervised Domain Adaptation for Disguised-Gait-Based Person Identification on Micro-Doppler Signatures

In recent years, gait-based person identification has gained significant interest for a variety of applications, including security systems and public security forensics. Meanwhile, this task is faced with the challenge of disguised gaits. When a human subject changes what he or she is wearing or carrying, it becomes challenging to reliably identify the subject’s identity using gait data. In this paper, we propose an unsupervised domain adaptation (UDA) model, named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Guided Subspace Alignment under the Class-aware</i> condition (G-SAC), to recognize human subjects based on their disguised gait data by fully exploiting the intrinsic information in gait biometrics. To accomplish this, we employ neighbourhood component analysis (NCA) to create an intrinsic feature subspace from which we can obtain similarities between normal and disguised gaits. With the aid of a proposed constraint for adaptive class-aware alignment, the class-level discriminative feature representation can be learned guided by this subspace. Our experimental results on a measured micro-Doppler radar dataset demonstrate the effectiveness of our approach. The comparison results with several state-of-the-art methods indicate that our work provides a promising domain adaptation solution for the concerned problem, even in cases where the disguised pattern differs significantly from the normal gaits. Additionally, we extend our approach to more complex multi-target domain adaptation (MTDA) challenge and video-based gait recognition tasks, the superior results demonstrate that the proposed model has a great deal of potential for tackling increasingly difficult problems.

Autoencoder-guided GAN for minority-class cloth-changing gait data generation

In the gait recognition task, cloth-changing data (e.g., CL-class data in the CASIA-B dataset) account for a very low proportion of the gait dataset, which brings serious recognition accuracy degradation to gait recognition algorithm in clothes changing scenarios. One way to solve this problem is to use data generation algorithms to expand the cloth-changing data. When facing the generation of minority-class data, the existing GAN-based image data expansion algorithms require a large amount of data to ensure the convergence, while mode collapse can occur due to the conflict between the true-false discriminant loss and the classification loss. In order to solve the above problems, we propose an autoencoder-guided GAN for cloth-changing gait data generation, which uses the autoencoder with embedding layer to extract the data features of the entire data set, rather than only extracting the features of the required CL data. Then the labeled latent vector (representing the extracted data features) in the autoencoder is taken as the input of the GAN, solving the problem that GAN convergence requires a lot of training data. The GAN has the same topology as the autoencoder and uses the parameter matrix of the autoencoder as the initial weights. Secondly, the true classification in GAN is canceled, and only the false classification is retained, which solves the problem of the collision of the loss function between the true-false discrimination and the classification discrimination. In the cloth-changing data generation experiments, compared with other GAN data expansion algorithms, our method generates clothes changing data with high quality, and ensures the diversity of the generated data. When using the CASIA-B dataset expanded by our algorithm, several gait recognition algorithms all show significant improvement, especially in the clothes changing scenario.

Symmetry-Driven hyper feature GCN for skeleton-based gait recognition

Gait recognition, as an attractive task in biometrics, remains challenging due to significant intra-class changes of clothing and pose variations across different cameras. Recent approaches mainly focus on silhouette-based gait mode, which is easy to model in Convolutional Neural Networks (CNNs). Compared with silhouettes, the dynamics of skeletons essentially convey more robust information, which is invariant to view and clothing changes. Conventional approaches for modeling skeletons usually rely on hand-crafted features or traversal rules, thus resulting in limited expressive power and difficulties of generalization. In this work, we address the skeleton-based gait recognition task with a novel Symmetry-Driven Hyper Feature Graph Convolutional Network (SDHF-GCN), which goes beyond the limitations of previous approaches by automatically learning multiple dynamic patterns and hierarchical semantic features in a unified Graph Convolutional Network (GCN). This model involves three dynamic patterns: natural connection, temporal correlation and symmetric interaction, which enriches the description of dynamic patterns by exploiting symmetry perceptual principles. Furthermore, a hyper feature network is proposed to aggregate the hierarchical semantic features, including dynamic features at the high level, structured features at the intermediate level, and static features at the low level, which complement each other to enhance the discriminative ability. By integrating different patterns in the hierarchical structure, the model is able to generate versatile and discriminative representations, thus improving the recognition rate. On the CASIA-B and OUMVLP-Pose datasets, the proposed SDHF-GCN renders substantial improvements over mainstream methods, especially in the coat-wearing scenario, with superior robustness to covariate factors.

Path-Independent Device-Free Gait Recognition Using mmWave Signals

Device-free gait recognition is a promising technique which could provide insensible identification for smart applications. It leverages the unique influence of the target's gait on surrounding wireless signals to achieve identity recognition in a contact-free and device-free manner. Existing device-free gait recognition methods could achieve good accuracy when a target walks along a predetermined path. However, the accuracy will drop dramatically when a target walks along an arbitrary path. This is due to the fact that different paths will exert different influence on the wireless signals. In order to address this issue, we develop a path-independent device-free gait recognition system, which can recognize the identity of a person no matter what path he/she walks. Specifically, we propose a novel robust path-independent gait spectrogram construction method, which leverages location information and doppler spectrogram to generate corrective velocity spectrogram so as to approximate the actual velocity spectrogram, and utilizes an energy normalization strategy to eliminate the influence of path on spectrogram energy so as to achieve the independence of the walking path. Based on the path-independent gait spectrogram, we use a convolutional neural network to extract deep features and accomplish the gait recognition task. Experimental results conducted on a 77 GHz mmWave testbed, and achieve the average accuracies of 91% and 87% in the non-radial straight path and curve path, respectively, when 10 targets are trained in the radial path.

A brain-inspired computational model for spatio-temporal information processing

Spatio-temporal information processing is fundamental in both brain functions and AI applications. Current strategies for spatio-temporal pattern recognition usually involve explicit feature extraction followed by feature aggregation, which requires a large amount of labeled data. In the present study, motivated by the subcortical visual pathway and early stages of the auditory pathway for motion and sound processing, we propose a novel brain-inspired computational model for generic spatio-temporal pattern recognition. The model consists of two modules, a reservoir module and a decision-making module. The former projects complex spatio-temporal patterns into spatially separated neural representations via its recurrent dynamics, the latter reads out neural representations via integrating information over time, and the two modules are linked together using known examples. Using synthetic data, we demonstrate that the model can extract the frequency and order information of temporal inputs. We apply the model to reproduce the looming pattern discrimination behavior as observed in experiments successfully. Furthermore, we apply the model to the gait recognition task, and demonstrate that our model accomplishes the recognition in an event-based manner and outperforms deep learning counterparts when training data is limited.

Gait Recognition and Understanding Based on Hierarchical Temporal Memory Using 3D Gait Semantic Folding.

Gait recognition and understanding systems have shown a wide-ranging application prospect. However, their use of unstructured data from image and video has affected their performance, e.g., they are easily influenced by multi-views, occlusion, clothes, and object carrying conditions. This paper addresses these problems using a realistic 3-dimensional (3D) human structural data and sequential pattern learning framework with top-down attention modulating mechanism based on Hierarchical Temporal Memory (HTM). First, an accurate 2-dimensional (2D) to 3D human body pose and shape semantic parameters estimation method is proposed, which exploits the advantages of an instance-level body parsing model and a virtual dressing method. Second, by using gait semantic folding, the estimated body parameters are encoded using a sparse 2D matrix to construct the structural gait semantic image. In order to achieve time-based gait recognition, an HTM Network is constructed to obtain the sequence-level gait sparse distribution representations (SL-GSDRs). A top-down attention mechanism is introduced to deal with various conditions including multi-views by refining the SL-GSDRs, according to prior knowledge. The proposed gait learning model not only aids gait recognition tasks to overcome the difficulties in real application scenarios but also provides the structured gait semantic images for visual cognition. Experimental analyses on CMU MoBo, CASIA B, TUM-IITKGP, and KY4D datasets show a significant performance gain in terms of accuracy and robustness.

Multimodal feature fusion for CNN-based gait recognition: an empirical comparison

People identification in video based on the way they walk (i.e., gait) is a relevant task in computer vision using a noninvasive approach. Standard and current approaches typically derive gait signatures from sequences of binary energy maps of subjects extracted from images, but this process introduces a large amount of non-stationary noise, thus conditioning their efficacy. In contrast, in this paper we focus on the raw pixels, or simple functions derived from them, letting advanced learning techniques to extract relevant features. Therefore, we present a comparative study of different convolutional neural network (CNN) architectures by using three different modalities (i.e., gray pixels, optical flow channels and depth maps) on two widely adopted and challenging datasets: TUM-GAID and CASIA-B. In addition, we perform a comparative study between different early and late fusion methods used to combine the information obtained from each kind of modalities. Our experimental results suggest that (1) the raw pixel values represent a competitive input modality, compared to the traditional state-of-the-art silhouette-based features (e.g., GEI), since equivalent or better results are obtained; (2) the fusion of the raw pixel information with information from optical flow and depth maps allows to obtain state-of-the-art results on the gait recognition task with an image resolution several times smaller than the previously reported results; and (3) the selection and the design of the CNN architecture are critical points that can make a difference between state-of-the-art results or poor ones.

An interpretable gait recognition method based on time series features

Gait recognition is a type of biometric recognition that can be used as an identification tool in various applications. Deep learning-based methods have recently exhibited promising accuracy in gait recognition tasks; however, in addition to an accurate prediction, these methods are required to explain the recognition results. The black-box nature of deep neural networks makes it very difficult to interpret the basis for their identification. The published studies on the interpretability of gait recognition are also in a blank state. Moreover, deep neural networks require a large amount of data to learn the model parameters and an effective generalization on unseen data is difficult when the problem size is small. Thus, this paper presents a gait recognition method combining accuracy and interpretability. The gait feature is represented as a multi-dimensional time series and a Shapelet-based time series classification method is used for gait recognition. A Shapelet is the most discriminative subsequence in time series that makes the proposed method provide interpretability and accuracy simultaneously. We conducted experiments on the CASIA-B dataset and compared the proposed method with several state-of-the-arts deep learning methods. Experiments show that the proposed method can provide an accuracy close to that of deep neural networks on small-scale data sets. At the same time, the decision-making reason of the model can be explained in detail. Concretely, our method can reveal discriminative gait features and frame numbers for specific subjects.

Joint Intensity Transformer Network for Gait Recognition Robust Against Clothing and Carrying Status

Clothing and carrying status variations are the two key factors that affect the performance of gait recognition because people usually wear various clothes and carry all kinds of objects, while walking in their daily life. These covariates substantially affect the intensities within conventional gait representations such as gait energy images. Hence, to properly compare a pair of input gait features, an appropriate metric for joint intensity is needed in addition to the conventional spatial metric. We therefore propose a unified joint intensity transformer network for gait recognition that is robust against various clothing and carrying statuses. Specifically, the joint intensity transformer network is a unified deep learning-based architecture containing three parts: a joint intensity metric estimation net, a joint intensity transformer, and a discrimination network. First, the joint intensity metric estimation net uses a well-designed encoder-decoder network to estimate a sample-dependent joint intensity metric for a pair of input gait energy images. Subsequently, a joint intensity transformer module outputs the spatial dissimilarity of two gait energy images using the metric learned by the joint intensity metric estimation net. Third, the discrimination network is a generic convolution neural network for gait recognition. In addition, the joint intensity transformer network is designed with different loss functions depending on the gait recognition task (i.e., a contrastive loss function for the verification task and a triplet loss function for the identification task). The experiments on the world’s largest datasets containing various clothing and carrying statuses demonstrate the state-of-the-art performance of the proposed method.

On view‐invariant gait recognition: a feature selection solution

The authors present an improved feature selection solution for the view-invariant gait recognition problem, based on their previously proposed method called view-invariant feature selector (ViFS), which automatically reconstruct an optimised gallery template from a set of multi-view gallery templates. They improved ViFS by introducing a constraint to make sure that the reconstructed features have the same scale as the original features, thus reducing the number of misclassifications caused by data misalignment. They evaluate the improved ViFS on the CASIA B and OU-ISIR large population datasets by performing a wide range of comparative studies in order to explore and confirm its effectiveness. Evaluation results indicate that the proposed framework is very effective for view-invariant gait recognition tasks.

Gait recognition from corrupted silhouettes: a robust statistical approach

This paper introduces a method based on robust statistics to build reliable gait signatures from averaging silhouette descriptions, mainly when gait sequences are affected by severe and persistent defects. The term robust refers to the ability of reducing the impact of silhouette defects (outliers) on the average gait pattern, while taking advantage of clean silhouette regions. An extensive experimental framework was defined based on injecting three types of realistic defects (salt and pepper noise, static occlusion, and dynamic occlusion) to clean gait sequences, both separately in an easy setting and jointly in a hard setting. The robust approach was compared against two other operation modes: (1) simple mean (weak baseline) and (2) defect exclusion (strong benchmark). Three gait representation methods based on silhouette averaging were used: Gait Energy Image (GEI), Gradient Histogram Energy Image (GHEI), and the joint use of GEI and HOG descriptors. Quality of gait signatures was assessed by their discriminant power in a large number of gait recognition tasks. Nonparametric statistical tests were applied on recognition results, searching for significant differences between operation modes.

An adaptive neural networks formulation for the two-dimensional principal component analysis

This study, for the first time, developed an adaptive neural networks (NNs) formulation for the two-dimensional principal component analysis (2DPCA), whose space complexity is far lower than that of its statistical version. Unlike the NNs formulation of principal component analysis (PCA, i.e., 1DPCA), the solution with lower iteration in nature aims to directly deal with original image matrices. We also put forward the consistence in the conceptions of `eigenfaces' or `eigengaits' in both 1DPCA and 2DPCA neural networks. To evaluate the performance of the proposed NN, the experiments were carried out on AR face database and on 64 × 64 pixels gait energy images on CASIA(B) gait database. The less reconstruction error was exploited using the proposed NN in the condition of a large sample set compared to adaptive estimation of learning algorithms for NNs of PCA. On the contrary, if the sample set was small, the proposed NN could achieve a higher residue error than PCA NNs. The amount of calculation for the proposed NN here could be smaller than that for the PCA NNs on the feature extraction of the same image matrix, which represented an efficient solution to the problem of training images directly. On face and gait recognition tasks, a simple nearest neighbor classifier test indicated a particular benefit of the neural network developed here which serves as an efficient alternative to conventional PCA NNs.

Uncorrelated Multilinear Discriminant Analysis With Regularization and Aggregation for Tensor Object Recognition

This paper proposes an uncorrelated multilinear discriminant analysis (UMLDA) framework for the recognition of multidimensional objects, known as tensor objects. Uncorrelated features are desirable in recognition tasks since they contain minimum redundancy and ensure independence of features. The UMLDA aims to extract uncorrelated discriminative features directly from tensorial data through solving a tensor-to-vector projection. The solution consists of sequential iterative processes based on the alternating projection method, and an adaptive regularization procedure is incorporated to enhance the performance in the small sample size (SSS) scenario. A simple nearest-neighbor classifier is employed for classification. Furthermore, exploiting the complementary information from differently initialized and regularized UMLDA recognizers, an aggregation scheme is adopted to combine them at the matching score level, resulting in enhanced generalization performance while alleviating the regularization parameter selection problem. The UMLDA-based recognition algorithm is then empirically shown on face and gait recognition tasks to outperform four multilinear subspace solutions (MPCA, DATER, GTDA, TR1DA) and four linear subspace solutions (Bayesian, LDA, ULDA, R-JD-LDA).

Synchronization of oscillations for machine perception of gaits

Substantial evidence supports a relationship between gait perception and gait synthesis. Furthermore, passive mechanical systems demonstrate that the jointed leg systems of humans have innate oscillations that form a gait. These observations suggest that systems may perceive gaits by synchronizing an internal oscillating model to observed oscillations. We present such a system in this paper that uses phase-locked loops to synchronize an internal oscillator with oscillations from a video source. Arrays of phase-locked loops, called video phase-locked loops, synchronize a system with the oscillations in pixel intensities. We then test the perception of the resulting synchronized-oscillator model in various gait recognition tasks. Tools based on Procrustes analysis and directional statistics provide the computational mechanism to compare patterns of oscillations. We discuss the possibility of an alternative model for motion perception based on synchronization with the transient oscillations of temporal band-pass filters that is consistent with other proposed models for human perception. Synchronization of a kinematic model to oscillations also suggests a path to bridge the gap between the model-free and model-based domains.