Cross-view Gait Recognition Research Articles

Cross-View Gait Recognition Using Pairwise Spatial Transformer Networks

In this paper, we propose a pairwise spatial transformer network (PSTN) for cross-view gait recognition, which reduces unwanted feature mis-alignment due to view differences before a recognition step for better performance. The proposed PSTN is a unified CNN architecture that consists of a pairwise spatial transformer (PST) and subsequent recognition network (RN). More specifically, given a matching pair of gait features from different source and target views, the PST estimates a non-rigid deformation field to register the features in the matching pair into their intermediate view, which mitigates distortion by registration compared with the case of direct deformation from the source view to target view. The registered matching pair is then fed into the RN to output a dissimilarity score. Although registration may reduce not only intra-subject variations but also inter-subject variations, we can still achieve a good trade-off between them using a loss function designed to optimize recognition accuracy. Experiments on three publicly available gait datasets demonstrate that the proposed method yields superior performance for both verification and identification scenarios by combining any gait recognition network benchmarks with the PST.

Human Gait Recognition Based on Frame-by-Frame Gait Energy Images and Convolutional Long Short-Term Memory.

Human gait recognition is one of the most promising biometric technologies, especially for unobtrusive video surveillance and human identification from a distance. Aiming at improving recognition rate, in this paper we study gait recognition using deep learning and propose a novel method based on convolutional Long Short-Term Memory (Conv-LSTM). First, we present a variation of Gait Energy Images, i.e. frame-by-frame GEI (ff-GEI), to expand the volume of available Gait Energy Images (GEI) data and relax the constraints of gait cycle segmentation required by existing gait recognition methods. Second, we demonstrate the effectiveness of ff-GEI by analyzing the cross-covariance of one person's gait data. Then, making use of the temporality of our human gait, we design a novel gait recognition model using Conv-LSTM. Finally, the proposed method is evaluated extensively based on the CASIA Dataset B for cross-view gait recognition, furthermore the OU-ISIR Large Population Dataset is employed to verify its generalization ability. Our experimental results show that the proposed method outperforms other algorithms based on these two datasets. The results indicate that the proposed ff-GEI model using Conv-LSTM, coupled with the new gait representation, can effectively solve the problems related to cross-view gait recognition.

Gait recognition using multichannel convolution neural networks

Human gait recognition has a wide range of applications in multiple fields, such as video surveillance, digital security, and forensics. In this paper, we investigate the challenging problem of cross-view gait recognition and propose a novel gait recognition scheme by utilizing the strong expression of convolution neural networks (CNN). First, instead of using gait energy images in traditional gait recognition, we will design a new gait feature representation, trituple gait silhouettes, constructed by using consecutive gait silhouette pictures. Second, we will construct a multichannel CNN network to tackle a set of sequential images in parallel. Each of the image datasets is treated as one input channel, and a different convolutional kernel is used. Finally, the proposed approach is evaluated extensively based on the CASIA gait dataset A/B for cross-view gait recognition, and further on the OU-ISIR large population gait dataset to verify its generalization capability with large-scale data. To the best of our knowledge, this is the first time that this gait recognition scheme is presented. All our experimental results show that the proposed method obtains better performance when compared to those existing methods.

On Input/Output Architectures for Convolutional Neural Network-Based Cross-View Gait Recognition

In this paper, we discuss input/output architectures for convolutional neural network (CNN)-based cross-view gait recognition. For this purpose, we consider two aspects: verification versus identification and the tradeoff between spatial displacements caused by subject difference and view difference. More specifically, we use the Siamese network with a pair of inputs and contrastive loss for verification and a triplet network with a triplet of inputs and triplet ranking loss for identification. The aforementioned CNN architectures are insensitive to spatial displacement, because the difference between a matching pair is calculated at the last layer after passing through the convolution and max pooling layers; hence, they are expected to work relatively well under large view differences. By contrast, because it is better to use the spatial displacement to its best advantage because of the subject difference under small view differences, we also use CNN architectures where the difference between a matching pair is calculated at the input level to make them more sensitive to spatial displacement. We conducted experiments for cross-view gait recognition and confirmed that the proposed architectures outperformed the state-of-the-art benchmarks in accordance with their suitable situations of verification/identification tasks and view differences.

Attentive Spatial–Temporal Summary Networks for Feature Learning in Irregular Gait Recognition

Gait recognition is an attractive human recognition technology. However, existing gait recognition methods mainly focus on the regular gait cycles, which ignore the irregular situation. In real-world surveillance, human gait is almost irregular, which contains arbitrary dynamic characteristics (e.g., duration, speed, and phase) and varied viewpoints. In this paper, we propose the attentive spatial–temporal summary networks to learn salient spatial–temporal and view-independence features for irregular gait recognition. First of all, we design the gate mechanism with attentive spatial–temporal summary to extract the discriminative sequence-level features for representing the periodic motion cues of irregular gait sequences. The designed general attention and residual attention components can concentrate on the discriminative identity-related semantic regions from the spatial feature maps. The proposed attentive temporal summary component can automatically assign adaptive attention to enhance the discriminative gait timesteps and suppress the redundant ones. Furthermore, to improve the accuracy of cross-view gait recognition, we combine the Siamese structure and Null Foley–Sammon transform to obtain the view-invariant gait features from irregular gait sequences. Finally, we quantitatively evaluate the impact of the irregular gait and viewpoint interval between matching pairs on gait recognition accuracy. Experimental results show that our method achieves state-of-the-art performance in irregular gait recognition on the OULP and CASIA-B datasets.

Color-mapped contour gait image for cross-view gait recognition using deep convolutional neural network

In recent decades, gait recognition has garnered a lot of attention from the researchers in the IT era. Gait recognition signifies verifying or identifying the individuals by their walking style. Gait supports in surveillance system by identifying people when they are at a distance from the camera and can be used in numerous computer vision and surveillance applications. This paper proposes a stupendous Color-mapped Contour Gait Image (CCGI) for varying factors of Cross-View Gait Recognition (CVGR). The first contour in each gait image sequence is extracted using a Combination of Receptive Fields (CORF) contour tracing algorithm which extracts the contour image using Difference of Gaussians (DoG) and hysteresis thresholding. Moreover, hysteresis thresholding detects the weak edges from the total pixel information and provides more well-balanced smooth features compared to an absolute one. Second CCGI encodes the spatial and temporal information via color mapping to attain the regularized contour images with fewer outliers. Based on the front view of a human walking pattern, the appearance of cross-view variations would reduce drastically with respect to a change of view angles. This proposed work evaluates the performance analysis of CVGR using Deep Convolutional Neural Network (CNN) framework. CCGI is considered a gait feature for comparing and evaluating the robustness of our proposed model. Experiments conducted on CASIA-B database show the comparisons of previous methods with the proposed method and achieved 94.65% accuracy with a better recognition rate.

GaitSet: Regarding Gait as a Set for Cross-View Gait Recognition

As a unique biometric feature that can be recognized at a distance, gait has broad applications in crime prevention, forensic identification and social security. To portray a gait, existing gait recognition methods utilize either a gait template, where temporal information is hard to preserve, or a gait sequence, which must keep unnecessary sequential constraints and thus loses the flexibility of gait recognition. In this paper we present a novel perspective, where a gait is regarded as a set consisting of independent frames. We propose a new network named GaitSet to learn identity information from the set. Based on the set perspective, our method is immune to permutation of frames, and can naturally integrate frames from different videos which have been filmed under different scenarios, such as diverse viewing angles, different clothes/carrying conditions. Experiments show that under normal walking conditions, our single-model method achieves an average rank-1 accuracy of 95.0% on the CASIA-B gait dataset and an 87.1% accuracy on the OU-MVLP gait dataset. These results represent new state-of-the-art recognition accuracy. On various complex scenarios, our model exhibits a significant level of robustness. It achieves accuracies of 87.2% and 70.4% on CASIA-B under bag-carrying and coat-wearing walking conditions, respectively. These outperform the existing best methods by a large margin. The method presented can also achieve a satisfactory accuracy with a small number of frames in a test sample, e.g., 82.5% on CASIA-B with only 7 frames. The source code has been released at https://github.com/AbnerHqC/GaitSet.

Cross-view Gait Recognition by Discriminative Feature Learning.

Recently, deep learning based cross-view gait recognition becomes popular owing to the strong capacity of convolutional neural networks (CNNs). Current deep learning methods often rely on loss functions used widely in the task of face recognition, e.g., contrastive loss and triplet loss. These loss functions have the problem of hard negative mining. In this paper, a robust, effective and gait-related loss function, called angle center loss (ACL), is proposed to learn discriminative gait features. The proposed loss function is robust to different local parts and temporal window sizes. Different from center loss which learns a center for each identity, the proposed loss function learns multiple sub-centers for each angle of the same identity. Only the largest distance between the anchor feature and the corresponding crossview sub-centers is penalized, which achieves better intra-subject compactness. We also propose to extract discriminative spatialtemporal features by local feature extractors and a temporal attention model. A simplified spatial transformer network is proposed to localize the suitable horizontal parts of the human body. Local gait features for each horizontal part are extracted and then concatenated as the descriptor. We introduce long-short term memory (LSTM) units as the temporal attention model to learn the attention score for each frame, e.g., focusing more on discriminative frames and less on frames with bad quality. The temporal attention model shows better performance than the temporal average pooling or gait energy images (GEI). By combing the three aspects, we achieve the state-of-the-art results on several cross-view gait recognition benchmarks.

Cross-view gait recognition through ensemble learning

Gait has been well known as an unobtrusive promising biometric to identify a person from a distance. However, the effectiveness of silhouette-based approaches in gait recognition is diluted due to variations of view angles. In this paper, we put forward a novel and effective method of gait recognition: cross-view gait recognition based on ensemble learning. The proposed method greatly enhances the effectiveness and reduces the sensitivity of gait recognition under various view angles conditions. Furthermore, in this paper we will introduce a novel algorithm based on ensemble learning for combining several gait learners together, which utilizes a well-designed gait feature based on area average distance. Through experimental evaluations on the well-known CASIA gait database and OU-ISIR gait database, our paper demonstrates the advantages of the proposed method in comparison with others. The contribution of this research work is to resolve the multiview angles problem of gait recognition through assembling several gait learners.

Cross-view gait recognition based on a restrictive triplet network

To overcome the influence of view variations, a restrictive triplet network (RTN) is proposed to solve the problem of cross-view gait recognition in this paper. This network comprises five shared convolutional layers. The restrictive triplet loss is adopted to optimize RTN based on the triplet-based sample groups. These gait samples are selected by a special strategy, so as to make RTN converges faster. The model optimized by this method is adopted to extract the view-invariant feature for cross-view gait recognition. Besides, two additional networks named BDN and TDN are proposed to compare with RTN, which match the adjacent features at different convolutional layers. Finally, extensive evaluations are conducted based on the CASIA-B, OU-ISIR and USF dataset. Experimental results indicate that RTN performs best. Besides, the state-of-the-art methods are selected to compare with RTN. Among them, RTN achieves the best recognition score, which reaches 94.62% under singe view angle and 91.68% under cross-view angle, respectively. The results demonstrate that RTN is robust against view variations, which shows the great potential of RTN for practical applications in the future.

Learning view invariant gait features with Two-Stream GAN

Gait recognition is an important yet challenging problem in computer vision. The changing view of gait is one of the most challenging factors, which could greatly affect the accuracy of cross-view gait recognition. In this paper, we propose a Two-Stream Generative Adversarial Network (TS-GAN) for cross-view gait recognition. For any view of gait representations, GAN can restore it to the corresponding standard view, to learn view invariant gait features. To achieve this goal, TS-GAN has two streams : (1) the global-stream can learn global contexts, and (2) the part-stream can learn local details. We combine the two streams to learn final identities. Moreover, we add a pixel-wise loss along with the generators of GAN to restore the gait details in pixel-level. We evaluate the proposed method on two widely used gait databases: CASIA-B and OU-ISIR. Experiment results show that our approach outperforms the compared state-of-the-art approaches.

Coupled Bilinear Discriminant Projection for Cross-View Gait Recognition

A problem that hinders good performance of general gait recognition systems is that the appearance features of gaits are more affected-prone by views than identities, especially when the walking direction of the probe gait is different from the register gait. This problem cannot be solved by traditional projection learning methods because these methods can learn only one projection matrix, and thus for the same subject, it cannot transfer cross-view gait features into similar ones. This paper presents an innovative method to overcome this problem by aligning gait energy images (GEIs) across views with the coupled bilinear discriminant projection (CBDP). Specifically, the CBDP generates the aligned gait matrix features for two views with two sets of bilinear transformation matrices, so that the original GEIs’ spatial structure information can be preserved. By iteratively maximizing the ratio of inter-class distance metric to intra-class distance metric, the CBDP can learn the optimal matrix subspace where the GEIs across views are aligned in both horizontal and vertical coordinates. Therefore, the CBDP is also able to avoid the under-sample problem. We also theoretically prove that the upper and lower bounds of the objective function sequence of the CBDP are both monotonically increasing, so the convergence of the CBDP is demonstrated. In the terms of accuracy, the comparative experiments on the CASIA (B) and OU-ISIR gait databases show that our method is superior to the state-of-the-art cross-view gait recognition methods. More impressively, encouraging performance is obtained by our method even in matching a lateral-view gait with a frontal-view gait.

Coupled Patch Alignment for Matching Cross-view Gaits.

Gait recognition has attracted growing attention in recent years as the gait of humans has a strong discriminative ability even under low resolution at a distance. Unfortunately, the performance of gait recognition can be largely affected by view change. To address this problem, we propose a Coupled Patch Alignment (CPA) algorithm that effectively matches a pair of gaits across different views. To realize CPA, we first build a certain amount of patches, and each of them is made up of a sample as well as its intra-class and inter-class nearest-neighbors. Then we design an objective function for each patch to balance the cross-view intra-class compactness and the cross-view inter-class separability. Finally, all the local independent patches are combined to render a unified objective function. Theoretically, we show that the proposed CPA has a close relationship with Canonical Correlation Analysis (CCA). Algorithmically, we extend CPA to "Multi-dimensional Patch Alignment" (MPA) that can handle an arbitrary number of views. Comprehensive experiments on CASIA(B), USF and OU-ISIR gait databases firmly demonstrate the effectiveness of our methods over other existing popular methods in terms of cross-view gait recognition.

A general tensor representation framework for cross-view gait recognition

Tensor analysis methods have played an important role in identifying human gaits using high dimensional data. However, when view angles change, it becomes more and more difficult to recognize cross-view gait by learning only a set of multi-linear projection matrices. To address this problem, a general tensor representation framework for cross-view gait recognition is proposed in this paper. There are three criteria of tensorial coupled mappings in the proposed framework. (1) Coupled multi-linear locality-preserved criterion (CMLP) aims to detect the essential tensorial manifold structure via preserving local information. (2) Coupled multi-linear marginal fisher criterion (CMMF) aims to encode the intra-class compactness and inter-class separability with local relationships. (3) Coupled multi-linear discriminant analysis criterion (CMDA) aims to minimize the intra-class scatter and maximize the inter-class scatter. For the three tensor algorithms for cross-view gaits, two sets of multi-linear projection matrices are iteratively learned using alternating projection optimization procedures. The proposed methods are compared with the recently published cross-view gait recognition approaches on CASIA(B) and OU-ISIR gait database. The results demonstrate that the performances of the proposed methods are superior to existing state-of-the-art cross-view gait recognition approaches.

Multi-Task GANs for View-Specific Feature Learning in Gait Recognition

Gait recognition is of great importance in the fields of surveillance and forensics to identify human beings since gait is the unique biometric feature that can be perceived efficiently at a distance. However, the accuracy of gait recognition to some extent suffers from both the variation of view angles and the deficient gait templates. On one hand, the existing cross-view methods focus on transforming gait templates among different views, which may accumulate the transformation error in a large variation of view angles. On the other hand, a commonly used gait energy image template loses temporal information of a gait sequence. To address these problems, this paper proposes multi-task generative adversarial networks (MGANs) for learning view-specific feature representations. In order to preserve more temporal information, we also propose a new multi-channel gait template, called period energy image (PEI). Based on the assumption of view angle manifold, the MGANs can leverage adversarial training to extract more discriminative features from gait sequences. Experiments on OU-ISIR, CASIA-B, and USF benchmark data sets indicate that compared with several recently published approaches, PEI + MGANs achieves competitive performance and is more interpretable to cross-view gait recognition.

Multi-view large population gait dataset and its performance evaluation for cross-view gait recognition

This paper describes the world’s largest gait database with wide view variation, the “OU-ISIR gait database, multi-view large population dataset (OU-MVLP)”, and its application to a statistically reliable performance evaluation of vision-based cross-view gait recognition. Specifically, we construct a gait dataset that includes 10,307 subjects (5114 males and 5193 females) from 14 view angles ranging 0° −90°, 180° −270°.In addition, we evaluate various approaches to gait recognition which are robust against view angles. By using our dataset, we can fully exploit a state-of-the-art method requiring a large number of training samples, e.g., CNN-based cross-view gait recognition method, and we validate effectiveness of such a family of the methods.

Multiview max-margin subspace learning for cross-view gait recognition

Cross-view gait recognition can be regarded as a domain adaption problem, in which, probe gait to be recognized in one view is different from gallery gaits collected in another view. In this paper, we present a subspace learning based method, called Multiview Max-Margin Subspace Learning (MMMSL), to learn a common subspace for associating gait data across different views. A group of projection matrices that respectively map data from different views into the common subspace are optimized via simultaneously minimizing the within-class variations and maximizing the local between-class variations of the low-dimensional embeddings from both inter-view and intra-view. In the learnt subspace, same-class samples from all views cluster together, and each different-class cluster is kept away from its nearest neighbors as far as possible. Experimental results on two benchmark gait databases, CASIA-B and OU-ISIR, demonstrate the effectiveness of the proposed method. Extensive experiments also show that our MMMSL achieves significant improvements compared with related subspace learning based methods.

GII Representation-Based Cross-View Gait Recognition by Discriminative Projection With List-Wise Constraints

Remote person identification by gait is one of the most important topics in the field of computer vision and pattern recognition. However, gait recognition suffers severely from the appearance variance caused by the view change. It is very common that gait recognition has a high performance when the view is fixed but the performance will have a sharp decrease when the view variance becomes significant. Existing approaches have tried all kinds of strategies like tensor analysis or view transform models to slow down the trend of performance decrease but still have potential for further improvement. In this paper, a discriminative projection with list-wise constraints (DPLC) is proposed to deal with view variance in cross-view gait recognition, which has been further refined by introducing a rectification term to automatically capture the principal discriminative information. The DPLC with rectification (DPLCR) embeds list-wise relative similarity measurement among intraclass and inner-class individuals, which can learn a more discriminative and robust projection. Based on the original DPLCR, we have introduced the kernel trick to exploit nonlinear cross-view correlations and extended DPLCR to deal with the problem of multiview gait recognition. Moreover, a simple yet efficient gait representation, namely gait individuality image (GII), based on gait energy image is proposed, which could better capture the discriminative information for cross view gait recognition. Experiments have been conducted in the CASIA-B database and the experimental results demonstrate the outstanding performance of both the DPLCR framework and the new GII representation. It is shown that the DPLCR-based cross-view gait recognition has outperformed the-state-of-the-art approaches in almost all cases under large view variance. The combination of the GII representation and the DPLCR has further enhanced the performance to be a new benchmark for cross-view gait recognition.

Coupled locality preserving projections for cross-view gait recognition

Existing methods for gait recognition mainly depend on the appearance of human. Their performances are greatly affected by changes of viewing angle. To achieve higher correct classification rates for cross-view gait recognition, we develop a coupled locality preserving projections (CLPP) method in this paper. It learns coupled projection matrices to project cross-view features into a unified subspace while preserving the essential manifold structure. In the projected subspace, cross-view gait features can be matched directly. By the virtue of structure information, the learnt subspace is more robust to the view change. Experiments based on CASIA and USF gait databases are conducted to verify the efficiency of our approach.

A Comprehensive Study on Cross-View Gait Based Human Identification with Deep CNNs.

This paper studies an approach to gait based human identification via similarity learning by deep convolutional neural networks (CNNs). With a pretty small group of labeled multi-view human walking videos, we can train deep networks to recognize the most discriminative changes of gait patterns which suggest the change of human identity. To the best of our knowledge, this is the first work based on deep CNNs for gait recognition in the literature. Here, we provide an extensive empirical evaluation in terms of various scenarios, namely, cross-view and cross-walking-condition, with different preprocessing approaches and network architectures. The method is first evaluated on the challenging CASIA-B dataset in terms of cross-view gait recognition. Experimental results show that it outperforms the previous state-of-the-art methods by a significant margin. In particular, our method shows advantages when the cross-view angle is large, i.e., no less than 36 degree. And the average recognition rate can reach 94 percent, much better than the previous best result (less than 65 percent). The method is further evaluated on the OU-ISIR gait dataset to test its generalization ability to larger data. OU-ISIR is currently the largest dataset available in the literature for gait recognition, with 4,007 subjects. On this dataset, the average accuracy of our method under identical view conditions is above 98 percent, and the one for cross-view scenarios is above 91 percent. Finally, the method also performs the best on the USF gait dataset, whose gait sequences are imaged in a real outdoor scene. These results show great potential of this method for practical applications.