Pedestrian Pose Research Articles

Pedestrian Pose Recognition Based on Frequency-Modulated Continuous-Wave Radar with Meta-Learning.

With the continuous advancement of autonomous driving and monitoring technologies, there is increasing attention on non-intrusive target monitoring and recognition. This paper proposes an ArcFace SE-attention model-agnostic meta-learning approach (AS-MAML) by integrating attention mechanisms into residual networks for pedestrian gait recognition using frequency-modulated continuous-wave (FMCW) millimeter-wave radar through meta-learning. We enhance the feature extraction capability of the base network using channel attention mechanisms and integrate the additive angular margin loss function (ArcFace loss) into the inner loop of MAML to constrain inner loop optimization and improve radar discrimination. Then, this network is used to classify small-sample micro-Doppler images obtained from millimeter-wave radar as the data source for pose recognition. Experimental tests were conducted on pose estimation and image classification tasks. The results demonstrate significant detection and recognition performance, with an accuracy of 94.5%, accompanied by a 95% confidence interval. Additionally, on the open-source dataset DIAT-μRadHAR, which is specially processed to increase classification difficulty, the network achieves a classification accuracy of 85.9%.

Identity-Guided Collaborative Learning for Cloth-Changing Person Reidentification.

Cloth-changing person reidentification (ReID) is a newly emerging research topic aimed at addressing the issues of large feature variations due to cloth-changing and pedestrian view/pose changes. Although significant progress has been achieved by introducing extra information (e.g., human contour sketching information, human body keypoints, and 3D human information), cloth-changing person ReID remains challenging because pedestrian appearance representations can change at any time. Moreover, human semantic information and pedestrian identity information are not fully explored. To solve these issues, we propose a novel identity-guided collaborative learning scheme (IGCL) for cloth-changing person ReID, where the human semantic is effectively utilized and the identity is unchangeable to guide collaborative learning. First, we design a novel clothing attention degradation stream to reasonably reduce the interference caused by clothing information where clothing attention and mid-level collaborative learning are employed. Second, we propose a human semantic attention and body jigsaw stream to highlight the human semantic information and simulate different poses of the same identity. In this way, the extraction features not only focus on human semantic information that is unrelated to the background but are also suitable for pedestrian pose variations. Moreover, a pedestrian identity enhancement stream is proposed to enhance the identity importance and extract more favorable identity robust features. Most importantly, all these streams are jointly explored in an end-to-end unified framework, and the identity is utilized to guide the optimization. Extensive experiments on six public clothing person ReID datasets (LaST, LTCC, PRCC, NKUP, Celeb-reID-light, and VC-Clothes) demonstrate the superiority of the IGCL method. It outperforms existing methods on multiple datasets, and the extracted features have stronger representation and discrimination ability and are weakly correlated with clothing.

Pose-Aided Video-Based Person Re-Identification via Recurrent Graph Convolutional Network

Existing methods for video-based person re-identification (ReID) mainly learn the appearance feature of a given pedestrian via a feature extractor and a feature aggregator. However, the appearance models would fail to learn a large inter-class variance when different pedestrians have similar appearances. Considering that different pedestrians have different walking postures and body proportions, we propose to learn the discriminative pose feature beyond the appearance feature for video retrieval. Specifically, we implement a two-branch architecture to separately learn the appearance feature and pose feature, and then concatenate them together for inference. To learn the pose feature, we first detect the pedestrian pose in each frame through an off-the-shelf pose detector, and construct a temporal graph using the pose sequence. We then exploit a recurrent graph convolutional network (RGCN) to learn the node embeddings of the temporal pose graph, which devises a global information propagation mechanism to simultaneously achieve the neighborhood aggregation of intra-frame nodes and message passing among inter-frame graphs. Finally, we propose a dual-attention method (DAM) consisting of node-attention and time-attention to obtain the temporal graph representation from the node embeddings, where the self-attention mechanism is employed to learn the importance of each node and each frame. We verify the proposed method on three video-based ReID datasets, i.e., Mars, DukeMTMC and iLIDS-VID, whose experimental results demonstrate that the learned pose feature can effectively improve the performance of existing appearance models.

Towards comprehensive understanding of pedestrians for autonomous driving: Efficient multi-task-learning-based pedestrian detection, tracking and attribute recognition

Safe interaction with pedestrians is important for autonomous vehicles in urban environments. Comprehensive understanding of pedestrian behavior is a challenge for autonomous driving, as different pedestrian attributes have different characteristics and there is not yet a dataset providing all attribute labels. In this paper, we design an efficient multi-task model for pedestrian detection, tracking and attribute recognition to fully mine information in multiple tasks and multiple datasets to reduce redundant image processing computation. Multiple training datasets are utilized to take full advantage of the available partially-labeled data. The model consists of two stages. The first stage is to detect and track pedestrians, using only low-resolution images to save computational resources. Accurate pedestrian detection is achieved in the bird's-eye-view (BEV), while pedestrian tracking is implemented on the basis of 3D bounding boxes. The second stage is image-based multi-attribute detection. This stage uses the detection and tracking results and some pre-processed information from the first stage along with high-resolution images. Multi-attributes of the pedestrian are identified, such as pedestrian's pose, age, gender, and motion direction. With a carefully-designed multi-task system that shares necessary feature maps among multiple tasks and reuses computation results, we can solve pedestrian detection, tracking, and multi-attribute recognition problems in an efficient framework. This leads to more accurate decisions and more efficient computations. The innovation of the model is to accomplish a simultaneous detection of multiple pedestrian attributes for autonomous driving, while fully sharing the features in the model to avoid repeated computation, as well as utilizing multiple partially-labeled datasets for the pedestrian detection, tracking and attribute recognition model training.

Pose‐guided adversarial video prediction for image‐to‐video person re‐identification

AbstractThe image‐to‐video (I2V) person re‐identification (Re‐ID) is a cross‐modality pedestrian retrieval task, whose crux is to reduce the large modality discrepancy between images and videos. To this end, this paper proposes to predict the following video frames from a single image. Thus, the I2V person Re‐ID can be transformed to video‐to‐video (V2V) Re‐ID. Considering that predicting video frames from a single image is an ill‐posed problem, this paper proposes two strategies to improve the quality of the predicted videos. First, a pose‐guided video prediction pipeline is proposed. The given single image and pedestrian pose are encoded via image encoder and pose encoder, respectively; then, the image feature and pose feature are concatenated as the input of the video decoder. The authors minimize the difference between the predicted video and true video, and simultaneously minimize the difference between the true pose and predicted pose. Second, the conditional adversarial training strategy is employed to generate high‐quality video frames. Specifically, the discriminator takes the source image as condition and distinguishes whether the input frames are fake or true following frames of the source image. Experimental results demonstrate that the pose‐guided adversarial video prediction can effectively improve accuracy of I2V Re‐ID.

Design of Real-time Pedestrian Trajectory Prediction System based on Jetson Xavier

This paper presents a vehicle-mounted real-time pedestrian trajectory prediction system based on the embedded device Jetson Xavier. It achieves low-cost real-time pedestrian trajectory prediction using only the front camera of the vehicle. Firstly, pedestrian detection and tracking are implemented based on YoLoV7, while also estimating pedestrian poses and optical flow to provide multiple information sequences for the trajectory prediction network. Secondly, the pedestrian trajectory algorithm from a driver's perspective is studied, and a trajectory prediction algorithm that considers pedestrian pose, optical flow, and trajectory information is proposed. A novel multi-information fusion network is designed to better integrate multiple features. The algorithm is tested on the JAAD and PIE datasets, and the displacement errors are reduced by 6.35% and 3.28%, respectively, compared to BiTraP. Finally, the algorithm is ported to the embedded device Xavier and installed on a simulated vehicle for testing. By predicting pedestrian future trajectories based on pedestrian detection, collisions can be avoided in advance, improving the safety of autonomous driving. The proposed system has significant practical value.

Prediction of the Future State of Pedestrians While Jaywalking Under Non-Lane-Based Heterogeneous Traffic Conditions

This study proposes a novel framework to predict jaywalkers' future state in non-lane-based heterogeneous traffic conditions by combining the effects of the surrounding dynamics with jaywalkers' poses. Different variables, such as the pedestrian pose, walking speed, location in the road environment, count and direction of approaching traffic, speed and type of closest approaching vehicle, and so forth, are used as input variables. The dataset for this study consists of 47,588 samples gathered by analyzing 1753 jaywalkers under non-lane-based heterogeneous traffic situations. Keypoint detection on the pedestrian body is made using MediaPipe. YOLOv4 and DeepSORT are used to detect and track road users to get trajectory data. Training and testing datasets are prepared for different prediction horizons to test the proposed models’ applicability for roads of varying design speeds. Four machine learning models based on ensemble techniques, namely random forest (RF), adaptive boosting (AdaBoost), gradient boosting, and extreme gradient boosting, are trained and tested for different prediction horizons from 0.5 to 4 s. Up to the prediction horizon of 1 s, all models performed equally well with Area under the ROC curve (AUC) values above 0.95. At higher prediction horizons, the RF is found to outperform the other models. All models, except AdaBoost, maintained an AUC value of greater than 0.9 when predicting future states up to a maximum of 2.5 s. The proposed model performs well for both short-term and long-term predictions by combining the effect of surrounding dynamics with pedestrian stance and speed. The outcomes can be utilized to assist infrastructure-to-vehicle connectivity in empowering vehicles to navigate through jaywalkers safely, enhancing pedestrian safety.

Indoor Positioning on Smartphones Using Built-In Sensors and Visual Images

With the rapid development of mobile Internet technology, localization using visual image information has become a hot problem in the field of indoor localization research, which is not affected by signal multipath and fading and can achieve high accuracy localization in indoor areas with complex electromagnetic environments. However, in practical applications, position estimation using visual images is easily influenced by the user's photo pose. In this paper, we propose a multiple-sensor-assisted visual localization method in which the method constructs a machine learning classifier using multiple smart sensors for pedestrian pose estimation, which improves the retrieval efficiency and localization accuracy. The method mainly combines the advantages of visual image location estimation and pedestrian pose estimation based on multiple smart sensors and considers the effect of pedestrian photographing poses on location estimation. The built-in sensors of smartphones are used as the source of pedestrian pose estimation data, which constitutes a feasible location estimation method based on visual information. Experimental results show that the method proposed in this paper has good localization accuracy and robustness. In addition, the experimental scene in this paper is a common indoor scene and the experimental device is a common smartphone. Therefore, we believe that the proposed method in this paper has the potential to be widely used in future indoor navigation applications in complex scenarios (e.g., mall navigation).

Real-time pedestrian pose estimation, tracking and localization for social distancing.

The corona virus pandemic has introduced limitations which were previously not a cause for concern. Chief among them are wearing face masks in public and constraints on the physical distance between people as an effective measure to reduce the virus spread. Visual surveillance systems, which are common in urban environments and initially commissioned for security surveillance, can be re-purposed to help limit the spread of COVID-19 and prevent future pandemics. In this work, we propose a novel integration technique for real-time pose estimation and multiple human tracking in a pedestrian setting, primarily for social distancing, using CCTV camera footage. Our technique promises a sizeable increase in processing speed and improved detection in very low-resolution scenarios. Using existing surveillance systems, pedestrian pose estimation, tracking and localization for social distancing (PETL4SD) is proposed for measuring social distancing, which combines the output of multiple neural networks aided with fundamental 2D/3D vision techniques. We leverage state-of-the-art object and pose estimation algorithms, combining their strengths, for increase in speed and improvement in detections. These detections are then tracked using a bespoke version of the FASTMOT algorithm. Temporal and analogous estimation techniques are used to deal with occlusions when estimating posture. Projective geometry along with the aforementioned posture tracking is then used to localize the pedestrians. Inter-personal distances are calculated and locally inspected to detect possible violations of the social distancing rules. Furthermore, a "smart violations detector" is employed which estimates if people are together based on their current actions and eliminates false social distancing violations within groups. Finally, distances are intuitively visualized with the right perspective. All implementation is in real time and is performed on Python. Experimental results are provided to validate our proposed method quantitatively and qualitatively on public domain datasets using only a single CCTV camera feed as input. Our results show our technique to outperform the baseline in speed and accuracy in low-resolution scenarios. The code of this work will be made publicly available on GitHub at https://github.com/bilalze/PETL4SD.

Person Re-Identification Combined with Style Transfer and Pose Generation

The number of existing person re-identification datasets is limited, and there are a series of changes such as illumination, background occlusion and pose among each dataset, which makes it difficult for the existing methods to learn robust feature representation, leading to a decline in recognition performance. To solve these problems, a person re-identification method combining style and pose generation is proposed in this paper. First with the impact of camera style differences in collecting images from different cameras, a style transformation method based on generative adversarial network is introduced into a person re-identification model, and cyclic generative adversarial networks (CycleGAN) is used to realize style transfer and reduce the influence of camera differences. Second, in view of the problem that when pedestrian pose changes greatly, easy to ignore identity-sensitive related information, AlphaPose is introduced to implement pose estimation. Combining style and posture for the first time and using improved deep convolution generative adversarial network (DCGAN) structure enrich the input sample information and generate unified style pose image; while using new synthetic data to train person re-identification network model improves the recognition performance of the model. Finally, further introducing random erasure method during data enhancement, in order to reduce the overfitting phenomena, improves the generalization ability of the network simultaneously and solves partial occlusion. The experimental results show that the proposed method outperforms typical style-based or pose-based methods. The accuracy of rank-1 and mAP on Market-1501 dataset is 90.4% and 74.5%, respectively, which are 2.28% and 5.78% higher, respectively. To a certain extent, the performance of person re-identification is improved.

Cross-domain person re-identification with pose-invariant feature decomposition and hypergraph structure alignment

Person Re-identification (Re-ID) has attracted more and more attention thanks to its great practical value in the field of video surveillance. Most works have focused on solving the problem of supervised Re-ID on a single domain and made significant progress. However, the cross-domain Re-ID is still challenging due to the domain bias between the source and target domains. To this end, we propose a dictionary learning algorithm based on matrix factorization to eliminate the influence of style and pedestrian pose information on the cross-domain Re-ID. Specifically, the proposed approach includes two novel parts: (1) the original visual feature is decomposed into pose-invariant feature space, camera-style feature space and residual feature space to extract discriminant pose-invariant feature that is not affected by style and pedestrian pose information, such that the influence of interference information between pedestrians on recognition can be eliminated; (2) considering the domain-invariance of attribute, a hypergraph structure alignment is introduced to integrate pose-invariant feature, attribute and pedestrian identity into a dictionary learning framework. The relationship between pose-invariant feature and attribute is built so that the pedestrian attribute of the target dataset can be accurately predicted during testing. Finally, the pedestrian similarity measurement can be carried out by combining the pose-invariant feature and attribute of pedestrians. The effectiveness of the proposed algorithm is verified with the experiments on several benchmark Re-ID datasets.

Visual and Visual‐Inertial SLAM: State of the Art, Classification, and Experimental Benchmarking

Simultaneous Localization and Mapping is now widely adopted by many applications, and researchers have produced very dense literature on this topic. With the advent of smart devices, embedding cameras, inertial measurement units, visual SLAM (vSLAM), and visual‐inertial SLAM (viSLAM) are enabling novel general public applications. In this context, this paper conducts a review of popular SLAM approaches with a focus on vSLAM/viSLAM, both at fundamental and experimental levels. It starts with a structured overview of existing vSLAM and viSLAM designs and continues with a new classification of a dozen main state‐of‐the‐art methods. A chronological survey of viSLAM’s development highlights the historical milestones and presents more recent methods into a classification. Finally, the performance of vSLAM is experimentally assessed for the use case of pedestrian pose estimation with a handheld device in urban environments. The performance of five open‐source methods Vins‐Mono, ROVIO, ORB‐SLAM2, DSO, and LSD‐SLAM is compared using the EuRoC MAV dataset and a new visual‐inertial dataset corresponding to urban pedestrian navigation. A detailed analysis of the computation results identifies the strengths and weaknesses for each method. Globally, ORB‐SLAM2 appears to be the most promising algorithm to address the challenges of urban pedestrian navigation, tested with two datasets.

Local Alignment Deep Network for Infrared-Visible Cross-Modal Person Reidentification in 6G-Enabled Internet of Things

In this article, we propose a novel deep framework termed local alignment deep network (LADN) for infrared-visible cross-modal person reidentification (IVCM ReID) in 6G-enabled IoT, which could meet the demands of all-day and real-time surveillance. The proposed LADN is designed as a two-stream structure, and it learns shallow interested feature maps and common subspace feature maps to reduce the gap between IR and RGB images. To overcome the challenge of pose and viewpoint variations of pedestrians, we learn the local features in the deep layers. We also propose the local alignment triplet loss (LAT) to align local features, which could capture the consistent local information via comparing noncorresponding local features in a certain range. Furthermore, we learn the global features to provide the global field of vision for the representation. The proposed LADN is optimized in an end-to-end way by combining different cross-modality losses. We evaluate the proposed method on two standard benchmark data sets, i.e., SYSU-MM01 and RegDB, and the results demonstrate the effectiveness of LADN.

Unsupervised Pedestrian Pose Prediction: A Deep Predictive Coding Network-Based Approach for Autonomous Vehicle Perception

Pedestrian pose prediction is an important topic, related closely to robotics and automation. Accurate predictions of human poses and motion can facilitate a more thorough understanding and analysis of human behavior, which benefits real-world applications such as human-robot interaction, humanoid and bipedal robot design, and safe navigation of mobile robots and autonomous vehicles. This article describes a deep predictive coding network (PredNet)-based approach for unsupervised pedestrian pose prediction from 2D camera imagery and provides experimental results of two real-world autonomous vehicle data sets. The article also discusses topics for future work in unsupervised and semisupervised pedestrian pose prediction and its potential applications in robotics and automation systems.

A Deep Learning-Based Hybrid Framework for Object Detection and Recognition in Autonomous Driving

As a key technology of intelligent transportation system, the intelligent vehicle is the carrier of comprehensive integration of many technologies. Although vision-based autonomous driving has shown excellent prospects, there is still a problem of how to analyze the complicated traffic situation by the collected data. Recently, autonomous driving has been formulated as many tasks separately by using different models, such as object detection task and intention recognition task. In this study, a vision-based system was developed to detect and identity various objects and predict the intention of pedestrians in the traffic scene. The main contributions of this research are (1) an optimized model was presented to detect 10 kinds of objects based on the structure of YOLOv4; (2) a fine-tuned Part Affinity Fields approach was proposed to estimate the pose of pedestrians; (3) Explainable Artificial Intelligence (XAI) technology is added to explain and assist the estimation results in the risk assessment phase; (4) an elaborate self-driving dataset that includes several different subsets for each corresponding task was introduced; and (5) an end-to-end system containing multiple models with high accuracy was developed. Experimental results proved that the total parameters of optimized YOLOv4 are reduced by 74%, which satisfies the real-time capability. In addition, the detection precision of the optimized YOLOv4 achieved an improvement of 2.6% compared to the state-of-the-art.

Recurrent matching networks of spatial alignment learning for person re-identification

Person re-identification (re-id) usually refers to matching people across disjoint camera views. Many existing methods focus on extracting discriminative features or learning distance metrics to make the intraclass distance smaller than interclass distances. These methods subconsciously assume that pedestrian images are well aligned. However, one major challenge in person re-id is the unconstrained spatial misalignment between image pairs due to view angle changes and pedestrian pose variations. To address this problem, in this paper, we propose Recurrent Matching Network of Spatial Alignment Learning (RMN-SAL) to simulate the human vision perception. Reinforcement learning is introduced to locate attention regions, since it provides a flexible learning strategy for sequential decision-making. A linear mapping is employed to convert the environment state into spatial constraint, comprising spatial alignment into feature learning. And recurrent models are used to extract information from a sequence of corresponding regions. Finally, person re-id is performed based on the global features and the features from the learned alignment regions. Our contributions are: 1) the recurrent matching network, which can subtly combine local feature learning and sequential spatial correspondence learning into an end-to-end framework; 2) the design of a location network, which is based on reinforcement learning and aims to learn task-specific sequential spatial correspondences for different image pairs through the local pairwise internal representation interactions. The proposed model is evaluated on three benchmarks, including Market-1501, DukeMTMC-reID and CUHK03, and achieves better performances than other methods.

Bio-LSTM: A Biomechanically Inspired Recurrent Neural Network for 3-D Pedestrian Pose and Gait Prediction

In applications such as autonomous driving, it is important to understand, infer, and anticipate the intention and future behavior of pedestrians. This ability allows vehicles to avoid collisions and improve ride safety and quality. This paper proposes a biomechanically inspired recurrent neural network (Bio-LSTM) that can predict the location and 3D articulated body pose of pedestrians in a global coordinate frame, given 3D poses and locations estimated in prior frames with inaccuracy. The proposed network is able to predict poses and global locations for multiple pedestrians simultaneously, for pedestrians up to 45 meters from the cameras (urban intersection scale). The outputs of the proposed network are full-body 3D meshes represented in Skinned Multi-Person Linear (SMPL) model parameters. The proposed approach relies on a novel objective function that incorporates the periodicity of human walking (gait), the mirror symmetry of the human body, and the change of ground reaction forces in a human gait cycle. This paper presents prediction results on the PedX dataset, a large-scale, in-the-wild data set collected at real urban intersections with heavy pedestrian traffic. Results show that the proposed network can successfully learn the characteristics of pedestrian gait and produce accurate and consistent 3D pose predictions.

Estimation of Pedestrian Pose Orientation Using Soft Target Training Based on Teacher⁻Student Framework.

Semi-supervised learning is known to achieve better generalisation than a model learned solely from labelled data. Therefore, we propose a new method for estimating a pedestrian pose orientation using a soft-target method, which is a type of semi-supervised learning method. Because a convolutional neural network (CNN) based pose orientation estimation requires large numbers of parameters and operations, we apply the teacher–student algorithm to generate a compressed student model with high accuracy and compactness resembling that of the teacher model by combining a deep network with a random forest. After the teacher model is generated using hard target data, the softened outputs (soft-target data) of the teacher model are used for training the student model. Moreover, the orientation of the pedestrian has specific shape patterns, and a wavelet transform is applied to the input image as a pre-processing step owing to its good spatial frequency localisation property and the ability to preserve both the spatial information and gradient information of an image. For a benchmark dataset considering real driving situations based on a single camera, we used the TUD and KITTI datasets. We applied the proposed algorithm to various driving images in the datasets, and the results indicate that its classification performance with regard to the pose orientation is better than that of other state-of-the-art methods based on a CNN. In addition, the computational speed of the proposed student model is faster than that of other deep CNNs owing to the shorter model structure with a smaller number of parameters.

Deep Fusion Feature Presentations for Nonaligned Person Re-Identification

Person re-identification aims to retrieve the pedestrian across different cameras. It is still a challenging task for the intelligent visual surveillance system because of similar appearances, camera shooting angles, scene illumination, and pedestrian pose. In this paper, we propose a novel two-stream network named spatial segmentation network that learns both the global and local features in a unified framework for nonaligned person re-identification. One stream focuses on spatial feature learning using global adaptive average pooling in deep convolutional neural networks. Another stream is utilized to learn the fine local features by adopting horizontal average pooling without division that depends on the pose predictor. To assess the importance ranking of all features, we also obtain the performance of every part feature and global features. Our evaluation of the proposed method on Market-1501 acquires 94.51% Rank-1 and 90.78% mAP, that on DukeMTMC-re-ID acquires 87.52% Rank-1 and 84.82% mAP, and that on CHUK03-detected acquires 69.71% Rank-1 and 71.67% mAP; these findings verify the state-of-the-art performance of the proposed method.

Visible Infrared Cross-Modality Person Re-Identification Network Based on Adaptive Pedestrian Alignment

Cross-modality person re-identification between the visible domain and infrared domain is important but extremely challenging for night-time surveillance. Besides the cross-modality discrepancies caused by different camera spectrums, visible infrared person re-identification (VI-REID) still suffers from much pedestrian misalignment as well as the variations caused by different camera viewpoints and various pedestrian pose deformations like traditional person re-identification. In this paper, we propose a multi-path adaptive pedestrian alignment network (MAPAN) to learn discriminative feature representations. The multi-path network learns features directly from the data in an end-to-end manner and aligns the pedestrians adaptively without any additional manual annotations. To alleviate the intra-modality discrepancies caused by image misalignment, we combine the aligned visible image features with the original visible image features and enhance the attention of the network towards pedestrians, resulting in significant improvements in distinguishability of the learning features. To mitigate the cross-modality discrepancies between the visible domain and the infrared domain, the discriminative features of the two modalities are mapped to the same feature embedding space, and the identity loss as well as triplet loss is incorporated as the overall loss. Extensive experiments demonstrate the superior performance of proposed method compared to the state-of-the-arts.