Pedestrian Segmentation Research Articles

DGSN: Learning how to segment pedestrians from other datasets for occluded person re-identification

In this paper, we present three major challenges in occluded person Re-Identification (ReID): different occlusions, background interference, and dataset bias. To address the first and second challenges, our approach incorporates pedestrian segmentation to distinguish between pedestrian and non-pedestrian regions. Additionally, to tackle the third challenge, we introduce an effective image enhancement method called Enhanced Random Occluding (ERO). ERO leverages other datasets with segmentation annotations to compensate for the lack of detailed annotations in ReID datasets. We compare the effectiveness of ERO with existing methods. To utilize the prior knowledge obtained by ERO, we introduce the Priori Segmentation Module (PSM) and the Domain Generalization Module (DGM). The PSM module enables learning out-of-domain prior knowledge without relying on external networks, while the DGM module transfers this knowledge to the current domain. Finally, we utilize the obtained segmentation results as attention maps for feature aggregation. ERO, PSM, and DGM together constitute the Domain Generalization Segmentation Network (DGSN). Our experimental results on occluded and holistic person ReID benchmarks demonstrate the superiority of the proposed DGSN. On the Occluded-Duke dataset, we achieved a mAP of 69.9% (+2.0%) and a rank-1 accuracy of 60.7% (+0.3%), surpassing state-of-the-art methods significantly.

Weakly Supervised Pedestrian Segmentation for Person Re-Identification

Person re-identification (RelD) is an important problem in intelligent surveillance and public security. Among all the solutions to this problem, existing mask-based methods first use a well-pretrained segmentation model to generate a foreground mask, in order to exclude the background from ReID. Then they perform the RelD task directly on the segmented pedestrian image. However, such a process requires extra datasets with pixel-level semantic labels. In this paper, we propose a Weakly Supervised Pedestrian Segmentation (WSPS) framework to produce the foreground mask directly from the RelD datasets. In contrast, our WSPS only requires image-level subject ID labels. To better utilize the pedestrian mask, we also propose the Image Synthesis Augmentation (ISA) technique to further augment the dataset. Experiments show that the features learned from our proposed framework are robust and discriminative. Compared with the baseline, the mAP of our framework is about 4.4%, 11.7%, and 4.0% higher on three widely used datasets including Market-1501, CUHK03, and MSMT17. The code will be available soon.

Butterfly network: a convolutional neural network with a new architecture for multi-scale semantic segmentation of pedestrians.

The detection of multi-scale pedestrians is one of the challenging tasks in pedestrian detection applications. Moreover, the task of small-scale pedestrian detection, i.e., accurate localization of pedestrians as low-scale target objects, can help solve the issue of occluded pedestrian detection as well. In this paper, we present a fully convolutional neural network with a new architecture and an innovative, fully detailed supervision for semantic segmentation of pedestrians. The proposed network has been named butterfly network (BF-Net) because of its architecture analogous to a butterfly. The proposed BF-Net preserves the ability of simplicity so that it can process static images with a real-time image processing rate. The sub-path blocks embedded in the architecture of the proposed BF-Net provides a higher accuracy for detecting multi-scale objective targets including the small ones. The other advantage of the proposed architecture is replacing common batch normalization with conditional one. In conclusion, the experimental results of the proposed method demonstrate that the proposed network outperform the other state-of-the-art networks such as U-Net + + , U-Net3 + , Mask-RCNN, and Deeplabv3 + for the semantic segmentation of the pedestrians.

Improving pedestrian segmentation using region proposal-based CNN semantic segmentation

Pedestrian segmentation is a critical task in computer vision, but it can be challenging for segmentation models to accurately classify pedestrians in images with challenging backgrounds and luminosity changes, as well as occlusions. This challenge is further compounded for compressed models that were designed to deal with the high computational demands of deep neural networks. To address these challenges, we propose a novel approach that integrates a region proposal-based framework into the segmentation process. To evaluate the performance of the proposed framework, we conduct experiments on the PASCAL VOC dataset, which presents challenging backgrounds. We use two different segmentation models, UNet and SqueezeUNet, to evaluate the impact of region proposals on segmentation performance. Our experiments show that the incorporation of region proposals significantly improves segmentation accuracy and reduces false positive pixels in the background, leading to better overall performance. Specifically, the SqueezeUNet model achieves a mean Intersection over Union (mIoU) of 0.682, which is a 12% improvement over the baseline SqueezeUNet model without region proposals. Similarly, the UNet model achieves a mIoU of 0.678, which is a 13% improvement over the baseline UNet model without region proposals.

Adaptive Polar-Grid Gaussian-Mixture Model for Foreground Segmentation Using Roadside LiDAR

Roadside LiDAR has become an important sensor for the detection of objects in cities, such as vehicles and pedestrians, which is due to its advantages of all-weather operation and high-ranging accuracy. In order to serve an intelligent transportation system, the efficient and accurate segmentation of vehicles and pedestrians is needed in the coverage area of the LiDAR. In this study, a roadside LiDAR was fixed on brackets on both sides of the road to obtain the point-cloud information on the urban road and the surrounding environment. A segmentation method that is based on a scanning LiDAR sensor is proposed. First, a polar grid that is based on polar coordinates is constructed to count the LiDAR rotations to obtain the original information of the angle and the distance of the point cloud, and the background point-cloud image is dynamically updated over time. By aiming at the complex urban road environment and the interference of trees and light poles in the background, an adaptive polar-grid Gaussian-mixture model (APG-GMM) that uses a point-cloud method is proposed to improve the accuracy of the foreground and background segmentation. A density-adaptive DBSCAN target-clustering algorithm is proposed, as well as a dynamic adaptive neighborhood radius, to solve the problem of the low clustering accuracy that is caused by the uneven density of point clouds that are collected by LiDAR, and to divide the point clouds in the foreground into vehicles and pedestrians. Finally, the method was tested at intersections and urban roads with dense traffic flows. The experimental results show that the proposed algorithm can segment the foreground and background well and can cluster vehicles and pedestrians while reducing the number of calculations and the time complexity.

Occlusion-Sensitive Person Re-Identification via Attribute-Based Shift Attention

Occluded person re-identification is one of the most challenging tasks in security surveillance. Most existing methods focus on extracting human body features from occluded pedestrian images. This paper prioritizes a difference between occluded and non-occluded person re-ID: When computing the similarity between a holistic pedestrian image and an occluded pedestrian image, a certain part of the human body in this holistic image can be distractive for pedestrian retrieval. To solve this problem, we propose an occluded person re-ID framework named attribute-based shift attention network (ASAN). First, unlike other methods that use off-the-shelf tools to locate pedestrian body parts in the occluded images, we design an attribute-guided occlusion-sensitive pedestrian segmentation (AOPS) module. AOPS is a weakly supervised method that leverages the semantic-level attribute annotations in person re-ID datasets. Second, guided by the pedestrian masks provided by AOPS, a shift feature adaption (SFA) module extracts the visible part of the human body feature in a part-based manner. After that, a visible region matching (VRM) algorithm is proposed to filter out the interfer-ence information in the holistic person images during the retrieval phase and further purify the representation of pedestrian features. Extensive experiments with ablation analysis demonstrate our method’s effectiveness. And the state-of-the-art results are achieved on four occluded datasets Partial-REID, Partial-iLIDS, Occluded-DukeMTMC, and Occluded REID. Moreover, the experiments on two holistic person re-ID datasets Market-1501 and DukeMTMC-reID, and a vehicle re-ID dataset VeRi-776 show that ASAN also has a good generality.

LiDAR-Based Dense Pedestrian Detection and Tracking

LiDAR-based pedestrian detection and tracking (PDT) with high-resolution sensing capability plays an important role in real-world applications such as security monitoring, human behavior analysis, and intelligent transportation. The problem of LiDAR-based PDT suffers from the complex gathering movements and the phenomenon of self- and inter-object occlusions. In this paper, the detection and tracking of dense pedestrians using three-dimensional (3D) real-measured LiDAR point clouds in surveillance applications is studied. To deal with the problem of undersegmentation of dense pedestrian point clouds, the kernel density estimation (KDE) is used for pedestrians center estimation which further leads to a pedestrian segmentation method. Three novel features are defined and used for further PDT performance improvements, which takes advantage of the pedestrians’ posture and body proportion. Finally, a new track management strategy for dense pedestrians is presented to deal with the tracking instability caused by dense pedestrians occlusion. The performance of the proposed method is validated with experiments on the KITTI dataset. The experiment shows that the proposed method can significantly increase F1 score from 0.5122 to 0.7829 compared with the STM-KDE. In addition, compared with AB3DMOT and EagerMOT, the tracking trajectories from the proposed method have the longest average survival time of 36.17 frames.

Mask-RCNN with spatial attention for pedestrian segmentation in cyber–physical systems

With the application of industrial cyber–physical systems in various fields such as transportation systems, smart cities, and medical systems, pedestrian scenarios are becoming more and more complex, which brings difficulties to pedestrian segmentation. The difficulty of pedestrian segmentation lies in the scene’s complexity where the pedestrian is located, including the pedestrian’s shooting angle, light, and obstructions, which makes it difficult to distinguish accurately. This paper proposes an S-Mask-RCNN network that integrates spatial attention mechanisms for pedestrian segmentation. Mask-RCNN uses residual neural networks in the feature extraction network, and the effect of model feature extraction is not ideal. Based on transfer learning, a spatial attention mechanism is introduced to focus more spatially on areas that need attention. The force mechanism focuses more on the areas that need attention in space. Experiments show that the S-Mask-RCNN method proposed in this paper has better performance than traditional Mask-RCNN in pedestrian segmentation. Experiments show that the S-Mask-RCNN method proposed in this paper has better performance than traditional Mask-RCNN in pedestrian segmentation, also can provide more comprehensive and practical information for the construction of cyber–physical systems.

Pedestrian instance segmentation with prior structure of semantic parts

• We propose a simple, easily interpreted semantic part branch for pedestrian instance segmentation. • We proposed a pedestrian parts annotations generation method. • Our method improves the accuracy of pedestrian instance segmentation on the baseline. • Our method does not need to introduce additional annotations. Existing pedestrian segmentation and detection methods often show a significant drop in performance when heavy occlusion and deformation happen because most approaches rely on holistic modeling. Unlike many previous deep models that directly learn a holistic detector, in this paper, we introduce a pedestrian instance segmentation method with a prior structure of semantic parts named Part Mask R-CNN. Based on pedestrian parts’ proportion structure, process the original dataset annotations and then generate parts annotations as prior. By combining the semantic part branch with other classic detection and segmentation branches, the network learns more about pedestrian instances. Besides, we get such a more accurate pedestrian instance segmentation model without any artificial annotations. By extensive evaluations on the Cityscapes dataset, the results demonstrate that the proposed method can improve approaches such as Mask R-CNN, inaccuracy on pedestrian single class instance segmentation.

Detection and Tracking of Pedestrians Using Doppler LiDAR

Pedestrian detection and tracking is necessary for autonomous vehicles and traffic management. This paper presents a novel solution to pedestrian detection and tracking for urban scenarios based on Doppler LiDAR that records both the position and velocity of the targets. The workflow consists of two stages. In the detection stage, the input point cloud is first segmented to form clusters, frame by frame. A subsequent multiple pedestrian separation process is introduced to further segment pedestrians close to each other. While a simple speed classifier is capable of extracting most of the moving pedestrians, a supervised machine learning-based classifier is adopted to detect pedestrians with insignificant radial velocity. In the tracking stage, the pedestrian’s state is estimated by a Kalman filter, which uses the speed information to estimate the pedestrian’s dynamics. Based on the similarity between the predicted and detected states of pedestrians, a greedy algorithm is adopted to associate the trajectories with the detection results. The presented detection and tracking methods are tested on two data sets collected in San Francisco, California by a mobile Doppler LiDAR system. The results of the pedestrian detection demonstrate that the proposed two-step classifier can improve the detection performance, particularly for detecting pedestrians far from the sensor. For both data sets, the use of Doppler speed information improves the F1-score and the recall by 15% to 20%. The subsequent tracking from the Kalman filter can achieve 83.9–55.3% for the multiple object tracking accuracy (MOTA), where the contribution of the speed measurements is secondary and insignificant.

Integrating Structural Symmetry and Local Homoplasy Information in Intuitionistic Fuzzy Clustering for Infrared Pedestrian Segmentation

Interferential background, boundary uncertainty, and noises are usually involved in infrared pedestrian imaging, which erect barrier for accurate segmentation. To counter the conundrum rising in these cases, we present a novel intuitionistic fuzzy clustering-based segmentation method, which integrates structural symmetry and local homoplasy information, for precise infrared pedestrian segmentation. Enlightened by the multiapplication and favorable performance of fuzzy clustering methods, intuitionistic fuzzy c-means (IFCM) is applied as the backbone of our segmentation method. The contributions of the proposed method mainly include two parts. First, motivated by potential target characteristics and tendency for clearer contour description, structural symmetry information is utilized, which is an intrinsic shape feature of objects and would be significant especially when the texture and details of the target are lost in infrared images. Further, a map that represents the probability of pixels belonging to the target is constructed in the form of ellipse symmetry region. Combined with the probability map, symmetry information is utilized to establish a novel dissimilarity function in fuzzy clustering. Second, local homoplasy information which is designed based on region similarity is introduced to suppress the intensity inhomogeneity and noises to further improve the performance of the proposed method. Finally, a dataset containing 500 infrared pedestrian images paired with corresponding pixel-wise annotation is constructed to verify segmentation effectiveness. The proposed SR-IFCM is compared with 12 state-of-the-art segmentation methods. The experimental results indicate that the proposed method outperforms the comparison methods and works better for infrared pedestrian segmentation.

Weak segmentation supervised deep neural networks for pedestrian detection

Semantic segmentation has been used successfully as a complementary information source in pedestrian detection. However, it requires accurate pixel-level semantic segmentation annotations for training, but it is extremely time-consuming to obtain these. In this work, we solve this problem by using weak segmentation masks automatically generated by depth images. This enables joint semantic segmentation and pedestrian detection with only ground truth bounding boxes for training. We show that this joint training boosts the performance of the pedestrian detector. Moreover, we show that fusing the outputs of the classification network and the generated segmentation masks leads to a further detection performance improvement. Extensive experiments have been conducted on three RGBD pedestrian datasets to demonstrate the effectiveness of our proposed method. As a byproduct, we also obtain pedestrian segmentation results of good quality, without using pixel-level segmentation annotations during training.

Instance Segmentation of Traffic Scene Based on YOLACT

As deep learning shines in the field of machine vision, this article uses the YOLACT instance segmentation algorithm to solve the detection and segmentation of pedestrians, vehicles and other objects in traffic scenes. In this paper, the backbone in the YOLACT algorithm is changed from the original resnet50 to the lightweight resnet18 to ensure that the algorithm can be deployed on Jetson-AGX-Xavier, and the processing speed can reach 9.4 frames per second, and the map can reach 81%. In NVIDIA 2070 super, the inference speed on the graphics card can reach 40 frames per second.

Infrared pedestrian segmentation algorithm based on the two-dimensional Kaniadakis entropy thresholding

Infrared pedestrian segmentation is a challenging problem in infrared image processing. Kaniadakis entropy thresholding method can segment the infrared image with long tail distribution histogram. However, it failed in images with noise and complex background. In this paper, a novel infrared pedestrian segmentation algorithm based on the two-dimensional Kaniadakis entropy thresholding is proposed to address this problem. First, in order to introduce the spatial information of pixels, the two-dimensional Kaniadakis entropy thresholding algorithm is proposed through extending the one-dimensional Kaniadakis entropy thresholding algorithm and using the two-dimensional histogram of image. The recursive formulas of the two-dimensional algorithm are also presented to improve the computation efficiency. Second, an intensity suppressed strategy is embedded in the optimal thresholds searching process of the two-dimensional Kaniadakis entropy thresholding algorithm to segment the pedestrian in infrared images with complex background. In the experiment, the proposed algorithm is compared with the state-of-the-art image segmentation methods on several infrared images with different backgrounds. The experimental results verify the effectiveness of the proposed method both qualitatively and quantitatively.

PEDESTRIAN SEGMENTATION FROM COMPLEX BACKGROUND BASED ON PREDEFINED POSE FIELDS AND PROBABILISTIC RELAXATION

The wide use of cameras enables the availability of a large amount of image frames that can be used for people counting or to monitor crowds or single individuals for security purposes. These applications require both, object detection and tracking. This task has shown to be challenging due to problems such as occlusion, deformation, motion blur, and scale variation. One alternative to perform tracking is based on the comparison of features extracted for the individual objects from the image. For this purpose, it is necessary to identify the object of interest, a human image, from the rest of the scene. This paper introduces a method to perform the separation of human bodies from images with changing backgrounds. The method is based on image segmentation, the analysis of the possible pose, and a final refinement step based on probabilistic relaxation. It is the first work we are aware that probabilistic fields computed from human pose figures are combined with an improvement step of relaxation for pedestrian segmentation. The proposed method is evaluated using different image series and the results show that it can work efficiently, but it is dependent on some parameters to be set according to the image contrast and scale. Tests show accuracies above 71%. The method performs well in other datasets, where it achieves results comparable to state-of-the-art approaches.

Moving shadow detection via binocular vision and colour clustering

A pedestrian segmentation algorithm in the presence of cast shadows is presented in this study. The novelty of this algorithm lies in the fusion of multi-view and multi-plane homographic projections of foregrounds and the use of the fused data to guide colour clustering. This brings about an advantage over the existing binocular algorithms in that it can remove cast shadows while keeping pedestrians’ body parts, which occlude shadows. Phantom detection, which is inherent with the binocular method, is also investigated. Experimental results with real-world videos have demonstrated the efficiency of this algorithm.

Cascading Scene and Viewpoint Feature Learning for Pedestrian Gender Recognition

Pedestrian gender recognition plays an important role in smart city. To effectively improve the pedestrian gender recognition performance, a new method, called cascading scene and viewpoint feature learning (CSVFL), is proposed in this article. The novelty of the proposed CSVFL lies on the joint consideration of two crucial challenges in pedestrian gender recognition, namely, scene and viewpoint variation. For that, the proposed CSVFL starts with the scene transfer (ST) scheme, followed by the viewpoint adaptation (VA) scheme in a cascading manner. Specifically, the ST scheme exploits the key pedestrian segmentation network to extract the key pedestrian masks for the subsequent key pedestrian transfer generative adversarial network, with the goal of encouraging the input pedestrian image to have the similar style to the target scene while preserving the image details of the key pedestrian as much as possible. Afterward, the obtained scene-transferred pedestrian images are fed to train the deep feature learning network with the VA scheme, in which each neuron will be enabled/disabled for different viewpoints depending on whether it has contribution on the corresponding viewpoint. Extensive experiments conducted on the commonly used pedestrian attribute data sets have demonstrated that the proposed CSVFL approach outperforms multiple recently reported pedestrian gender recognition methods.

Segmentation mask guided end-to-end person search

Person search aims to search for a target person among multiple images recorded by multiple surveillance cameras, which faces various challenges from both pedestrian detection and person re-identification. Besides the large intra-class variations owing to various illumination conditions, occlusions and varying poses, background clutters in the detected pedestrian bounding boxes further deteriorate the extracted features for each person, making them less discriminative. To tackle these problems, we develop a novel approach which guides the network with segmentation masks so that discriminative features can be learned invariant to the background clutters. We demonstrate that joint optimization of pedestrian detection, person re-identification and pedestrian segmentation enables to produce more discriminative features for pedestrian, and consequently leads to better person search performance. Extensive experiments on two widely used benchmark datasets prove the superiority of our approach. In particular, our proposed model achieves the state-of-the-art performance (86.3% mAP and 86.5% top-1 accuracy) on CUHK-SYSU dataset.

The impact of the characteristics of riverbanks settlements on the sustainability of Kali Pepe Surakarta

Surakarta City is one of the cities in Indonesia that has experienced the effects of urbanization and rapid development. This is evident from the slums along the banks of the Kali Pepe. This river intersects the city of Solo and stretches for about five kilometers from Sangkrah Village to Gilingan Village. This river used to have an important role in accommodating urban mobility. In 2016, Kali Pepe received attention for its pedestrian segments, parks, and riverbank development. This study describes the settlement characteristics along the riverbanks in relation to the sustainability of the Kali Pepe. Descriptive-qualitative methods accompanied by in-depth interviews were used to collect data. Moreover, the study exploration of field conditions was recorded in the form of sketches and photographs. The results of the study show two prominent potentials from the characteristics of the Kali Pepe riverbanks in relation to its sustainability, i.e., (1) nature characteristics that can be developed through integrated water transportation tourism; (2) The emergence of home-industry that is backed by the potential of employment in the informal sector and accommodated riverbank tourism.

Human skeleton mutual learning for person re-identification

Person re-identification refers to matching people across non-overlapping camera views on different locations and at different times. In the case of changes in perspective, light, background, veil, and person's clothing, traditional method cannot achieve person recognition effectively and reliably. In this paper, we propose a novel biometric metric learning method named Human Skeleton Mutual Learning person re-identification (HSMLP-Reid). The purpose of HSML person re-identification method (HSMLP-Reid) largely aims to use the new pedestrian local segmentation method proposed in this paper combined with the global skeleton information to solve the influence of background and local posture change. Firstly, bottom-up method is used to estimate the pedestrian posture and skeleton and the joint points of the pedestrians will be marked in this process. A new local segmentation method proposed in this paper named joint segmentation is used to locally segment pedestrians and perform local block matching. Furthermore, we learn the global skeleton information by defined joint distances from the pedestrian 2D skeleton estimation by the bottom-up method and use global skeleton information to global skeleton matching. Finally, we use local match and global skeleton match for mutual learning. Local match based on pedestrian nodes and global skeleton match based on pedestrian skeleton are based on biometrics. We learn the classification loss and metric learning loss to train model. Metric loss includes global skeletal distance and local block metric distance. Extensive experimental results on the large-scale Market1501, CUHK03 and CUHK-SYSU data sets demonstrate that the proposed method achieves consistently superior performance and outperforms most of the state-of-the-art methods.