Pedestrian Video Research Articles

Multiple Object Detection on Surveillance Videos for Improving Accuracy Using Enhanced Faster R-CNN

Computer vision is a dynamic and rapidly evolving field within the broader domain of artificial intelligence. Within surveillance monitoring systems, one of the central tasks is object detection, which involves identifying and localizing objects of interest in video sequences to provide safety and security of the people. Detection of multiple objects is a challenging task in video sequences which interprets less accuracy and false Bounding box regression. In this paper, enhanced faster R-CNN model is proposed and trained to compute regional proposal through Convolutional layers on the different scene of the sequences in term of lighting, motion capture related to spatial analysis. These enhancements could encompass architectural improvements, novel training strategies, or the incorporation of additional data sources to improve the model's overall performance. Proposed model is experimented on pedestrian video gives an improved accuracy detection rate than single detector techniques.

A 2D image 3D reconstruction function adaptive denoising algorithm.

To address the issue of image denoising algorithms blurring image details during the denoising process, we propose an adaptive denoising algorithm for the 3D reconstruction of 2D images. This algorithm takes into account the inherent visual characteristics of human eyes and divides the image into regions based on the entropy value of each region. The background region is subject to threshold denoising, while the target region undergoes processing using an adversarial generative network. This network effectively handles 2D target images with noise and generates a 3D model of the target. The proposed algorithm aims to enhance the noise immunity of 2D images during the 3D reconstruction process and ensure that the constructed 3D target model better preserves the original image's detailed information. Through experimental testing on 2D images and real pedestrian videos contaminated with noise, our algorithm demonstrates stable preservation of image details. The reconstruction effect is evaluated in terms of noise reduction and the fidelity of the 3D model to the original target. The results show an average noise reduction exceeding 95% while effectively retaining most of the target's feature information in the original image. In summary, our proposed adaptive denoising algorithm improves the 3D reconstruction process by preserving image details that are often compromised by conventional denoising techniques. This has significant implications for enhancing image quality and maintaining target information fidelity in 3D models, providing a promising approach for addressing the challenges associated with noise reduction in 2D images during 3D reconstruction.

TrEP: Transformer-Based Evidential Prediction for Pedestrian Intention with Uncertainty

With rapid development in hardware (sensors and processors) and AI algorithms, automated driving techniques have entered the public’s daily life and achieved great success in supporting human driving performance. However, due to the high contextual variations and temporal dynamics in pedestrian behaviors, the interaction between autonomous-driving cars and pedestrians remains challenging, impeding the development of fully autonomous driving systems. This paper focuses on predicting pedestrian intention with a novel transformer-based evidential prediction (TrEP) algorithm. We develop a transformer module towards the temporal correlations among the input features within pedestrian video sequences and a deep evidential learning model to capture the AI uncertainty under scene complexities. Experimental results on three popular pedestrian intent benchmarks have verified the effectiveness of our proposed model over the state-of-the-art. The algorithm performance can be further boosted by controlling the uncertainty level. We systematically compare human disagreements with AI uncertainty to further evaluate AI performance in confusing scenes. The code is released at https://github.com/zzmonlyyou/TrEP.git.

Pedestrian Detection and Tracking System Based on Deep-SORT, YOLOv5, and New Data Association Metrics

Pedestrian tracking and detection have become critical aspects of advanced driver assistance systems (ADASs), due to their academic and commercial potential. Their objective is to locate various pedestrians in videos and assign them unique identities. The data association task is problematic, particularly when dealing with inter-pedestrian occlusion. This occurs when multiple pedestrians cross paths or move too close together, making it difficult for the system to identify and track individual pedestrians. Inaccurate tracking can lead to false alarms, missed detections, and incorrect decisions. To overcome this challenge, our paper focuses on improving data association in our pedestrian detection system’s Deep-SORT tracking algorithm, which is solved as a linear optimization problem using a newly generated cost matrix. We introduce a set of new data association cost matrices that rely on metrics such as intersections, distances, and bounding boxes. To evaluate trackers in real time, we use YOLOv5 to identify pedestrians in images. We also perform experimental evaluations on the Multiple Object Tracking 17 (MOT17) challenge dataset. The proposed cost matrices demonstrate promising results, showing an improvement in most MOT performance metrics compared to the default intersection over union (IOU) data association cost matrix.

Graph learning model for saliency detection in thermal pedestrian videos

Thermal imaging is becoming popular recently due to its all-weather and all-time capability. However, video saliency detection, an active topic in computer vision, has not been well studied for thermal videos. This paper proposes an effective saliency detection method for thermal pedestrian videos. Unlike graph-based saliency detection methods with fixed graph structures, we introduce a saliency-guided graph learning model (SGL), which integrates a multi-view graph and saliency to achieve joint graph learning and saliency diffusion. Then, we apply SGL to thermal video saliency detection by constructing the objectness descriptor based motion saliency and a multi-view spatiotemporal graph. Furthermore, a challenging thermal pedestrian video dataset IPV with 67 sequences and 1462 frames in total is built to compensate for the insufficient scale and complexity of existing datasets. Extensive experiments on a public dataset and our newly built dataset demonstrate the superior performance of our method. Compared to 11 state-of-the-art (SOTA) video saliency detection methods, the proposed approach achieves the best performance of 74.0 %, 75.1 %, and 1.8 % in F-measure, S-measure, and MAE respectively, on the IPV dataset. It also achieves better performance than SOTA on the public dataset GTFD with 69.9 %, 75.6 %, and 1.1 %, respectively.

Multi-granularity graph pooling for video-based person re-identification

The video-based person re-identification (ReID) aims to identify the given pedestrian video sequence across multiple non-overlapping cameras. To aggregate the temporal and spatial features of the video samples, the graph neural networks (GNNs) are introduced. However, existing graph-based models, like STGCN, perform the mean/max pooling on node features to obtain the graph representation, which neglect the graph topology and node importance. In this paper, we propose the graph pooling network (GPNet) to learn the multi-granularity graph representation for the video retrieval, where the graph pooling layer is implemented to downsample the graph. We construct a multi-granular graph by using node features learned from backbone, then implement multiple graph convolutional layers to perform the spatial and temporal aggregation on nodes. To downsample the graph, we propose a multi-head full attention graph pooling (MHFAPool) layer, which integrates the advantages of existing node clustering and node selection pooling methods. Specifically, MHFAPool first learns a full attention matrix for each pooled node, then obtains the principal eigenvector of the attention matrix via the power iteration algorithm, finally takes the softmax of the principal eigenvector as the aggregation coefficients. Extensive experiments demonstrate that our GPNet achieves the competitive results on four widely-used datasets, i.e., MARS, DukeMTMC-VideoReID, iLIDS-VID and PRID-2011.

Progressive unsupervised video person re-identification with accumulative motion and tracklet spatial–temporal correlation

Person re-identification (re-id) methods based on supervised learning require a large number of manually labeled samples for training, resulting in poor scalability in actual re-id applications. Existing unsupervised video person re-id methods typically focus on extracting appearance features from pedestrian videos, ignoring motion information and the fact that people usually move in groups, i.e., pedestrian spatio–temporal co-occurrence patterns. The key factor for unsupervised video person re-id is to effectively exploit both spatio–temporal clues from video sequences and cross-camera tracklet association. In this work, we propose a progressive deep learning method for unsupervised person re-id via tracklet association with spatio–temporal correlation (TASTC). Specifically, we first divide uniformly each tracklet into multiple temporally localized slices according to a time pyramid structure. Then, an initial re-id model is trained based on a two-stream convolutional architecture, and the accumulative motion context information of temporally localized slices of the tracklets per camera is learned. Finally, combining accumulative motion and tracklet spatial–temporal correlation, we associate tracklets across cameras and update the re-id model. The above steps are iterated to optimize the re-id model progressively. Experiment results demonstrate that the proposed method significantly outperforms the current state-of-the-art unsupervised video person re-identification methods on three video-based benchmark datasets, ILIDS-VID, MARS, and DukeMTMC-VideoReID.

Towards detection of abnormal event and reporting for pedestrian video surveillance

An Anomaly can be defined as an observation that does follow the normal activities with respect to others. Anomaly is polysemy that varies in a different context. For video sequences, an anomaly can be defined in terms of motion or state of motion that obtrude concerning to state and time. This work is based on Pedestrian Video Surveillance, having a normal event as a pedestrian and abnormal event as cars, skaters, bikers, wheelchairs, etc. i.e., non-pedestrian. A comprehensive study has been carried out of many existing systems for the Anomaly detection system. Based on many existing techniques we select a method called optical flow for the feature presentation of the detection of motion and Histogram of optical flow (HOOF) for the action contour of optical flow for every time instant irrespective of scale and direction of motion. As HOOF represents the motion of an object in a region, so there may be cases that the information related to the motion is not well represented. We used these two techniques along with the labelling of the frame inorderto overcome the problem of HOOF for the classifier. And gives the improved results.

Crowd Stability Analysis Based on Pedestrian Abnormal Postures

Abnormal behaviors of pedestrians in crowd gathering public places are important factors affecting the stability of crowd flow. Pedestrian abnormal postures are important manifestation of abnormal behaviors, which often leads to local turbulence, disturbance and density-speed fluctuations. It is urgent to discover the disturbance mechanism of abnormal pedestrian posture on the stability of crowd flow. This study intends to establish machine vision, kinematics, dynamic models and crowd confluence dynamic models for typical abnormal pedestrian postures in public places. We mainly use computer vision related technology based to recognize abnormal postures of pedestrians in videos, constructs a network matrix of key posture nodes, and studys the kinematics characteristics of abnormal posture nodes. Considering the number of pedestrians and the characteristics of the architectural scenes, we design a workflow to select the appropriate macro or micro dynamic model to build the crowd flow model. To validate the propuesd model, case in Shanghai Hongqiao railway station is studied.

Heterogeneous Pedestrian Traffic Flow Modelling at Signalised Intersections

Different types of pedestrians exhibit different speed characteristics and heterogeneity. In the case of mixed pedestrian flow at signalised intersections, pedestrian traffic flow modelling is important in research of the con-ditions at signalised intersections and the evaluation of services for pedestrians. The characteristics of pedestri-an traffic flow at signalised intersections were investi-gated in this study against the background of pedestrian heterogeneity using videos of pedestrians crossing three signalised intersections in Chongqing recorded in a field survey. The pedestrian walking speeds were manually ex-tracted from the videos and used as the data basis for dis-tinguishing pedestrian heterogeneity. The walking speed data of three types (young, middle-aged, and elderly) of pedestrians at different pedestrian flows were obtained by using a microsimulation software. Based on this, a pe-destrian traffic flow model for mixed-type pedestrians at signalised intersections was established and verified by actual cases. In comparison with the HCM model, the model outperforms the HCM model overall in practical applications, indicating its strong applicability and reli-ability.

Graph Learning Model for Saliency Detection in Thermal Pedestrian Videos

Graph Learning Model for Saliency Detection in Thermal Pedestrian Videos

Development of an Automated Minimum Foot Clearance Measurement System: Proof of Principle.

Over half of older adult falls are caused by tripping. Many of these trips are likely due to obstacles present on walkways that put older adults or other individuals with low foot clearance at risk. Yet, Minimum Foot Clearance (MFC) values have not been measured in real-world settings and existing methods make it difficult to do so. In this paper, we present the Minimum Foot Clearance Estimation (MFCE) system that includes a device for collecting calibrated video data from pedestrians on outdoor walkways and a computer vision algorithm for estimating MFC values for these individuals. This system is designed to be positioned at ground level next to a walkway to efficiently collect sagittal plane videos of many pedestrians’ feet, which is then processed offline to obtain MFC estimates. Five-hundred frames of video data collected from 50 different pedestrians was used to train (370 frames) and test (130 frames) a convolutional neural network. Finally, data from 10 pedestrians was analyzed manually by three raters and compared to the results of the network. The footwear detection network had an Intersection over Union of 85% and was able to find the bottom of a segmented shoe with a 3-pixel average error. Root Mean Squared (RMS) errors for the manual and automated methods for estimating MFC values were 2.32 mm, and 3.70 mm, respectively. Future work will compare the accuracy of the MFCE system to a gold standard motion capture system and the system will be used to estimate the distribution of MFC values for the population.

A Two-Stream Dynamic Pyramid Representation Model for Video-Based Person Re-Identification.

Video-based person re-identification (Re-ID) leverages rich spatio-temporal information embedded in sequence data to further improve the retrieval accuracy comparing with single image Re-ID. However, it also brings new difficulties. 1) Both spatial and temporal information should be considered simultaneously. 2) Pedestrian video data often contains redundant information and 3) suffers from data quality problems such as occlusion, background clutter. To solve the above problems, we propose a novel two-stream Dynamic Pyramid Representation Model (DPRM). DPRM mainly consists of three sub-models, i.e., Pyramidal Distribution Sampling Method (PDSM), Dynamic Pyramid Dilated Convolution (DPDC) and Pyramid Attention Pooling (PAP). PDSM is applied for more effective data pre-processing according to sequence semantic distribution. DPDC and PAP can be considered as two streams to describe the motion context and static appearance of a video sequence, respectively. By fusing the two-stream features together, we finally achieve comprehensive spatio-temporal representation. Notably, dynamic pyramid strategy is applied throughout the whole model. This strategy exploits multi-scale features under attention mechanism to maximally capture the most discriminative features and mitigate the impact of video data quality problems such as partial occlusion. Extensive experiments demonstrate the outperformance of DPRM. For instance, it achieves 83.0% mAP and 89.0% Rank-1 accuracy on MARS dataset and reaches state of the art.

Video-based person re-identification by intra-frame and inter-frame graph neural network

In the past few years, video-based person re-identification (Re-ID) have attracted growing research attention. The crucial problem for this task is how to learn robust video feature representation, which can weaken the influence of factors such as occlusion, illumination, and background etc. A great deal of previous works utilize spatio-temporal information to represent pedestrian video, but the correlations between parts of human body are ignored. In order to take advantage of the relationship among different parts, we propose a novel Intra-frame and Inter-frame Graph Neural Network (I2GNN) to solve the video-based person Re-ID task. Specifically, (1) the features from each part are treated as graph nodes from each frame; (2) the intra-frame edges are established by the correlation between different parts; (3) the inter-frame edges are constructed between the same parts across adjacent frames. I2GNN learns video representations by employing the adjacent matrix of the graph and input features to conduct graph convolution, and then adopts projection metric learning on Grassman manifold to measure the similarities between learned pedestrian features. Moreover, this paper proposes a novel occlusion-invariant term to make the part features close to their center, which can relive several uncontrolled complicated factors, such as occlusion and pose invariance. Besides, we have carried out extensive experiments on four widely used datasets: MARS, DukeMTMC-VideoReID, PRID2011, and iLIDS-VID. The experimental results demonstrate that our proposed I2GNN model is more competitive than other state-of-the-art methods.

New observations from real-world vehicle-pedestrian collisions in reducing ground related injury by controlling vehicle braking

Pedestrians suffer significant injuries from ground contact, but attempts to reduce them have been limited. A recent study using multi-body simulations showed ground contact injury may be improved by controlling vehicle braking. The aim of the study is to assesses whether controlled braking would be beneficial in real-world collisions. PC-Crash was used to reconstruct 150 freely available real-world vehicle pedestrian video collisions. Pairwise comparison of actual versus controlled braking was then performed for each case. Pedestrian injury evaluation metrics included head injury criterion HIC15 and pelvis contact force, and another metric named separation distance between the vehicle and pedestrian at the instant of ground contact was used to test the feasibility of external airbags for mitigating ground contact. Substantial head and pelvis injury reductions may be possible through the application of controlled braking. Full vehicle braking after first vehicle-pedestrian contact has little influence on overall pedestrian injury outcome. In controlled braking, an external semi-elliptic airbag with a range 2 m can prevent 83.1% ground contact (only 46.6% in actual cases) and there is a high probability (50%) that the available space to control braking is limited if the vehicle deviates from its original lane. Various kinematic explanations are presented, consistent with the principle that the vertical component of the impact on the ground can be reduced by controlling the vehicle braking. Despite the variation in real-world pedestrian collisions, controlled braking may generally provide a benefit for ground contact injuries.

MENDETEKSI SECARA OTOMATIS OBJEK GERAKAN BERDASARKAN GAUSSIAN MIXTURE MODEL MENGGUNAKAN APLIKASI MATLAB

Automatically detects moving objects on a pedestrian video, each object will be detected by counting objects that appear at the beginning of the video based on a number. In detecting this object, the model used is the Gaussian Mixture Model whose performance is very effective if applied to any area that occurs in an object motion detected in the video. In this study, the Gaussian Mixture Model is used to model the background colors of each pixel. To facilitate the detection of this object, the Matlab 2016 application is also used, this application helps facilitate detection based on the size that will be adjusted according to the object to be detected.

Frame-Guided Region-Aligned Representation for Video Person Re-Identification

Pedestrians in videos are usually in a moving state, resulting in serious spatial misalignment like scale variations and pose changes, which makes the video-based person re-identification problem more challenging. To address the above issue, in this paper, we propose a Frame-Guided Region-Aligned model (FGRA) for discriminative representation learning in two steps in an end-to-end manner. Firstly, based on a frame-guided feature learning strategy and a non-parametric alignment module, a novel alignment mechanism is proposed to extract well-aligned region features. Secondly, in order to form a sequence representation, an effective feature aggregation strategy that utilizes temporal alignment score and spatial attention is adopted to fuse region features in the temporal and spatial dimensions, respectively. Experiments are conducted on benchmark datasets to demonstrate the effectiveness of the proposed method to solve the misalignment problem and the superiority of the proposed method to the existing video-based person re-identification methods.

Unsupervised domain adaption for image-to-video person re-identification

Recently, person re-identification technique has been successfully applied to many fields, such as suspect tracking and lost human location. As video always contains more valuable information, more and more researchers focus on video based person re-identification, especially in image-to-video person re-identification (IVPR). However, most of existing IVPR models are under the supervised framework. In fact, marking enough training samples will cost numbers of labors, which limits the practical value of them. At the same time, the 2D features extracted from pedestrian image and 3D features extracted from pedestrian video are heterogeneous, which brings significant challenge for IVPR task. To effective solve the above problems, we propose an unsupervised domain adaption image-to-video person re-identification model by cross-modal feature generating and target information preserving transfer network (CMGTN). On one hand, the designed generator in our model can not only transform target domain unlabeled sample features into source domain feature space, but also can preserve target identity information. On the other hand, we eliminate the gap between pedestrian images and videos by embedding a cross-modal loss term. To evaluate the performance of our approach, we conduct extensive experiments on PRID-2011, iLIDS-VID and MARS datasets, and compare our approach with existing state-of-the-art IVPR models including four unsupervised methods and three supervised methods. Experimental results demonstrate the effectiveness of our approach.

Heterogeneous Distance Learning Based on Kernel Analysis-Synthesis Dictionary for Semi-Supervised Image to Video Person Re-Identification

Image to video person re-identification (IVPR), i.e., matching between pedestrian video and image, is an important task in practice. Although several methods have been presented for IVPR, most of these methods investigate the IVPR problem under the supervised setting, and require a large number of labeled image-video pairs for training. In this article, we study the IVPR problem under the semi-supervised setting, and propose a Kernel Analysis-synthesis Dictionary based heterogeneous Distance Learning (KADDL) approach. Specifically, KADDL first learns two pairs of kernel analysis-synthesis dictionaries from the labeled and unlabeled training image-video data in the kernel space. With the learned dictionary pairs, the heterogeneous image and video features can be transformed into coding coefficients of the same representation space, such that the gap between image and video can be bridged. Then, KADDL learns a discriminative distance metric over the transformed coding coefficients, to make the coding coefficients of positive image-video pair become similar, while those of negative image-video pair dissimilar. To make better use of the unlabeled data, we further designed a reliability-based semi-supervised strategy for KADDL. Experiments on several publicly available pedestrian sequence datasets demonstrate the effectiveness of the proposed approach.

A hybrid generative-discriminative model for abnormal event detection in surveillance video scenes

Detecting anomalous events in densely pedestrian traffic video scenes remains challenging task, due to object's tracking difficulties and noise in the scene. In this paper, a Novel Hybrid Generative-Discriminative framework is proposed for detecting and localising the anomalous events of illegal vehicles present in the scene. This paper introduces a novelty in the application of Hybrid usage of latent Dirichlet allocation (LDA) and support vector machines (SVMs) over dynamic texture at sub-region level. The proposed HLDA-SVM model consists mainly of three steps: first local binary patterns from twelve orthogonal planes (LBP-TwP) technique is applied in each spatio-temporal video patch to extract dynamic texture; then LDA technique is applied to the extracted dynamic textures for finding the latent topic distribution and finally, training is done on the distribution of topic vector for each video sequence using multi way SVM classifier. The proposed HLDA-SVM model is validated on UCSD dataset data set and is compared with mixture of dynamic texture and motion context technique. Experimental results show that the HLDA-SVM approach performs well in par with current algorithms for anomaly detection.