Caltech Pedestrian Dataset Research Articles

Towards faster yet accurate video prediction for resource-constrained platforms

Video prediction has emerged as a critical task in the field of computer vision. While conventional deep learning-based video prediction models excel in accuracy, they often impose significant computational overhead. Moreover, they tend to disregard crucial factors like inference latency and memory consumption, which may render them unsuitable for real-time video prediction applications. This issue is further exacerbated by the necessity to deploy many of these applications on resource-limited embedded devices. In this study, we introduce an innovative faster yet accurate video prediction (FAVP) model designed to mitigate inference latency and memory footprint, all the while maintaining predictive accuracy. Our model employs an encoder-convertor-decoder architecture with a novel multi-layer convertor. Each layer within the convertor uses consistent kernel sizes to precisely capture dimensional features, while varying kernel sizes between layers effectively capture both local and global features. This combination of uniformity within layers and diversity between layers enhances the model’s ability to grasp dynamic changes. Furthermore, we demonstrate that replacing conventional large-kernel convolutions with involutions significantly trims the model’s parameter count and inference latency, without compromising prediction accuracy. Through comprehensive experiments conducted on resource-adequate x86 platforms, utilizing KTH, TrafficBJ, Human3.6M, Moving MNIST, and KTH-Enhanced datasets, we showcase our model’s exceptional prediction performance, particularly in terms of inference latency. Additionally, we evaluate the model on resource-constrained NVIDIA Jetson Nano platform using KITTI & CalTech Pedestrian dataset. The results underscore our model’s superior inference speed and its competence in meeting real-time video prediction requirements, even under stringent resource limitations.

Enhanced YOLOX with United Attention Head for Road Detetion When Driving

Object detection plays a crucial role in autonomous driving assistance systems. It requires high accuracy for prediction, a small size for deployment on mobile devices, and real-time inference speed to ensure safety. In this paper, we present a compact and efficient algorithm called YOLOX with United Attention Head (UAH-YOLOX) for detection in autonomous driving scenarios. By replacing the backbone network with GhostNet for feature extraction, the model reduces the number of parameters and computational complexity. By adding a united attention head before the YOLO head, the model effectively detects the scale, position, and contour features of targets. In particular, an attention module called Spatial Self-Attention is designed to extract spatial location information, demonstrating great potential in detection. In our network, the IOU Loss (Intersection of Union) has been replaced with CIOU Loss (Complete Intersection of Union). Further experiments demonstrate the effectiveness of our proposed methods on the BDD100k dataset and the Caltech Pedestrian dataset. UAH-YOLOX achieves state-of-the-art results by improving the detection accuracy of the BDD100k dataset by 1.70% and increasing processing speed by 3.37 frames per second (FPS). Visualization provides specific examples in various scenarios.

A spatiotemporal motion prediction network based on multi-level feature disentanglement

The prediction task is significantly challenged by the intricate scene information and motion variations present in spatiotemporal data. Existing prediction methods struggle to accurately forecast long-term outcomes, particularly for transient motions characterized by notable trends, such as hand lifts, jumps, or vehicle turns. To address these challenges, we introduce a Spatiotemporal Motion Prediction Network based on Multi-level Feature Disentanglement (FDPNet). The model delineates spatiotemporal prediction into two distinct stages: feature disentanglement and motion prediction. We first devise a Multi-level Feature Disentanglement (MFD) model to disentangle the multilayer features of motion within the temporal sequence, encompassing period, trend, and residual components. This disentanglement is based on disentangling spatiotemporal coupling, enabling the network to comprehensively grasp the genuine laws governing motion in the spatiotemporal evolution process. Second, to enhance the prediction accuracy of the network over extended periods, we introduce the Motion Differential Self-Attention LSTM unit (MDSA-LSTM). This unit employs differential operations to extract inter-frame motion trends, elevating the network's proficiency in capturing spatiotemporal correlations over long distances through an enhanced self-attention mechanism. FDPNet attains state-of-the-art performance on the Moving MNIST, UCF101, KITTI, and Caltech pedestrian datasets. These outcomes substantiate the substantial potential of this research within the realm of spatiotemporal prediction.

Research on Pedestrian Detection Based on the Multi-Scale and Feature-Enhancement Model

Pedestrian detection represents one of the critical tasks of computer vision; however, detecting pedestrians can be compromised by problems such as the various scale of pedestrian features and cluttered background, which can easily cause a loss of accuracy. Therefore, we propose a pedestrian detection method based on the FCOS network. Firstly, we designed a feature enhancement module to ensure that effective high-level semantics are obtained while preserving the detailed features of pedestrians. Secondly, we defined a key-center region judgment to reduce the interference of background information on pedestrian feature extraction. By testing on the Caltech pedestrian dataset, the AP value is improved from 87.36% to 94.16%. The results of the comparison experiment illustrate that the model proposed in this paper can significantly increase the accuracy.

Lightweight Pedestrian Detection Based on Feature Multiplexed Residual Network

As an important part of autonomous driving intelligence perception, pedestrian detection has high requirements for parameter size, real-time, and model performance. Firstly, a novel multiplexed connection residual block is proposed to construct the lightweight network for improving the ability to extract pedestrian features. Secondly, the lightweight scalable attention module is investigated to expand the local perceptual field of the model based on dilated convolution that can maintain the most important feature channels. Finally, we verify the proposed model on the Caltech pedestrian dataset and BDD 100 K datasets. The results show that the proposed method is superior to existing lightweight pedestrian detection methods in terms of model size and detection performance.

Future video frame prediction based on generative motion-assistant discriminative network

With the continuous development of deep learning, video frame prediction has become a hotspot in the field of computer vision due to its wide range of applications in anomaly detection, robot decision-making, weather forecasting, and autonomous driving. Although current video frame prediction methods have made remarkable progress, the majority of them directly generate prediction frames by extracting potential spatial distribution patterns from the video data. They lack spatiotemporal information modeling, which leads to high latency, ambiguity, and unrealistic results. In this work, we propose an end-to-end video prediction network model (Generative Differential-Assisted Discriminative Network, abbreviated as GDDNet). It combines the advantages of the difference generation method to extract short-term variations from the image and attention mechanisms to recall global contextual motion information. Furthermore, the differential attention mechanism (DAM) module can guide the model to allocate attention resources more efficiently. These strategies considerably improve the model’s ability to represent motion features in video frames. To further optimize the prediction effect, we introduce adversarial training to enhance the clarity and authenticity of the video frames. In order to ensure the consistency of spatiotemporal distribution between predicted and real frames, we introduce a sequential frame discriminator. Experimental results on the KITTI, UCF-101, and Caltech pedestrian datasets demonstrate the effectiveness of the GDDNet and compare it to the state-of-the-art model. Multi-frame prediction and ablation experiments show that our proposed model not only improves the quality of predictions, but also provides a more flexible prediction framework.

Small-Scale and Occluded Pedestrian Detection Using Multi Mapping Feature Extraction Function and Modified Soft-NMS.

In autonomous driving and Intelligent transportation systems, pedestrian detection is vital in reducing traffic accidents. However, detecting small-scale and occluded pedestrians is challenging due to the ineffective utilization of the low-feature content of small-scale objects. The main reasons behind this are the stochastic nature of weight initialization and the greedy nature of nonmaximum suppression. To overcome the aforesaid issues, this work proposes a multifocus feature extractor module by fusing feature maps extracted from the Gaussian and Xavier mapping function to enhance the effective receptive field. We also employ a focused attention feature selection on a higher layer feature map of the single shot detector (SSD) region proposal module to blend with its low-layer feature to tackle the vanishing of the feature detail due to convolution and pooling operation. In addition, this work proposes a decaying nonmaximum suppression function considering score and Intersection Over Union (IOU) parameters to tackle high miss rates caused by greedy nonmaximum suppression used by SSD. Extensive experiments have been conducted on the Caltech pedestrian dataset with the original annotations and the improved annotations. Experimental results demonstrate the effectiveness of the proposed method, particularly for small and occluded pedestrians.

Real-time social distance measurement and face mask detection in public transportation systems during the COVID-19 pandemic and post-pandemic Era: Theoretical approach and case study in Italy

Due to its remarkable learning ability and benefits in several areas of real-life, deep learning-based applications have recovered to be a research topic of great importance in the last few years. This article presents a method devoted to guaranteeing safety conditions in public transportation systems (PTS) during the COVID-19 pandemic and post-pandemic era. The paper describes a viable real-time model based on deep learning for monitoring social distance between users and detecting face masks in stop areas and inside vehicles of public transportation systems. Detections are made using the deep learning approach and YOLOv3 algorithm. The safety rule violations are represented by red bounding boxes and red circles in a bird’s eye view as output of the video surveillance analysis. The datasets used to train the neural network are the “Caltech Pedestrian Dataset” and the “COVID-19 Medical Face Mask Detection Dataset”. Metrics, such Loss Accuracy, and Precision, obtained in the testing process of the neural network were used to evaluate the performance of the model in detecting users and face masks. The proposed method was recently tested in the Public Transportation System of the Municipality of Piazza Armerina (Italy). The results show a significant reliability of the method in detecting real-time interactions between users of the PTS in terms of over-time variations in their mutual distancing, as well as in recognising cases of violation of the imposed social distancing and FFP2 face mask use.

MS-ML-SNYOLOv3: A robust lightweight modification of SqueezeNet based YOLOv3 for pedestrian detection

Pedestrian detection is a vital problem area in computer vision applications such as surveillance, automotive safety, and robotics. To continue the progress made in this field, a deep learning based approach is given in this work. A modified lightweight YOLOv3 is proposed, which is named as MultiScale-MultiLevel-SqueezeNetYOLOv3 (MS-ML-SNYOLOv3). An improved base network forms the basis of the proposed YOLOv3. Two Squeeze & Expand blocks are embodied in SqueezeNet at specific levels of the network to extract a hierarchical feature representation. An addition of a larger receptive field is made in the Expand section as well to encapsulate the feature information at various scales. The proposed method is compared against state-of-the-art detection methods, Faster RCNN, YOLOv2, YOLOv3, and SSD. The comparison also includes various base networks for YOLOv2 and YOLOv3. The performance metrics used for comparison are Detection Error Trade-Off, and Precision-Recall Curve. The proposed method is evaluated on benchmark datasets INRIA and Caltech Pedestrian datasets. A statistical analysis, Friedman Test is conducted to verify the application of MS-ML-SNYOLOv3 in real world data.

Spatial-Temporal Wavelet Analysis Video Prediction Based on DifferentialAttention Mechanism

Inspired by the visual process of human, a video prediction algorithm based on spatial-temporal wavelet analysis and differential attention is proposed to solve the problem that it is difficult to accurately predict the details of spatial structure information and the dependence of temporal motion. Firstly, the spatial-temporal wavelet analysis module is used to decompose the video in multiple frequencies, so as to enhance the model’s ability to understand high-frequency details and procedural motion. Then, the differential attention mechanism guides the model to allocate attention resources more efficiently and reasonably, and improves the expression ability of instantaneous motion. Experimental results on the KTH, Cityscapes, BAIR, KITTI, Caltech Pedestrian datasets show the proposed algorithm achieves better results than the existing algorithms in the quantitative evaluation metrics of PSNR, SSIM and LPIPS. Meanwhile, the visualization results also show that the prediction of the proposed algorithm is clearer.

Occluded Pedestrian Detection Techniques by Deformable Attention-Guided Network (DAGN)

Although many deep-learning-based methods have achieved considerable detection performance for pedestrians with high visibility, their overall performances are still far from satisfactory, especially when heavily occluded instances are included. In this research, we have developed a novel pedestrian detector using a deformable attention-guided network (DAGN). Considering that pedestrians may be deformed with occlusions or under diverse poses, we have designed a deformable convolution with an attention module (DCAM) to sample from non-rigid locations, and obtained the attention feature map by aggregating global context information. Furthermore, the loss function was optimized to get accurate detection bounding boxes, by adopting complete-IoU loss for regression, and the distance IoU-NMS was used to refine the predicted boxes. Finally, a preprocessing technique based on tone mapping was applied to cope with the low visibility cases due to poor illumination. Extensive evaluations were conducted on three popular traffic datasets. Our method could decrease the log-average miss rate (MR−2) by 12.44% and 7.8%, respectively, for the heavy occlusion and overall cases, when compared to the published state-of-the-art results of the Caltech pedestrian dataset. Of the CityPersons and EuroCity Persons datasets, our proposed method outperformed the current best results by about 5% in MR−2 for the heavy occlusion cases.

A High‐Dimensional Video Sequence Completion Method with Traffic Data Completion Generative Adversarial Networks

The lack of traffic data is a bottleneck restricting the development of Intelligent Transportation Systems (ITS). Most existing traffic data completion methods aim at low‐dimensional data, which cannot cope with high‐dimensional video data. Therefore, this paper proposes a traffic data complete generation adversarial network (TDC‐GAN) model to solve the problem of missing frames in traffic video. Based on the Feature Pyramid Network (FPN), we designed a multiscale semantic information extraction model, which employs a convolution mechanism to mine informative features from high‐dimensional data. Moreover, by constructing a discriminator model with global and local branch networks, the temporal and spatial information are captured to ensure the time‐space consistency of consecutive frames. Finally, the TDC‐GAN model performs single‐frame and multiframe completion experiments on the Caltech pedestrian dataset and KITTI dataset. The results show that the proposed model can complete the corresponding missing frames in the video sequences and achieve a good performance in quantitative comparative analysis.

Dense-RefineDet for Traffic Sign Detection and Classification.

Detecting and classifying real-life small traffic signs from large input images is difficult due to their occupying fewer pixels relative to larger targets. To address this challenge, we proposed a deep-learning-based model (Dense-RefineDet) that applies a single-shot, object-detection framework (RefineDet) to maintain a suitable accuracy–speed trade-off. We constructed a dense connection-related transfer-connection block to combine high-level feature layers with low-level feature layers to optimize the use of the higher layers to obtain additional contextual information. Additionally, we presented an anchor-design method to provide suitable anchors for detecting small traffic signs. Experiments using the Tsinghua-Tencent 100K dataset demonstrated that Dense-RefineDet achieved competitive accuracy at high-speed detection (0.13 s/frame) of small-, medium-, and large-scale traffic signs (recall: 84.3%, 95.2%, and 92.6%; precision: 83.9%, 95.6%, and 94.0%). Moreover, experiments using the Caltech pedestrian dataset indicated that the miss rate of Dense-RefineDet was 54.03% (pedestrian height > 20 pixels), which outperformed other state-of-the-art methods.

Improved pedestrian detection with peer AdaBoost cascade

Focusing on data imbalance and intraclass variation, an improved pedestrian detection with a cascade of complex peer AdaBoost classifiers is proposed. The series of the AdaBoost classifiers are learned greedily, along with negative example mining. The complexity of classifiers in the cascade is not limited, so more negative examples are used for training. Furthermore, the cascade becomes an ensemble of strong peer classifiers, which treats intraclass variation. To locally train the AdaBoost classifiers with a high detection rate, a refining strategy is used to discard the hardest negative training examples rather than decreasing their thresholds. Using the aggregate channel feature (ACF), the method achieves miss rates of 35% and 14% on the Caltech pedestrian benchmark and Inria pedestrian dataset, respectively, which are lower than that of increasingly complex AdaBoost classifiers, i.e., 44% and 17%, respectively. Using deep features extracted by the region proposal network (RPN), the method achieves a miss rate of 10.06% on the Caltech pedestrian benchmark, which is also lower than 10.53% from the increasingly complex cascade. This study shows that the proposed method can use more negative examples to train the pedestrian detector. It outperforms the existing cascade of increasingly complex classifiers.

Hybrid channel based pedestrian detection

Pedestrian detection has achieved great improvements with the help of Convolutional Neural Networks (CNNs). CNN can learn high-level features from input images, but the insufficient spatial resolution of CNN feature channels (feature maps) may cause a loss of information, which is harmful especially to small instances. In this paper, we propose a new pedestrian detection framework, which extends the successful RPN+BF framework to combine handcrafted features and CNN features. RoI-pooling is used to extract features from both handcrafted channels (e.g. HOG+LUV, CheckerBoards or RotatedFilters) and CNN channels. Since handcrafted channels always have higher spatial resolution than CNN channels, we apply RoI-pooling with larger output resolution to handcrafted channels to keep more detailed information. Our ablation experiments show that the developed handcrafted features can reach better detection accuracy than the CNN features extracted from the VGG-16 net, and a performance gain can be achieved by combining them. Experimental results on Caltech pedestrian dataset with the original annotations and the improved annotations demonstrate the effectiveness of the proposed approach. When using a more advanced RPN in our framework, our approach can be further improved and get competitive results on both benchmarks.

An Improved Faster R-CNN Pedestrian Detection Algorithm Based on Feature Fusion and Context Analysis

Considering the multi-scale and occlusion problem of pedestrian detection in natural scenes, we propose an improved Faster R-CNN pedestrian detection algorithm based on feature fusion and context analysis (FCF R-CNN). We design a feature fusion method of progressive cascade on VGG16 network, and add LRN to speed up the convergence of the network. The improved feature extraction network enables our model to generate high-resolution feature maps containing rich, detailed and semantic information. We also adjust the RPN parameters to improve the proposal efficiency. In addition, we add a multi-layer iterative LSTM module to the detection model, which uses LSTM’s memory ability to extract the global context information of the candidate boxes. This method only needs the feature map of the image itself as input, which highlights useful context information and enables the model to generate more accurate candidate boxes containing potential pedestrians. Our method performs better than existing methods in detecting small-size and occluded pedestrians, and has strong robustness in challenging scenes. Our method achieves competitive results in both accuracy and speed on Caltech pedestrian dataset, achieving a LAMR value of 36.75% and a runtime of 0.20 seconds per image. The validity of the algorithm has been proved.

Integrated feature set using aggregate channel features and histogram of sparse codes for human detection

The scientific community witnessed revolutionary changes with algorithms and data sets aiming for precise human detection from images and videos which are largely driven by the quality of features extracted. Regardless of the labyrinth of the existing detectors, featured human detection with near accuracy from complicated real-time data sets remains a major challenge. Here we propose an improved feature set by merging the fast and accurate aggregate channel features (ACF) and the data specific dictionary learned histogram of sparse codes (HSC) for human detection. This integrated feature set efficiently fuses the first-order information from the histogram of oriented gradient channels embedded in the ACF detector and the data specific intelligence contained in the HSC channels. The proposed detector outperforms the state-of-the-art ACF detector in terms of miss rate and average precision on challenging datasets. It is worth to be noted that there is a decrease with the miss rate by a factor of 13% and 5% for INRIA and Caltech pedestrian datasets respectively in comparison with baseline detector. Along with the detection of more instances, our detector reduced the number of false positives compared to other existing detectors. Although further modifications are warranted, our proposed detector could produce a tangible and palpable response with human detection in the vast arena of computer vision.

JCS-Net: Joint Classification and Super-Resolution Network for Small-Scale Pedestrian Detection in Surveillance Images

While convolutional neural network (CNN)-based pedestrian detection methods have proven to be successful in various applications, detecting small-scale pedestrians from surveillance images is still challenging. The major reason is that the small-scale pedestrians lack much detailed information compared to the large-scale pedestrians. To solve this problem, we propose to utilize the relationship between the large-scale pedestrians and the corresponding small-scale pedestrians to help recover the detailed information of the small-scale pedestrians, thus improving the performance of detecting small-scale pedestrians. Specifically, a unified network (called JCS-Net) is proposed for small-scale pedestrian detection, which integrates the classification task and the super-resolution task in a unified framework. As a result, the super-resolution and classification are fully engaged, and the super-resolution sub-network can recover some useful detailed information for the subsequent classification. Based on HOG+LUV and JCS-Net, multi-layer channel features (MCF) are constructed to train the detector. The experimental results on the Caltech pedestrian dataset and the KITTI benchmark demonstrate the effectiveness of the proposed method. To further enhance the detection, multi-scale MCF based on JCS-Net for pedestrian detection is also proposed, which achieves the state-of-the-art performance.

Small-Scale Pedestrian Detection Based on Deep Neural Network

Pedestrian detection is a crucial component for intelligent transport system and advanced driver assistance system. In recent years, pedestrian detection methods have achieved higher accuracy. However, the existing algorithms are insufficient for small-scale pedestrian detection that is relatively far from cameras in practical applications. In this paper, we propose a novel deep small-scale sense network (termed SSN) for small-scale pedestrian detection. The proposed architecture could generate some proposal regions which are more effective to detect small-scale pedestrians. Furthermore, we design a novel loss function based on cross entropy loss to increase the loss contribution from hard-to-detect small-scale pedestrians. In addition, a novel evaluation metric is introduced, which can measure the location precision of the pedestrian detection methods. In addition an Asian pedestrian detection dataset named VIP pedestrian dataset is constructed from various road condition data. Our method achieves good detection performance on Caltech pedestrian dataset and our VIP pedestrian dataset.

Pedestrian Detection Using Pixel Difference Matrix Projection

Pedestrian detection in the embedded system, such as video surveillance equipment, usually involves low-resolution pedestrian samples and requires a low computational cost. Many pedestrian detectors rely on a large feature pool and suffer in their efficiency and performance for real-time monitoring. In this paper, a set of light-weight features is proposed to enhance the pedestrian detection performance when a small-medium scale of training data with low-resolution images is available. To address this issue, a difference matrix projection (DMP) is developed to compute aggregated multi-oriented pixel differences using global matrix operations. Both the pixel differences and aggregation are computed using global matrix projection to avoid the laborious iterative operations. We tested our method on the INRIA, Daimler Chrysler classification (Daimler-CB), NICTA, and Caltech Pedestrian datasets. The experiments on these benchmark data sets show encouraging results in terms of detection performance, particularly for image datasets with low-resolution pedestrians.