Person Detection Research Articles

STMixer: A One-Stage Sparse Action Detector.

Traditional video action detectors typically adopt the two-stage pipeline, where a person detector is first employed to generate actor boxes and then 3D RoIAlign is used to extract actor-specific features for action recognition. This detection paradigm requires multi-stage training and inference, and the feature sampling is only constrained inside the box, failing to effectively leverage richer context information outside. Recently, several query-based action detectors have been proposed to predict action instances in an end-to-end manner. However, they still lack adaptability in feature sampling and decoding, thus suffering from the issues of inferior performance or slower convergence. In this paper, we propose two core designs for a more flexible one-stage sparse action detector. First, we present a query-based adaptive feature sampling module, which endows the detector with the flexibility of mining a group of discriminative features from the entire spatio-temporal domain. Second, we devise a decoupled feature mixing module, which dynamically attends to and mixes video features along the spatial and temporal dimensions respectively for better feature decoding. Based on these designs, we instantiate two detection pipelines, that is, STMixer-K for keyframe action detection and STMixer-T for action tubelet detection. Without bells and whistles, our STMixer detectors obtain the state-of-the-art results on five challenging spatio-temporal action detection benchmarks for keyframe action detection or action tube detection.

Enhancing robustness of person detection: A universal defense filter against adversarial patch attacks

Person detection is one of the most popular object detection applications, and has been widely used in safety-critical systems such as autonomous driving. However, recent studies have revealed that person detectors are vulnerable to physically adversarial patch attacks and may suffer detection failure. Data-side defense is an effective approach to address this issue, owing to its low computational cost and ease of deployment. However, existing data-side defenses have limited effectiveness in resisting adaptive patch attacks. To overcome this challenge, we propose a new data-side defense, called Universal Defense Filter (UDFilter). UDFilter covers the input images with an equal-size defense filter to weaken the negative impact of adversarial patches. The defense filter is generated using a self-adaptive learning algorithm that facilitates iterative competition between adversarial patch and defense filter, thus bolstering UDFilter’s ability to defense adaptive attacks. Furthermore, to maintain the clean performance, we propose a plug-and-play Joint Detection Strategy (JDS) during the model testing phase. Extensive experiments have shown that UDFilter can significantly enhance robustness of person detection against adversarial patch attacks. Moreover, UDFilter does not result in a discernible reduction in the model’s clean performance.

Speedy Vision-based Human Detection Using Lightweight Deep Learning Network

Person detection plays a role as the initial system of video surveillance analysis with various implementations, such as activity analysis, person re-id, behavior analysis, and tracking analysis. The demand for efficient models drives a deep learning architecture with a superficial structure that can operate in real-time. You look only once (YOLO) object detection has been presented as an accurate detector that can operate in real-time. The speed limitation, huge computation cost, and abundant parameters still leave vital issues to improve the efficiency of this architecture. Lightweight human detection is proposed by utilizing the YOLOv5n framework. Modifying layer depth promotes a detection system that can operate fast and without stuttering. As a result, the proposed detector has satisfactory performance and is competitive with existing models. It achieves a mAP of 45.2%, closely competing with other person detectors. Additionally, it can run fast without stumbling at 26 frames per second. The detector's speed offers the advantage of this work that it can be feasibly implemented on a cpu device without a graphics accelerator.

MTP-YOLO: You Only Look Once Based Maritime Tiny Person Detector for Emergency Rescue

Tiny person detection based on computer vision technology is critical for maritime emergency rescue. However, humans appear very small on the vast sea surface, and this poses a huge challenge in identifying them. In this study, a single-stage tiny person detector, namely the “You only look once”-based Maritime Tiny Person detector (MTP-YOLO), is proposed for detecting maritime tiny persons. Specifically, we designed the cross-stage partial layer with two convolutions Efficient Layer Aggregation Networks (C2fELAN) by drawing on the Generalized Efficient Layer Aggregation Networks (GELAN) of the latest YOLOv9, which preserves the key features of a tiny person during the calculations. Meanwhile, in order to accurately detect tiny persons in complex backgrounds, we adopted a Multi-level Cascaded Enhanced Convolutional Block Attention Module (MCE-CBAM) to make the network attach importance to the area where the object is located. Finally, by analyzing the sensitivity of tiny objects to position and scale deviation, we proposed a new object position regression cost function called Weighted Efficient Intersection over Union (W-EIoU) Loss. We verified our proposed MTP-YOLO on the TinyPersonv2 dataset. All these results confirm that this method significantly improves model performance while maintaining a low number of parameters and can therefore be applied to maritime emergency rescue missions.

A Tracking-Based Two-Stage Framework for Spatio-Temporal Action Detection

Spatio-temporal action detection (STAD) is a task receiving widespread attention and has numerous application scenarios, such as video surveillance and smart education. Current studies follow a localization-based two-stage detection paradigm, which exploits a person detector for action localization and a feature processing model with a classifier for action classification. However, many issues occur due to the imbalance between task settings and model complexity in STAD. Firstly, the model complexity of heavy offline person detectors adds to the inference overhead. Secondly, the frame-level actor proposals are incompatible with the video-level feature aggregation and Region-of-Interest feature pooling in action classification, which limits the detection performance under diverse action motions and results in low detection accuracy. In this paper, we propose a tracking-based two-stage spatio-temporal action detection framework called TrAD. The key idea of TrAD is to build video-level consistency and reduce model complexity in our STAD framework by generating action track proposals among multiple video frames instead of actor proposals in a single frame. In particular, we utilize tailored tracking to simulate the behavior of human cognitive actions and used the captured motion trajectories as video-level proposals. We then integrate a proposal scaling method and a feature aggregation module into action classification to enhance feature pooling for detected tracks. Evaluations in the AVA dataset demonstrate that TrAD achieves SOTA performance with 29.7 mAP, while also facilitating a 58% reduction in overall computation compared to SlowFast.

CMOT: A cross-modality transformer for RGB-D fusion in person re-identification with online learning capabilities

Person re-identification (reID) is a crucial aspect of intelligent surveillance systems, enabling the recognition of individuals across non-overlapping camera views. As compared to the RGB modality, RGB-D based reID has the potential to achieve robust and high performance by leveraging rich complementary features of both modalities, making it applicable in various occluded scenarios. However, current multimodal reID approaches often rely on late fusion or feature-level fusion techniques to combine multiple modalities, which limits their ability to proficiently exploit complementary visual and depth-related semantic information and capture complex interactions between unimodal features. To address these limitations, this paper introduces a cross-modality online transformer (CMOT) for RGB-D based person reID with online learning capabilities, which effectively utilizes both RGB and depth modalities for the extraction of spatio-temporal features and fuses across modalities. Our CMOT is composed of three main components: (1) a hypothesis generation module based on a person detector and tracker, (2) dual-stream feature extractors via convolutional neural networks (CNNs), (3) and a fusion transformer based on a self-attention-driven self-attentive modality refinement module (SAMR) and a cross-attention-driven cross-attentive modality interaction module (CAMI) to refine and fuse RGB-D complementary features extracted from the dual-stream, RGB and depth stream, CNNs. Additionally, we introduce a bottleneck enhancement feed-forward block to enhance the model’s representation capability within SAMR and CAMI, significantly reducing parameters and computations compared to the traditional feed-forward network. Moreover, we design the triplet loss function with distance measuring ability for incorporating online learning and finally CMOT works as a few-shot network for reID. Experimental results on three RGB-D person re-identification datasets, namely BIWI RGBD-ID, RobotPKU RGBD-ID, and TVPR2, demonstrate the effectiveness and robustness of CMOT.

Application of raycast method for person geolocalization and distance determination using UAV images in Real-World land search and rescue scenarios

People enjoy spending time in the wilderness for numerous reasons. However, they occasionally get lost or injured, and their survival depends on being efficiently found and rescued in the shortest possible time. A search and rescue operation (SAR) is launched after the accident is reported, and all possible resources are activated. The inclusion of drones in SAR operations has enabled the use of computer vision methods to detect persons in aerial imagery automatically. When searching by drone, preference is given to oblique photographs that cover a larger area within a single image, reducing the search time. Unlike vertical photographs, oblique photographs include a significant scale change, making it challenging to locate a person in the real world and determine their distance from the drone. In order to solve this problem, encouraged by our previous successful simulations, we explored the possibility of applying the raycast method for person geolocalization and distance determination for use in real-world scenarios. In this paper, we propose a system able to precisely geolocate persons automatically detected in offline processed images recorded during the SAR mission. After a series of experiments on terrains of different configurations and complexity, using a custom-made 3D terrain generator and raycaster, along with a deep neural network-based person detector trained on our custom dataset, we defined a method for geolocating detected person based on raycast, which allows using low-cost commercial drones with a monocular camera and no Real-Time Kinematic module while enabling laser rangefinder emulation during offline image analysis. Our person geolocating method overcomes the problems faced by previous methods and, using a single flight sequence with only 4 consecutive detections, significantly outperforms the previous best results, with reliability of 42,85% (geolocating error of 0.7 m on recording from a 30 m height). Also, a short time of only 247 s enables offline processing of data recorded during a 21-minute drone flight covering approximately an area of 10 ha, proving that the proposed method can be effectively used in actual SAR missions. We also proposed a new evaluation metric (ErrDist) for person geolocalization and provided recommendations for using the proposed system for person detection and geolocation in real-world scenarios.

SMARTBOX PENERIMA PAKET BELANJA ONLINE

SMARTBOX PENERIMA PAKET BELANJA ONLINE

SMPR: Single-stage multi-person pose regression

Existing multi-person pose estimators can be roughly divided into two-stage approaches (top-down and bottom-up approaches) and one-stage approaches. The two-stage methods either suffer high computational redundancy for additional person detectors or group keypoints heuristically after predicting all the instance-free keypoints. The recently proposed single-stage methods do not rely on the above two extra stages but have lower performance than the latest bottom-up approaches. In this work, a novel single-stage multi-person pose regression, termed SMPR, is presented. It follows the paradigm of dense prediction and predicts instance-aware keypoints from every location. Besides feature aggregation, we propose better strategies to define positive pose hypotheses for training which all play an important role in dense pose estimation. The network also learns the scores of estimated poses. The pose scoring strategy further improves the pose estimation performance by prioritizing superior poses during non-maximum suppression (NMS). We show that our method not only outperforms existing single-stage methods but also be competitive with the latest bottom-up methods, with 70.2 AP and 77.5 AP75 on the COCO test-dev pose benchmark. The code is available at https://github.com/cmdi-dlut/SMPR .

Using Frequency Attention to Make Adversarial Patch Powerful Against Person Detector

Deep neural networks (DNNs) are vulnerable to adversarial attacks. In particular, object detectors may be attacked by applying a particular adversarial patch to the image. However, because the patch will lose information during the shrinking process, most existing approaches that employ adversarial patches to attack object detectors would diminish the attack success rate on small and medium targets. This paper proposes a Frequency Module(FRAN), a frequency-domain attention module for guiding patch generation to solve this problem. We use FRAN to allow the patched attack to rely on the low-frequency signals which are lost less in the shrinking process. This is the first study to introduce frequency domain attention to optimize the attack capabilities of adversarial patches. Our method increases the attack success rates of small and medium targets by 4.18% and 3.89%, respectively, over the state-of-the-art attack method for fooling the human detector while assaulting YOLOv3 without reducing the attack success rate of big targets.

Maliciously roaming person's detection around hospital surface using intelligent cloud-edge based federated learning

Maliciously roaming person's detection around hospital surface using intelligent cloud-edge based federated learning

A person-following method based on monocular camera for quadruped robots

This paper proposes a person-following method based on monocular vision, which allows quadruped robots to track a target person in both indoor and outdoor environments with different illumination conditions. Our method is composed of a person detector, a Kalman filter (KF) tracker, and a re-identification module. To be more specific, the person detector uses a human pose estimation method to detect persons. The KF is applied to predict the position of the target person and update its state with detection results. A re-identification module is proposed to deal with distractions, where the Convolutional Channel Features (CCF) is used to extract appearance features and Online Boosting is used to distinguish the target person from others. Especially, we design a target recapture mechanism based on the Recurrent Neural Network (RNN). Combining motion information with appearance features, the system can accurately re-identify the target person. Without extra customized markers, our method can track the target person steadily in real-time only using a monocular camera. Experiments results can validate the robustness and effectiveness of the proposed method.

Weakly Supervised Violence Detection in Surveillance Video.

Automatic violence detection in video surveillance is essential for social and personal security. Monitoring the large number of surveillance cameras used in public and private areas is challenging for human operators. The manual nature of this task significantly increases the possibility of ignoring important events due to human limitations when paying attention to multiple targets at a time. Researchers have proposed several methods to detect violent events automatically to overcome this problem. So far, most previous studies have focused only on classifying short clips without performing spatial localization. In this work, we tackle this problem by proposing a weakly supervised method to detect spatially and temporarily violent actions in surveillance videos using only video-level labels. The proposed method follows a Fast-RCNN style architecture, that has been temporally extended. First, we generate spatiotemporal proposals (action tubes) leveraging pre-trained person detectors, motion appearance (dynamic images), and tracking algorithms. Then, given an input video and the action proposals, we extract spatiotemporal features using deep neural networks. Finally, a classifier based on multiple-instance learning is trained to label each action tube as violent or non-violent. We obtain similar results to the state of the art in three public databases Hockey Fight, RLVSD, and RWF-2000, achieving an accuracy of 97.3%, 92.88%, 88.7%, respectively.

RGB camera-based fallen person detection system embedded on a mobile platform

Most injuries in the elderly are due to falls. The response time, to attend to the critically injured in such fall cases, is crucial to their survival. This paper presents a low-cost, autonomous assistive patrol robot which additionally includes a fallen person detection module with facial recognition that allows identification of patients. Patrol robots could be beneficial for care centers, where there is a considerable number of patients that require care. In these conditions, falls can be generally detected by the robotic platform during the post-fall phase. This allows the system to work with no frame rate constraints, allowing other tasks to be run simultaneously. Based on the YOLO network, we propose two approaches for the fallen person detector. The first approach can differentiate between fallen persons and persons doing ordinary activity in a single stage, while the second is a two-staged approach. The network weights were obtained using a fine-tuning process by retraining with our own extended Fall Person Dataset (E-FPDS), which we release as a benchmark for other RGB vision-based approaches. Quantitative evaluations confirm that the detector performs robustly in detecting fallen persons in different situations. The results also show a recall of 98.97% in our test set.

Physical Distancing Device with Edge Computing for COVID-19 (PADDIE-C19).

The most effective methods of preventing COVID-19 infection include maintaining physical distancing and wearing a face mask while in close contact with people in public places. However, densely populated areas have a greater incidence of COVID-19 dissemination, which is caused by people who do not comply with standard operating procedures (SOPs). This paper presents a prototype called PADDIE-C19 (Physical Distancing Device with Edge Computing for COVID-19) to implement the physical distancing monitoring based on a low-cost edge computing device. The PADDIE-C19 provides real-time results and responses, as well as notifications and warnings to anyone who violates the 1-m physical distance rule. In addition, PADDIE-C19 includes temperature screening using an MLX90614 thermometer and ultrasonic sensors to restrict the number of people on specified premises. The Neural Network Processor (KPU) in Grove Artificial Intelligence Hardware Attached on Top (AI HAT), an edge computing unit, is used to accelerate the neural network model on person detection and achieve up to 18 frames per second (FPS). The results show that the accuracy of person detection with Grove AI HAT could achieve 74.65% and the average absolute error between measured and actual physical distance is 8.95 cm. Furthermore, the accuracy of the MLX90614 thermometer is guaranteed to have less than 0.5 °C value difference from the more common Fluke 59 thermometer. Experimental results also proved that when cloud computing is compared to edge computing, the Grove AI HAT achieves the average performance of 18 FPS for a person detector (kmodel) with an average 56 ms execution time in different networks, regardless of the network connection type or speed.

Artificial Dummies for Urban Dataset Augmentation

Existing datasets for training pedestrian detectors in images suffer from limited appearance and pose variation. The most challenging scenarios are rarely included because they are too difficult to capture due to safety reasons, or they are very unlikely to happen. The strict safety requirements in assisted and autonomous driving applications call for an extra high detection accuracy also in these rare situations. Having the ability to generate people images in arbitrary poses, with arbitrary appearances and embedded in different background scenes with varying illumination and weather conditions, is a crucial component for the development and testing of such applications. The contributions of this paper are three-fold. First, we describe an augmentation method for the controlled synthesis of urban scenes containing people, thus producing rare or never-seen situations. This is achieved with a data generator (called DummyNet) with disentangled control of the pose, the appearance, and the target background scene. Second, the proposed generator relies on novel network architecture and associated loss that takes into account the segmentation of the foreground person and its composition into the background scene. Finally, we demonstrate that the data generated by our DummyNet improve the performance of several existing person detectors across various datasets as well as in challenging situations, such as night-time conditions, where only a limited amount of training data is available. In the setup with only day-time data available, we improve the night-time detector by 17% log-average miss rate over the detector trained with the day-time data only.

A Visual Leader-Following Approach With a T-D-R Framework for Quadruped Robots

The quadruped robot imitates the motions of four-legged animals with a superior flexibility and adaptability to complex terrains, compared with the wheeled and tracked robots. Its leader-following ability is unique to help a human to accomplish complex tasks in a more convenient way. However, long-term following is severely obstructed due to the high-frequency vibration of the quadruped robot and the unevenness of terrains. To solve this problem, a visual approach under a novel T-D-R framework is proposed. The proposed T-D-R framework is composed of a visual tracker based on correlation filter, a person detector with deep learning, and a person re-identification (re-ID) module. The result of the tracker is verified by the detector to improve tracking performance. Especially, the re-ID module is introduced to handle distractions and occlusion caused by other persons, where the convolutional correlation filter (CCF) is employed to discriminate the leader among multiple persons through recording the appearance information in the long run. By comparing the results of the tracker and the detector as well as their similarity scores with the leader identified by the re-ID module, a stable and real-time tracking of the leader can be guaranteed. Experiments reveal that our approach is effective in handling distractions, appearance changes, and illumination variations. A long-distance experiment on a quadruped robot indicates the validity of the proposed approach.

Multimodal person detection system

Person detection is often critical for personal safety, property protection, and national security. Most person detection technologies implement unimodal classification, making predictions based on a single sensor data modality, which is most often vision. There are many ways to defeat unimodal person detectors, and many more reasons to ensure technologies responsible for detecting the presence of a person are accurate and precise. In this paper, we design and implement a multimodal person detection system which can acquire data from multiple sensors and detect persons based on a variety of unimodal classifications and multimodal fusions. We present two methods of generating system-level predictions: (1) device perspectives which makes a final decision based on multiple device-level predictions and (2) system perspectives which combines data samples from multiple devices into a single data sample and then makes a decision. Our experimental results show that system-level predictions from system perspectives are generally more accurate than system-level predictions from device perspectives. We achieve an accuracy of 100%, zero false positive rate and zero false negative rate with fusion of system perspectives motion and distance data.

Joint Person Objectness and Repulsion for Person Search.

Person search targets to search the probe person from the unconstrainted scene images, which can be treated as the combination of person detection and person matching. However, the existing methods based on the Detection-Matching framework ignore the person objectness and repulsion (OR) which are both beneficial to reduce the effect of distractor images. In this paper, we propose an OR similarity by jointly considering the objectness and repulsion information. Besides the traditional visual similarity term, the OR similarity also contains an objectness term and a repulsion term. The objectness term can reduce the similarity of distractor images that not contain a person and boost the performance of person search by improving the ranking of positive samples. Because the probe person has a different person ID with its neighbors, the gallery images having a higher similarity with the neighbors of probe should have a lower similarity with the probe person. Based on this repulsion constraint, the repulsion term is proposed to reduce the similarity of distractor images that are not most similar to the probe person. Treating the Faster R-CNN as the person detector, the OR similarity is evaluated on PRW and CUHK-SYSU datasets by the Detection-Matching framework with six description models. The extensive experiments demonstrate that the proposed OR similarity can effectively reduce the similarity of distractor samples and further boost the performance of person search, e.g., improve the mAP from 92.32% to 93.23% for CUHK-SYSY dataset, and from 50.91% to 52.30% for PRW datasets.

A Robust Visual Person-Following Approach for Mobile Robots in Disturbing Environments

This article proposes a robust visual following approach with a deep learning-based person detector, a Kalman filter (KF), and a reidentification module. The KF is introduced to predict the position of the target person, and its state is updated by the associated detection result. To deal with severe distractions and even full occlusion, the reidentification module with an identification model, a verification model, and an appearance gallery is employed in multi-person disturbing environments. Without any customized markers, the proposed approach can follow the target person steadily, and it is robust to occlusion and posture changes of the target person. Experiments results validate the effectiveness of the proposed approach.