Surveillance Scenes Research Articles

One-Shot Any-Scene Crowd Counting With Local-to-Global Guidance.

Due to different installation angles, heights, and positions of the camera installation in real-world scenes, it is difficult for crowd counting models to work in unseen surveillance scenes. In this paper, we are interested in accurate crowd counting based on the data collected by any surveillance camera, that is to count the crowd from any scene given only one annotated image from that scene. To this end, we firstly pose crowd counting as a one-shot learning task. Through the metric-learning, we propose a simple yet effective method that firstly estimates crowd characteristics and then transfers them to guide the model to count the crowd. Specifically, to fully capture these crowd characteristics of the target scene, we devise the Multi-Prototype Learner to learn the prototypes of foreground and density from the limited support image using the Expectation-Maximization algorithm. To learn the adaptation capability for any unseen scene, estimated multi prototypes are proposed to guide the crowd counting of query images in a local-to-global way. CNN is utilized to activate the local features. And transformer is introduced to correlate global features. Extensive experiments on three surveillance datasets suggest that our method outperforms the SOTA methods in the few-shot crowd counting.

Optimal deep transfer learning enabled object detector for anomaly recognition in pedestrian ways

Anomaly detection is a branch of behavior understanding in surveillance scenes, where anomalies represent a deviation in the behavior of scene entities (viz.,humans, vehicles, and environment) from regular patterns. In pedestrian walkways, this plays a vital role in enhancing safety. With the widespread use of video surveillance systems and the escalating video volume, manual examination of abnormal events becomes time-intensive.Hence, the need for an automated surveillance system adept at anomaly detection is crucial, especially within the realm of computer vision (CV) research. The surge in interest towards deep learning (DL) algorithms has significantly impacted CV techniques, including object detection and classification. Unlike traditional reliance on supervised learning requiring labeled datasets, DL offers advancements in these applications. Thus, this study presents an Optimal Deep Transfer Learning Enabled Object Detector for Anomaly Recognition in Pedestrian Ways (ODTLOD-ARPW) technique. The purpose of the ODTLOD-ARPW method is to recognize the occurrence of anomalies in pedestrian walkways using a DL-based object detector. In the ODTLOD-ARPW technique, the image pre-processing initially takes place using two sub-processes namely Wiener filtering (WF) based pre-processing and dynamic histogram equalization-based contrast enhancement. For anomaly detection, the ODTLOD-ARPW technique employs the YOLOV8s model which offers enhanced accuracy and performance. The hyperparameter tuning process takes place using a root mean square propagation (RMSProp) optimizer. The performance analysis of the ODTLOD-ARPW method is tested under the UCSD anomaly detection dataset. An extensive comparative study reported that the ODTLOD-ARPW technique reaches an effective performance with other models with maximum accuracy of 98.67%.

Audio–visual representation learning for anomaly events detection in crowds

In recent years, anomaly events detection in crowd scenes attracts many researchers’ attentions, because of its importance to public safety. Existing methods usually exploit visual information to analyze whether any abnormal events have occurred due to only visual sensors are generally equipped in public places. However, when an abnormal event in crowds occurs, sound information may be discriminative to assist the crowd analysis system to determine whether there is an abnormality. Compared with vision information that is easily occluded, audio signals have a certain degree of penetration. Thus, this paper attempt to exploit multi-modal learning for modeling the audio and visual signals simultaneously. To be specific, we design a two-branch network to model different types of information. The first is a typical 3D CNN model to extract temporal appearance feature from video clips. The second is an audio CNN for encoding Log Mel-Spectrogram of audio signals. Finally, by fusing the above features, the more accurate prediction will be produced. We conduct the experiments on SHADE dataset, a synthetic audio–visual dataset in surveillance scenes, and find introducing audio signals effectively improves the performance of anomaly events detection and outperforms other state-of-the-art methods. Furthermore, we will release the code and the pre-trained models as soon as possible.

Enhancing Surveillance Systems: Integration of Object, Behavior, and Space Information in Captions for Advanced Risk Assessment.

This paper presents a novel approach to risk assessment by incorporating image captioning as a fundamental component to enhance the effectiveness of surveillance systems. The proposed surveillance system utilizes image captioning to generate descriptive captions that portray the relationship between objects, actions, and space elements within the observed scene. Subsequently, it evaluates the risk level based on the content of these captions. After defining the risk levels to be detected in the surveillance system, we constructed a dataset consisting of [Image-Caption-Danger Score]. Our dataset offers caption data presented in a unique sentence format, departing from conventional caption styles. This unique format enables a comprehensive interpretation of surveillance scenes by considering various elements, such as objects, actions, and spatial context. We fine-tuned the BLIP-2 model using our dataset to generate captions, and captions were then interpreted with BERT to evaluate the risk level of each scene, categorizing them into stages ranging from 1 to 7. Multiple experiments provided empirical support for the effectiveness of the proposed system, demonstrating significant accuracy rates of 92.3%, 89.8%, and 94.3% for three distinct risk levels: safety, hazard, and danger, respectively.

Texture-guided transfer learning for low-quality face recognition.

Although many advanced works have achieved significant progress for face recognition with deep learning and large-scale face datasets, low-quality face recognition remains a challenging problem in real-word applications, especially for unconstrained surveillance scenes. We propose a texture-guided (TG) transfer learning approach under the knowledge distillation scheme to improve low-quality face recognition performance. Unlike existing methods in which distillation loss is built on forward propagation; e.g., the output logits and intermediate features, in this study, the backward propagation gradient texture is used. More specifically, the gradient texture of low-quality images is forced to be aligned to that of its high-quality counterpart to reduce the feature discrepancy between the high-and low-quality images. Moreover, attention is introduced to derive a soft-attention (SA) version of transfer learning, termed as SA-TG, to focus on informative regions. Experiments on the benchmark low-quality face DB's TinyFace and QMUL-SurFace confirmed the superiority of the proposed method, especially more than 6.3% Rank1 accuracy improvement is achieved on TinyFace.

Motion estimation and multi-stage association for tracking-by-detection

Multi-object tracking (MOT) aims to locate and identify objects in videos. As deep learning brings excellent performances to object detection, the tracking-by-detection (TBD) has gradually become a mainstream tracking framework. However, some drawbacks still exist in the current TBD framework: (1) inaccurate prediction of the bounding boxes would occur in the detection part, which is caused by overlooking the actual pedestrian ratio in the surveillance scene. (2) The width of the bounding boxes in the next frame might be indirectly predicted by the aspect ratio, which increases the error of width prediction in the motion prediction part. (3) Association is only performed for high-confidence detection boxes, and the low-confidence boxes caused by occlusion are discarded in the data association part, resulting in fragmentation of trajectories. To address the above issues, we propose a multi-target tracking model incorporating motion estimation and multi-stage association (MEMA). First, the aspect ratio of the ground-true bounding box is introduced to improve the fit of the detection and the ground-true bounding box, and we design the elliptical Gaussian kernel to improve the positioning accuracy of the object center point. Then, the prediction state vector of the Kalman filter is modified to predict the width and its corresponding velocity directly. It can reduce the width error of the prediction box and eliminate the velocity error of the motion estimation, which leads to a more pedestrian-friendly prediction bounding box. Finally, we propose a multi-stage association strategy to correlate different confidence boxes. Without using the appearance feature, the strategy can reduce the impact of occlusion and improve the tracking performance. On the MOT17 test set, the method proposed in this paper achieves a MOTA of 74.3% and an IDF1 of 72.4%, outperforming the current SOTA.

Spatio-temporal human action localization in indoor surveillances

Spatio-temporal action localization is a crucial and challenging task in the field of video understanding. Existing benchmarks for spatio-temporal action detection are limited by factors such as incomplete annotations, high-level non-universal actions, and uncommon scenarios. To address these limitations and facilitate research in real-world security applications, we introduce a novel human-centric dataset for spatio-temporal localization of atomic actions in indoor surveillance settings, termed as HIA (Human-centric Indoor Actions). The HIA dataset is constructed by selecting 30 atomic action classes, compiling 100 surveillance videos, and annotating 219,225 frames with 370,937 bounding boxes. The primary characteristics of HIA include (1) accurate spatio-temporal annotations for atomic actions, (2) human-centric annotations at the frame level, (3) temporal linking of persons across discontinuous tracks, and (4) utilization of indoor surveillance videos. Our HIA, with its realistic settings in indoor surveillance scenes and comprehensive annotations, presents a valuable and novel challenge to the spatio-temporal action localization domain. To establish a benchmark, we evaluate various methods and provide an in-depth analysis of the HIA dataset. The HIA dataset will be made available soon, and we anticipate that it will serve as a standard and practical benchmark for the research community.

HS-YOLO: Small Object Detection for Power Operation Scenarios

Object detection methods are commonly employed in power safety monitoring systems to detect violations in surveillance scenes. However, traditional object detection methods are ineffective for small objects that are similar to the background information in the power monitoring scene, which consequently affects the performance of violation behavior detection. This paper proposed a small object detection algorithm named HS-YOLO, based on High-Resolution Network (HRNet) and sub-pixel convolution. First, to fully extract the microfeature information of the object, a small object feature extraction backbone network is proposed based on the HRNet structure. The feature maps of different scales are processed by multiple parallel branches and fused with each other in the network. Then, to fully retain the effective features of small objects, the sub-pixel convolution module is incorporated as the upsampling operator in the feature fusion network. The low-resolution feature map is upsampled to a higher resolution by reorganizing pixel values and performing padding operations in this module. On our self-constructed power operation dataset, the HS-YOLO algorithm achieved a mAP of 87.2%, which is a 3.5% improvement compared to YOLOv5. Particularly, the dataset’s AP for detecting small objects such as cuffs, necklines, and safety belts is improved by 10.7%, 5.8%, and 4.4%, respectively. These results demonstrate the effectiveness of our proposed method in detecting small objects in power operation scenarios.

Efficient background subtraction method based on fast independent component analysis in video-surveillance

Modern video surveillance has now become an active area of research with a large set of requirements and various applications. In order to detect moving objects in video surveillance scenes, background subtraction techniques are the most used. In this paper, we developed and tested an efficient background subtraction technique in video surveillance based on the fast-independent component analysis (fast-ICA) method. The proposed technique initiated, first, on the use of a developed fast-ICA algorithm in order to estimate the de-mixing matrix and the denoising matrix parameters. Second, the estimated foreground can simply model by multiplying the data matrix with the de-mixing matrix. After that, the data matrix is multiplied by the denoising matrix for removing the noise. In addition, we propose a pre-processing and post-processing operations to effectively segment the true foreground objects and improve our results. The proposed method is evaluated on the publicly available change detection datasets CDnet 2012 and CDnet 2014 using performance parameters such as recall, precision and experimental results show that our algorithm can detect effectively and accurately the moving objects in several background and foreground conditions compared to other methods in literature with real-time frame rate.

Multispectral Invisible Coating: Laminated Visible-Thermal Physical Attack against Multispectral Object Detectors Using Transparent Low-E Films

Multispectral object detection plays a vital role in safety-critical vision systems that require an around-the-clock operation and encounter dynamic real-world situations(e.g., self-driving cars and autonomous surveillance systems). Despite its crucial competence in safety-related applications, its security against physical attacks is severely understudied. We investigate the vulnerability of multispectral detectors against physical attacks by proposing a new physical method: Multispectral Invisible Coating. Utilizing transparent Low-e films, we realize a laminated visible-thermal physical attack by attaching Low-e films over a visible attack printing. Moreover, we apply our physical method to manufacture a Multispectral Invisible Suit that hides persons from the multiple view angles of Multispectral detectors. To simulate our attack under various surveillance scenes, we constructed a large-scale multispectral pedestrian dataset which we will release in public. Extensive experiments show that our proposed method effectively attacks the state-of-the-art multispectral detector both in the digital space and the physical world.

Survey on Abnormal Event Detection and Signalling in Multiple Video Surveillance Scenes Using CNN

Survey on Abnormal Event Detection and Signalling in Multiple Video Surveillance Scenes Using CNN

Abnormal Event Detection and Signaling in Multiple Video Surveillance Scenes Using CNN

Abnormal Event Detection and Signaling in Multiple Video Surveillance Scenes Using CNN

Human skeleton behavior recognition model based on multi-object pose estimation with spatiotemporal semantics

Multi-object pose estimation in surveillance scenes is challenging and inaccurate due to object motion blur and pose occlusion in video data. Targeting at the temporal dependence and coherence among video frames, this paper reconstructs a multi-object pose estimation model that integrates spatiotemporal semantics for different scales and poses of video multi-objects. The model firstly, with an end-to-end detection framework, detects multiple targets in the video. Secondly, it enhances the positioning of key points of human body using the temporal cues among video frames and designs modular components to enrich the pose information, effectively refining the pose estimation. Finally, the improved human skeleton behavior recognition model based on pose estimation is employed to recognize the classroom behaviors of students oriented to video streams. Comparison with multiple classifiers through experiments reveals that the human skeleton behavior recognition model for multi-object pose estimation combined with spatiotemporal semantics exhibits an effectively improved accuracy.

Apron surveillance video coding based on compositing virtual reference frame with object library

AbstractA video coding framework for the apron surveillance scene has been proposed in this paper, which aims to improve coding efficiency by eliminating long‐term redundancy at the object level. To achieve this goal, this study first develops an existing block‐based hybrid video coding framework by exploiting the video redundancy on the object level to perform video coding. Second, an object‐library mechanism is designed to collect the representative object images as coding references on larger temporal and spatial scales. Finally, a virtual reference frame, which blends background and foreground references from the object library, is adaptively composited according to the video content to improve the inter‐prediction performance. Preliminary experimental results demonstrate that the proposed method achieves a high BD rate reduction of up to 23.97% in apron surveillance video sequences, compared to the standard high efficiency video coding (HEVC).

Low-Cost Systematic Methodology for Rapidly Constructing a Physiological Monitoring Interface in ICU

During the COVID-19 pandemic, which emerged in 2020, many patients were treated in isolation wards because of the high infectivity and long incubation period of COVID-19. Therefore, monitoring systems have become critical to patient care and to safeguard medical professional safety. The user interface is very important to the surveillance system; therefore, we used web technology to develop a system that can create an interface based on user needs. When the surveillance scene needs to be changed, the surveillance location can be changed at any time, effectively reducing the costs and time required, so that patients can achieve timely and appropriate goals of treatment. ZigBee was employed to develop a monitoring system for intensive care units (ICUs). Unlike conventional GUIs, the proposed GUI enables the monitoring of various aspects of a patient, and the monitoring interface can be modified according to the user needs. A simulated ICU environment monitoring system was designed to test the effectiveness of the system. The simulated environment and monitoring nodes were set up at positions consistent with the actual clinical environments to measure the time required to switch between the monitoring scenes or targets on the GUI. A novel system that can construct ZigBee-simulated graphical monitoring interfaces on demand was proposed in this study. The locations of the ZigBee monitoring nodes in the user interface can be changed at any time. The time required to deploy the monitoring system developed in this study was 4 min on average, which is much shorter than the time required for conventional methods (131 min). The system can effectively overcome the limitations of the conventional design methods for monitoring interfaces. This system can be used to simultaneously monitor the basic physiological data of numerous patients, enabling nursing professionals to instantly determine patient status and provide appropriate treatments. The proposed monitoring system can be applied to remote medical care after official adoption.

Railway intrusion detection based on refined spatial and temporal features for UAV surveillance scene

Railway intrusion detection is important for the safety of the railway operation. However, how to detect intruding events in the scene of unmanned aerial vehicle (UAV) surveillance is a challenging task, especially when there are multiple railway regions or unknown types of intruding objects. To tackle the above problems, a railway intrusion detection method is proposed for the scene of UAV surveillance. It can detect intruding events accurately without pre-setting the intruding object type by detecting the normal railway region and the abnormal railway region with intruding objects rather than recognizing the intruding objects directly. The proposed network consists of a novel Fused-convolutional long short-term memory (Fused-ConvLSTM) module for efficiently propagating spatial and temporal information; several attention modules for adaptively enhancing the feature information, such as the improved feature fusion layer; several lightweight strategies for reducing the computation complexity, such as lightweight feature extraction and width multiplier. Experiments are performed on the real aerial videos obtained by a micro UAV. The experimental results demonstrate that the proposed method can accurately detect railway intrusion events in aerial video, and effectively deal with the case of multiple railway regions and unknown intruding object types. The experimental results also demonstrate that the proposed network can achieve better performance than other object detection networks.

Visual tracking in complex scenes: A location fusion mechanism based on the combination of multiple visual cognition flows

In recent years, deep learning has revolutionized computer vision and has been widely used for monitoring in diverse visual scenes. However, in terms of some aspects such as complexity and explainability, deep learning is not always preferable over traditional machine-learning methods. Traditional visual tracking approaches have shown certain advantages in terms of data collection efficiency, computing requirements, and power consumption and are generally easier to understand and explain than deep neural networks. At present, traditional feature-based techniques relying on correlation filtering (CF) have become common for understanding complex visual scenes. However, current CF algorithms use a single feature to describe the information of the target and locate it accordingly. They cannot fully express changeable target appearances in a complex scene, which can easily lead to inaccurate target locations in time-varying visual scenes. Moreover, owing to the complexity of surveillance scenes, monitoring algorithms can lose their target. The original template update strategy uses each frame with a fixed interval length as a new template, which may lead to unreliable feature extraction and low tracking accuracy. To overcome these issues, in this work, we introduce an original location fusion mechanism based on multiple visual cognition processing streams to achieve real-time and efficient visual monitoring in complex scenes. First, we propose a process for extracting multiple forms of visual cognitive information, and it is periodically used to extract multiple feature information flows of a target of interest. Subsequently, a cognitive information fusion process is employed to fuse the positioning results of different visual cognitive information flows to achieve high-quality visual monitoring and positioning. Finally, a novel feature template memory storage and retrieval strategy is adopted. When the location result is unreliable, the target is retrieved from memory to ensure robust and accurate tracking. In addition, we provide an extensive set of performance results showing that our proposed approach exhibits more robust performance at a lower computational cost compared with 36 state-of-the-art algorithms for visual tracking in complex scenes.

Center-point-pair detection and context-aware re-identification for end-to-end multi-object tracking

Online multi-object tracking aims at generating the trajectories for multiple objects in the surveillance scene. It remains a challenging problem in crowded scenes because objects often gather together and occlude in tracking frames. The main impact of crowd occlusions is that it severely harms the performance of the detector and significantly increases the difficulty in extracting object features. In this paper, we propose an end-to-end tracking framework that alleviates such issues and estimates more accurate trajectories. Firstly, We design a Center-Point-Pair detection branch for object detection, which learns the correlations between the object head and the body to simultaneously predict the head and body regions to alleviate unreliable detection in tracking scenes. Secondly, we introduce the context information around the object to the tracker, inspired by the human search pattern. We propose a Context-Aware Re-Identification branch that includes the Previous-Frame Guided Spatial-Attention Model and the Previous-Frame Guided Channel-Attention Model to extract more discriminative object features. Thirdly, to harness the power of deep features for data association in generating reliable trajectories, we propose the Similarity Cluster Trajectory Management method that expands affinity descriptor and adopts the minority obeying the majority principle to association trajectories and detections. The experiments on diverse and challenging MOT datasets show that our tracking framework achieves superior results compared to other state-of-the-arts offline and online multi-object tracking methods.

An Adaptive Vector-Based Vehicles Detection for Urban Intersection Camera Sensors Under Nighttime Illumination

Vehicles detection via surveillance sensors is an intractable and essential task under nighttime urban intersection scenes. To efficiently resolve the problem that most of the current color or texture feature-based vehicles detection models easily suffer contamination from dynamic and sudden or gradual illumination variations on nighttime, an adaptive vector-based background model without complex artificial texture feature is proposed for vehicles detection in night intersection surveillance scenes. For each pixel of image, a series of filters are employed to take the values from the past at the same location and its neighborhood, and a vector is assigned to each filter to represent the neighborhood region information of the pixel. Then, the series of vectors comprising the background model and the vector of the current pixel is compared to the vectors of the background model based on the theorem of linear dependence to describe whether that pixel belongs to foreground or background. Eventually, to adapt the dynamic scenes, a random update scheme is employed to update the model. As our experimental results demonstrated in real-world nighttime urban intersection surveillance scenarios, the proposed model attains superior vehicles detection performance compared to other state-of-the-art algorithms based on extensive qualitative and quantitative evaluations.

Fast intensive crowd counting model of Internet of Things based on multi‐scale attention mechanism

AbstractObject detection based on deep learning plays an important role in the application of the Internet of Things (IoT). Traditional methods consume a lot of computing resources and cannot be well deployed in the IoT environment. A lightweight object detection method based on attention mechanism is proposed and applied to crowd counting. In view of the low accuracy and poor real‐time performance of multi‐scale crowd detection, we design a crowd counting model based on YOLO v5, and apply it to the IoT environment. It is proposed to insert the transformer into the YOLO v5 backbone network. Based on the multi‐head attention mechanism in the transformer encoder, the global dependency is modelled to make full use of the context information. The CNN is used to realize the fusion of multi‐scale feature maps, and the feature enhancement modules concerned by the attention network are further counted. Experiments show that it can not only detect multi‐scale targets, but also achieve real‐time performance in video surveillance scenes.