Video Surveillance Environment Research Articles

CrimeNet: Neural Structured Learning using Vision Transformer for violence detection

The state of the art in violence detection in videos has improved in recent years thanks to deep learning models, but it is still below 90% of average precision in the most complex datasets, which may pose a problem of frequent false alarms in video surveillance environments and may cause security guards to disable the artificial intelligence system.In this study, we propose a new neural network based on Vision Transformer (ViT) and Neural Structured Learning (NSL) with adversarial training. This network, called CrimeNet, outperforms previous works by a large margin and reduces practically to zero the false positives. Our tests on the four most challenging violence-related datasets (binary and multi-class) show the effectiveness of CrimeNet, improving the state of the art from 9.4 to 22.17 percentage points in ROC AUC depending on the dataset. In addition, we present a generalisation study on our model by training and testing it on different datasets. The obtained results show that CrimeNet improves over competing methods with a gain of between 12.39 and 25.22 percentage points, showing remarkable robustness.

Research on 24-Hour Dense Crowd Counting and Object Detection System Based on Multimodal Image Optimization Feature Fusion

Motivation. In the environment of day and night video surveillance, in order to improve the accuracy of machine vision dense crowd counting and target detection, this paper designs a day and night dual-purpose crowd counting and crowd detection network based on multimode image fusion. Methods. Two sub-models, RGBD-Net and RGBT-Net, are designed in this paper. The depth image features and thermal imaging features are effectively fused with the features of visible light images, so that the model has stronger anti-interference characteristics and robustness to the light noise interference caused by the sudden fall of light at night. The above models use density map regression-guided detection method to complete population counting and detection. Results. The model completed daytime training and testing on MICC dataset. Through verification, the average absolute error of the model was 1.025, the mean square error was 1.521, and the recall rate of target detection was 97.11%. Night vision training and testing were completed on the RGBT-CC dataset. After verification, the average absolute error of the network was 18.16, the mean square error was 32.14, and the recall rate of target detection was 97.65%. By verifying the effectiveness of the multimode medium-term fusion network, it is found to exceed the current most advanced bimodal fusion method. Conclusion. The experimental results show that the proposed multimodal fusion network can solve the counting and detection problem in the video surveillance environment during day and night. The ablation experiment further proves the effectiveness of the parameters of the two models.

Spatio-Temporal association rule based deep annotation-free clustering (STAR-DAC) for unsupervised person re-identification

Multi-camera video surveillance environment has a variety of emerging research problems among, which person re-identification is the premier one. Unsupervised person re-identification has been explored less in literature than the supervised approach. Images acquired from the video surveillance systems are unlabeled, which denotes that it is naturally an unsupervised learning problem. The state-of-the-art unsupervised methods seek external annotations support such as incorporating transfer learning techniques, partial labeling of train images, etc., which makes them not purely unsupervised and unsuitable for practical real-world surveillance settings. Identity mismatch happens due to the similar costumes and complex environmental factors. To resolve this issue, we introduce a new framework named Spatio-Temporal Association Rule based Deep Annotation-free Clustering (STAR-DAC) which incrementally clusters the unlabeled person re-identification images based on visual features and performs cluster fine-tuning through the mined spatio-temporal association rules. STAR formulations leveraged upto 75% of images for reliable sample selection through cluster fine-tuning. STAR based fine-tune algorithm aims to attain ground-truth labels of an unlabeled dataset and eliminate cluster outliers to stabilize the evaluation. Experiments are performed on image and video-based benchmark person re-identification datasets such as DukeMTMC re-ID, Market1501, MSMT17, CUHK03, GRID and Dukevideo re-ID, iLIDSVid, ViPer respectively. Experimental results clearly show that the proposed STAR-DAC framework outperforms the state-of-the-art methods in case of large scale datasets with multiple cameras.

CCTV-RFID enabled multifactor authentication model for secure differential level video access control

In the intelligent CCTV surveillance environment, personal identity is confirmed based on face recognition. However, the recognition rate of the current face recognition technology is still faulty. In particular, face recognition may not work correctly due to various causes such as CCTV shot quality, weather, personal pose and facial expression, hairstyle, lighting condition, and so on. In this case, there is a great risk of exposing object’s privacy information in the video surveillance environment due to erroneous object judgment. This paper proposes a video surveillance-based access control technique that combines a facial recognition system using CCTV machine learning with radio-frequency identification (RFID). The proposed method is implemented when accurate facial recognition is difficult to achieve due to poor video quality or low levels of similarity against feature vectors, in which cases multi-channel authentication is performed with the use of RFID features available on a mobile device in possession of the individual. The dual-channel authentication approach can still help identify the entity and protect his or her privacy with greater security even if the RFID tags for authentication are breached or accurate facial detection becomes challenging due to various factors such as CCTV video quality deterioration.

AdViSED: Advanced Video SmokE Detection for Real-Time Measurements in Antifire Indoor and Outdoor Systems

This paper proposes a video-based smoke detection technique for early warning in antifire surveillance systems. The algorithm is developed to detect the smoke behavior in a restricted video surveillance environment, both indoor (e.g., railway carriage, bus wagon, industrial plant, or home/office) or outdoor (e.g., storage area or parking area). The proposed technique exploits a Kalman estimator, color analysis, image segmentation, blob labeling, geometrical features analysis, and M of N decisor, in order to extract an alarm signal within a strict real-time deadline. This new technique requires just a few seconds to detect fire smoke, and it is 15 times faster compared to the requirements of fire-alarm standards for industrial or transport systems, e.g., the EN50155 standard for onboard train fire-alarm systems. Indeed, the EN50155 considers a response time of at least 60 s for onboard systems. The proposed technique has been tested and compared with state-of-art systems using the open access Firesense dataset developed as an output of a European FP7 project, including several fire/smoke indoor and outdoor scenes. There is an improvement of all the detection metrics (recall, accuracy, F1 score, precision, etc.) when comparing Advanced Video SmokE Detection (AdViSED) with other video-based antifire works recently proposed in literature. The proposed technique is flexible in terms of input camera type and frame size and rate and has been implemented on a low-cost embedded platform to develop a distributed antifire system accessible via web browser.

Appearance model update based on online learning and soft‐biometrics traits for people re‐identification in multi‐camera environments

Intelligent surveillance systems in multi-camera environments pose a hard-open problem for computer vision. The way the people look changes inside and also among cameras, so people re-identification task can be largely improved collecting data about people already identified and take advantage of it as time advances in surveillance video. Furthermore, a camera change or a slight change in the objective traits may require the complete re-formulation of the appearance models. In this paper, we propose several heuristics for updating the appearance model in a multi-camera surveillance environment. Through these heuristics, the subject's appearance model is updated across different time and environmental conditions. The update process is carried out primarily in three different aspects: 1) based on time lapses, 2) based on the change of camera, and 3) based on the automatic selection of the most representative samples selected through decision functions of the classifier. The proposed system focuses on video surveillance environments, that is, the objective is to identify an individual across the set of cameras in the surveillance area, the comparison considers only those people that share time and space. We used four public benchmarks to test our claims; the results confirm the importance of continuous appearance model's updating.

Lightweight knowledge-based authentication model for intelligent closed circuit television in mobile personal computing

In the intelligent CCTV surveillance environment, personal identity is confirmed based on face recognition. However, the recognition rate of the current face recognition technology is still faulty. In particular, face recognition may not work correctly due to various causes such as CCTV shot quality, weather, personal pose and facial expression, hairstyle, lighting condition, and so on. In this case, there is a great risk of exposing an object’s privacy information in the video surveillance environment due to erroneous object judgment. This paper proposes a video surveillance-based access control technique that combines a facial recognition system using CCTV machine learning with radio-frequency identification (RFID). The proposed method is implemented when accurate facial recognition is difficult to achieve due to poor video quality or low levels of similarity against feature vectors, in which cases multi-channel authentication is performed with the use of RFID features available on a mobile device in possession of the individual. The dual-channel authentication approach can still help identify the entity and protect his or her privacy with greater security even if the RFID tags for authentication are breached or accurate facial detection becomes challenging due to various factors such as CCTV video quality deterioration.

CCTV Video Processing Metadata Security Scheme Using Character Order Preserving-Transformation in the Emerging Multimedia

Intelligent video surveillance environments enable the gathering of various types of information about the object being recorded, through the analysis of real-time video data collected from CCTV systems and the automated processing that utilize the information. However, the surveillance environments face the risks of privacy exposure, which necessitates secure countermeasures. Video meta-data, in particular, contain various types of personal information that is analyzed based on big data and are thus fraught with high levels of confidentiality breaches. Despite such risks, it is not appropriate to implement encryption for video meta-data considering the efficiency issue. This paper proposes a character order preserving (COP)-transformation technique that allows the secure protection of video meta-data. The proposed technique has the merits of preventing the recovery of original meta information through meta transformation and allowing direct queries on the data transformed, increasing significantly both security and efficiency in the video meta-data processing.

People detection in surveillance: classification and evaluation

Nowadays, people detection in video surveillance environments is a task that has been generating great interest. There are many approaches trying to solve the problem either in controlled scenarios or in very specific surveillance applications. The main objective of this study is to give a comprehensive and extensive evaluation of the state of the art of people detection regardless of the final surveillance application. For this reason, first, the different processing tasks involved in the automatic people detection in video sequences have been defined, then a proper classification of the state of the art of people detection has been made according to the two most critical tasks, object detection and person model, that are needed in every detection approach. Finally, experiments have been performed on an extensive dataset with different approaches that completely cover the proposed classification and support the conclusions drawn from the state of the art.

A Single-Pass Connected Component Labeler without Label Merging Period

In this paper, for real-time multi-channel video analytics, an architecture of a single-pass connected component labeler without label merging period is proposed; this structure can handle multiple video channels from incoming video signals or from memory. During the line scan period, after determining an event based on the 8-connectivity check of incoming pixels, an 8-connectivity checker notifies all the label registers inside a label shift register of the event and the control signals for label handling. Each label register takes a label input either normally from the shifted label in the label shift register, or from the merging label without additional cycles for the label merging event. A label stack is designed for efficient label allocation with label reuse. An information extractor calculates bounding boxes and centers of gravity for labeled objects using the information from the 8-connectivity checker and the pixel counter. The proposed architecture is finally implemented on a field programmable gate array (FPGA) device, integrated into a video analytics system, and verified as to its functions and performance. More than 28 % of the memory usage by conventional architectures is reduced in the proposed architecture for the same maximum number of labels in video surveillance environments.

Soft-biometrics evaluation for people re-identification in uncontrolled multi-camera environments

A novel method for person identification based on soft-biometrics and oriented to work in real video surveillance environments is proposed in this paper. Thus, an evaluation of relevance’s level of several appearance features is carried out with this purpose. First, a bag-of-soft-biometric features related to color, texture, local features, and geometry are extracted from individuals. The relevance of each feature has been deeply analyzed through different proposed methods. Features are ranked and weighted according to their relevance value. Later, each method is evaluated under two different scenarios: mono-camera and multi-camera surveillance images. In order to test the system in a realistic way, it has been evaluated over standard databases in the surveillance community: PETS 2006, PETS 2009, CAVIAR, SAIVT-SoftBio, and CAVIAR4REID. Moreover, a new database was acquired at Adolfo Suarez Madrid-Barajas international airport. This database was acquired under regular conditions and infrastructure of the Barajas airport, no additional camera or special settings were installed for this purpose. An analysis of relevance for each feature acquired in these two scenarios is presented. The results obtained demonstrate the promising potential of the soft-biometric approach. Finally, an optimal system configuration according to each scenario is obtained.

Adaptive skew-sensitive ensembles for face recognition in video surveillance

Decision support systems for surveillance rely more and more on face recognition (FR) to detect target individuals of interest captured with video cameras. FR is a challenging problem in video surveillance due to variations in capture conditions, to camera interoperability, and to the limited representativeness of target facial models used for matching. Although adaptive classifier ensembles have been applied for robust face matching, it is often assumed that the proportions of faces captured for target and non-target individuals are balanced, known a priori, and do not change over time. Recently, some techniques have been proposed to adapt the fusion function of an ensemble according to class imbalance of the input data stream. For instance, Skew-Sensitive Boolean combination (SSBC) is a active approach that estimates target vs. non-target proportions periodically during operations using Hellinger distance, and adapts its ensemble fusion function to operational class imbalance. Beyond the challenges of estimating class imbalance, such techniques commonly generate diverse pools of classifiers by selecting balanced training data, limiting the potential diversity produced using the abundant non-target data. In this paper, adaptive skew-sensitive ensembles are proposed to combine classifiers trained by selecting data with varying levels of imbalance and complexity, to sustain a high level the performance for video-to-video FR. Faces captured for each person in the scene are tracked and regrouped into trajectories. During enrollment, captures in a reference trajectory are combined with selected non-target captures to generate a pool of 2-class classifiers using data with various levels of imbalance and complexity. During operations, the level of imbalance is periodically estimated from the input trajectories using the HDx quantification method, and pre-computed histogram representations of imbalanced data distributions. This approach allows one to adapt pre-computed histograms and ensemble fusion functions based on the imbalance and complexity of operational data. Finally, the ensemble scores are accumulated of trajectories for robust spatio-temporal recognition. Results on synthetic data show that adapting the fusion function of ensemble trained with different complexities and levels of imbalance can significantly improve performance. Results on the Face in Action video data show that the proposed method can outperform reference techniques (including SSBC and meta-classification) in imbalanced video surveillance environments. Transaction-based analysis shows that performance is consistently higher across operational imbalances. Individual-specific analysis indicates that goat- and lamb-like individuals can benefit the most from adaptation to the operational imbalance. Finally, trajectory-based analysis shows that a video-to-video FR system based on the proposed approach can maintain, and even improve overall system discrimination.

Post-processing approaches for improving people detection performance

People detection in video surveillance environments is a task that has been generating great interest. There are many approaches trying to solve the problem either in controlled scenarios or in very specific surveillance applications. We address one of the main problems of people detection in video sequences: every people detector from the state of the art must maintain a balance between the number of false detections and the number of missing pedestrians. This compromise limits the global detection results. In order to reduce or relax this limitation and improve the detection results, we evaluate two different post-processing subtasks. Firstly, we propose the use of people-background segmentation as a filtering stage in people detection. Then, we evaluate the combination of different detection approaches in order to add robustness to the detection and therefore improve the detection results. And, finally, we evaluate the successive application of both post-processing approaches. Experiments have been performed on two extensive datasets and using different people detectors from the state of the art: the results show the benefits achieved using the proposed post-processing techniques.

Adaptive ensembles for face recognition in changing video surveillance environments

Recognizing faces corresponding to target individuals remains a challenging problem in video surveillance. Face recognition (FR) systems are exposed to videos captured under various operating conditions, and, since data distributions change over time, face captures diverge w.r.t. stored facial models. Although these models may be adapted when new reference videos become available, incremental learning with faces captured under different conditions may lead to knowledge corruption. This paper presents an adaptive multi-classifier system (AMCS) for video-to-video FR in changing surveillance environments. During enrolment, faces captured in reference videos are employed to design an individual-specific classifier. During operations, a tracker allows to regroup facial captures for individuals in the scene, and accumulate the predictions per track for robust spatiotemporal FR. Given a new reference video, the corresponding facial model is adapted according to the type of concept change. If a gradual pattern of change is detected, the individual-specific classifier(s) are adapted through incremental learning. To preserve knowledge, another classifier is learned and combined with the individuals previously-trained classifier(s) if an abrupt change is detected. For proof-of-concept, the performance of a particular implementation of this AMCS is assessed using videos from the Faces in Action dataset. By adapting facial models according to changes detected in new reference videos, this AMCS allows to sustain a high level of accuracy comparable to the same system that is always updated using a learn-and-combine approach, while reducing time and memory complexity. It also provides higher accuracy than incremental learning classifiers that suffer the effects of knowledge corruption.

Object Detection based on Combination of Visible and Thermal Videos using A Joint Sample Consensus Background Model

In uncontrolled video surveillance environments, performing efficient foreground segmentation is very challenging. In order to improve robustness and accuracy of object detection, we take advantage of spectral information of both visible and thermal videos. This paper presents a novel joint background model combining visible and thermal videos for foreground object detection in complex scenarios. Different from traditional methods that first detected moving objects in either domain respectively and then fused the detection results, we provide a joint sample consensus background model with four channels (red, green, blue and thermal) to accomplish the object detection and fusion of complementary information simultaneously, which lowers the computational cost of our method. Raw foreground segmentation is obtained in the thermal domain, making initial foreground more accurate. Meantime this can enhance the efficiency of further steps. Time out map (TOM) is utilized to deal with the problem that a newly exposed background is wrongly marked as foreground for a long time. In the updating phase, unlike most sample-based methods using first-in first-out policy, we intentionally employ a random update policy to reserve some older samples. That is, when a pixel is classified as background, we randomly pick up one of the background samples stored for the corresponding pixel to discard. In this manner, the backgrounds, occluded by slow moving foreground or temporally still foreground, can be recovered promptly when they reappear. Experimental results show that the proposed method can achieve accurate and precise detection results.

Optimization of Storage Method for Video Segment Captured by Embedded System in Camera

Video processing and cache capacity in embedded system in camera is the key to decide intelligence of shooting system. In particular, to meet the demand for continuous monitoring in video surveillance environment, the embedded system must be able to store the video in case of failure in network or server. In this paper, a storage method for videos with fixed size is proposed, which can effectively restrain occurrence of fragment during storage and raise I/O performance while ensuring continuity of monitoring. Furthermore, storage of video data based on H.264/AVC encoding system and its optimization are also discussed.

Video Data Hiding for Managing Privacy Information in Surveillance Systems

From copyright protection to error concealment, video data hiding has found usage in a great number of applications. In this work, we introduce the detailed framework of using data hiding for privacy information preservation in a video surveillance environment. To protect the privacy of individuals in a surveillance video, the images of selected individuals need to be erased, blurred, or re-rendered. Such video modifications, however, destroy the authenticity of the surveillance video. We propose a new rate-distortion-based compression-domain video data hiding algorithm for the purpose of storing that privacy information. Using this algorithm, we can safeguard the original video as we can reverse the modification process if proper authorization can be established. The proposed data hiding algorithm embeds the privacy information in optimal locations that minimize the perceptual distortion and bandwidth expansion due to the embedding of privacy data in the compressed domain. Both reversible and irreversible embedding techniques are considered within the proposed framework and extensive experiments are performed to demonstrate the effectiveness of the techniques.

Rule-based real-time detection of context-independent events in video shots

The purpose of this paper is to investigate a real-time system to detect context-independent events in video shots. We test the system in video surveillance environments with a fixed camera. We assume that objects have been segmented (not necessarily perfectly) and reason with their low-level features, such as shape, and mid-level features, such as trajectory, to infer events related to moving objects. Our goal is to detect generic events, i.e., events that are independent of the context of where or how they occur. Events are detected based on a formal definition of these and on approximate but efficient world models. This is done by continually monitoring changes and behavior of features of video objects. When certain conditions are met, events are detected. We classify events into four types: primitive, action, interaction, and composite. Our system includes three interacting video processing layers: enhancement to estimate and reduce additive noise, analysis to segment and track video objects, and interpretation to detect context-independent events. The contributions in this paper are the interpretation of spatio-temporal object features to detect context-independent events in real time, the adaptation to noise, and the correction and compensation of low-level processing errors at higher layers where more information is available. The effectiveness and real-time response of our system are demonstrated by extensive experimentation on indoor and outdoor video shots in the presence of multi-object occlusion, different noise levels, and coding artifacts.