Video Surveillance Applications Research Articles

Large-Scale Indoor Camera Positioning Using Fiducial Markers.

Estimating the pose of a large set of fixed indoor cameras is a requirement for certain applications in augmented reality, autonomous navigation, video surveillance, and logistics. However, accurately mapping the positions of these cameras remains an unsolved problem. While providing partial solutions, existing alternatives are limited by their dependence on distinct environmental features, the requirement for large overlapping camera views, and specific conditions. This paper introduces a novel approach to estimating the pose of a large set of cameras using a small subset of fiducial markers printed on regular pieces of paper. By placing the markers in areas visible to multiple cameras, we can obtain an initial estimation of the pair-wise spatial relationship between them. The markers can be moved throughout the environment to obtain the relationship between all cameras, thus creating a graph connecting all cameras. In the final step, our method performs a full optimization, minimizing the reprojection errors of the observed markers and enforcing physical constraints, such as camera and marker coplanarity and control points. We validated our approach using novel artificial and real datasets with varying levels of complexity. Our experiments demonstrated superior performance over existing state-of-the-art techniques and increased effectiveness in real-world applications. Accompanying this paper, we provide the research community with access to our code, tutorials, and an application framework to support the deployment of our methodology.

React: recognize every action everywhere all at once

In the realm of computer vision, Group Activity Recognition (GAR) plays a vital role, finding applications in sports video analysis, surveillance, and social scene understanding. This paper introduces Recognize Every Action Everywhere All At Once (REACT), a novel architecture designed to model complex contextual relationships within videos. REACT leverages advanced transformer-based models for encoding intricate contextual relationships, enhancing understanding of group dynamics. Integrated Vision-Language Encoding facilitates efficient capture of spatiotemporal interactions and multi-modal information, enabling comprehensive scene understanding. The model’s precise action localization refines joint understanding of text and video data, enabling precise bounding box retrieval and enhancing semantic links between textual descriptions and visual reality. Actor-Specific Fusion strikes a balance between actor-specific details and contextual information, improving model specificity and robustness in recognizing group activities. Experimental results demonstrate REACT’s superiority over state-of-the-art GAR approaches, achieving higher accuracy in recognizing and understanding group activities across diverse datasets. This work significantly advances group activity recognition, offering a robust framework for nuanced scene comprehension.

Synthetic Data for Video Surveillance Applications of Computer Vision: A Review

AbstractIn recent years, there has been a growing interest in synthetic data for several computer vision applications, such as automotive, detection and tracking, surveillance, medical image analysis and robotics. Early use of synthetic data was aimed at performing controlled experiments under the analysis by synthesis approach. Currently, synthetic data are mainly used for training computer vision models, especially deep learning ones, to address well-known issues of real data, such as manual annotation effort, data imbalance and bias, and privacy-related restrictions. In this work, we survey the use of synthetic training data focusing on applications related to video surveillance, whose relevance has rapidly increased in the past few years due to their connection to security: crowd counting, object and pedestrian detection and tracking, behaviour analysis, person re-identification and face recognition. Synthetic training data are even more interesting in this kind of application, to address further, specific issues arising, e.g., from typically unconstrained image or video acquisition conditions and cross-scene application scenarios. We categorise and discuss the existing methods for creating synthetic data, analyse the synthetic data sets proposed in the literature for each of the considered applications, and provide an overview of their effectiveness as training data. We finally discuss whether and to what extent the existing synthetic data sets mitigate the issues of real data, highlight existing open issues, and suggest future research directions in this field.

Effect of Positive-Negative Image Ratio on the Performance of Pedestrian Detection Model

Pedestrian detection holds significant importance in computer vision, finding applications in video surveillance, human-computer interaction, and autonomous vehicles. Surprisingly, there is a scarcity of research addressing the optimal ratio of positive to negative images for training detection models. This study endeavors to fill this research gap by exploring various detection models and determining the ideal ratio. Two distinct scenarios are investigated, each characterized by an equal total image count and an equivalent number of positive images sourced from CVC-14 night/visible, night/FIR, and INRIA databases. The study leverages the Histogram of Oriented Gradient, utilizing both Support Vector Machines and Medium Neural Networks to construct the detection models. Notably, the experiments reveal that the accuracy of the models remains relatively stable, even with an increase in the ratio of negative images. However, a noteworthy inverse relationship between sensitivity and specificity emerges as the ratio escalates. The findings, guided by the Youden Index, pinpoint the optimal training ratio for pedestrian detection models, falling within the range of 1:0.5 to 1:2In the CVC-14 nighttime database, the Youden index reached 100% when the model was trained with a 1:0.5 ratio using SVM, and a total of 4500 images were employed in the training process. On the other hand, in the INRIA dataset, the Youden index exhibited its highest value at 98.50%. This occurred when both SVM and a Medium neural network were utilized to train the model with a ratio of 1:2, utilizing a total of 3000 images for the training phase. It's worth highlighting that the processing time for SVM models lags behind that of Medium Neural Networks. This disparity arises from the heightened computational complexity inherent to medium-sized neural networks, making them computationally demanding compared to SVMs. This study contributes valuable insights into the nuanced relationship between image ratios and the performance of pedestrian detection models.

Instance-aware image dehazing

The existing deep learning-based image dehazing algorithms commonly employ an encoder–decoder structure to learn a direct mapping from hazy images to haze-free images. However, these state-of-the-art methods often fail to consider the varying contents of hazy images across different scenes, resulting in unsatisfactory dehazing outcomes. To address this issue, this paper attempts to integrate a novel instance-aware subnet into the classic encoder–decoder structure in order to achieve a clear separation between figure and background, conducting the selective incorporation of instance features into the dehazing network. Specifically, we introduce a novel architecture called the hybrid residual attention network, which is capable of separately extracting full-image features and instance-level features. This architecture incorporates attention mechanisms and a multi-scale dilated convolution structure, enabling adaptive perception of haze density in different scenes. Additionally, we introduce a global feature fusion subnet that employs a pixel attention structure to fuse features from the entire image and multiple individual instances, thus being aware of instance features. Compared to existing methods, our approach offers a major advantage in accurately estimating the haze density of individual instances, reducing color distortion, and mitigating noise amplification in the output images. Experimental results demonstrate that our method outperforms existing methods across different evaluation metrics and testing benchmarks. Therefore, we believe that our method will serve as a valuable addition to the current collection of artificial intelligence models and will benefit engineering applications in video surveillance and high-level computer vision tasks.

Adaptive imaging through dense dynamic scattering media using transfer learning

Imaging through scattering media is a long-standing challenge in optical imaging, holding substantial importance in fields like biology, transportation, and remote sensing. Recent advancements in learning-based methods allow accurate and rapid imaging through optically thick scattering media. However, the practical application of data-driven deep learning faces substantial hurdles due to its inherent limitations in generalization, especially in scenarios such as imaging through highly non-static scattering media. Here we utilize the concept of transfer learning toward adaptive imaging through dense dynamic scattering media. Our approach specifically involves using a known segment of the imaging target to fine-tune the pre-trained de-scattering model. Since the training data of downstream tasks used for transfer learning can be acquired simultaneously with the current test data, our method can achieve clear imaging under varying scattering conditions. Experiment results show that the proposed approach (with transfer learning) is capable of providing more than 5dB improvements when optical thickness varies from 11.6 to 13.1 compared with the conventional deep learning approach (without transfer learning). Our method holds promise for applications in video surveillance and beacon guidance under dense dynamic scattering conditions.

A Review Work: Human Action Recognition in Video Surveillance Using Deep Learning Techniques

Despite being extensively used in numerous uses, precise and effective human activity identification continues to be an interesting research issue in the area of vision for computers. Currently, a lot of investigation is being done on themes like pedestrian activity recognition and ways to recognize people's movements employing depth data, 3D skeletal data, still picture data, or strategies that utilize spatiotemporal interest points. This study aims to investigate and evaluate DL approaches for detecting human activity in video. The focus has been on multiple structures for detecting human activities that use DL as their primary strategy. Based on the application, including identifying faces, emotion identification, action identification, and anomaly identification, the human occurrence forecasts are divided into four different subcategories. The literature has been carried several research based on these recognitions for predicting human behavior and activity for video surveillance applications. The state of the art of four different applications' DL techniques is contrasted. This paper also presents the application areas, scientific issues, and potential goals in the field of DL-based human behavior and activity recognition/detection.

Sampling-Resilient Multi-Object Tracking

Multi-Object Tracking (MOT) is a cornerstone operator for video surveillance applications. To enable real-time processing of large-scale live video streams, we study an interesting scenario called down-sampled MOT, which performs object tracking only on a small subset of video frames. The problem is challenging for state-of-the-art MOT methods, which exhibit significant performance degradation under high frame reduction ratios. In this paper, we devise a sampling-resilient tracker with a novel sparse-observation Kalman filter (SOKF). It integrates an LSTM network to capture non-linear and dynamic motion patterns caused by sparse observations. Since the LSTM-based state transition is not compatible with the original noise estimation mechanism, we propose new estimation strategies based on Bayesian neural networks and derive the optimal Kalman gain for SOKF. To associate the detected bounding boxes robustly, we also propose a comprehensive similarity metric that systematically integrates multiple spatial matching signals. Experiments on three benchmark datasets show that our proposed tracker achieves the best trade-off between efficiency and accuracy. With the same tracking accuracy, we reduce the total processing time of ByteTrack by 2× in MOT17 and 3× in DanceTrack.

Human Activity Detection Events Through Human Eye Reflection using Bystander Analyzer

ABSTRACT Automatic video-based human activity recognition has shown promise and is mostly used in video surveillance applications for diverse purposes. However, there are still substantial performance issues that make real-world implementation challenging. The main barrier to the accurate detection of human movement remains to the perspective problem, which results from the fact that video sequences are frequently shot from random camera angles. Therefore, the main focus of this study is on how human eyesight reflects in different camera views that are used to detect human activity or identify intruders. First, a gaussian mixture model that previously used the Minimal Spanning Tree for segmentation is used for pre-processing. Using the pixel-based Kruskal methodology and this method, the input data set’s minimal weight is precisely determined. Segmentation comes next. Independent discriminant features are extracted using transformation using Karhunen-Loeve expansion, which isolates human behavior based on the Person Correlation Coefficient. To effectively identify data, Deep Lens Classifier is also employed to look for any suspect human behavior. With an impressive 78.8774% accuracy, 28.6961% sensitivity, 98.50% specificity, 75.6734% precision, 48.781% recall, and 65.10% F-measure, this approach is unique in the field of detecting human activity. Finally, our proposed system’s MATLAB performance retrieves the accurate detection.

Robust Object Detection Using Fire Hawks Optimizer with Deep Learning Model for Video Surveillance

In recent years, video surveillance has become an integral part of computer vision research, addressing a variety of challenges in security, memory management and content extraction from video sequences. This paper introduces the Robust Object Detection using Fire Hawks Optimizer with Deep Learning (ROD-FHODL) technique, a novel approach designed specifically for video surveillance applications. Combining object detection and classification the proposed technique employs a two-step procedure. Utilizing the power of the Mask Region-based Convolutional Neural Network (Mask-RCNN) for object detection, we optimize its hyperparameters using the Fire Hawks Optimizer (FHO) algorithm to improve its efficacy. Our experimental results on the UCSD dataset demonstrate the significant impact of the proposed work. It achieves an extraordinary RUNNT of 1.34[Formula: see text]s on the pedestrian-1 dataset, significantly outperforming existing models. In addition, the proposed system surpasses in accuracy, with a pedestrian-1 accuracy rate of 97.45% and Area Under the Curve (AUC) values of 98.92%. Comparative analysis demonstrates the superiority of the proposed system in True Positive Rate (TPR) versus False Positive Rate (FPR) across thresholds. In conclusion, the proposed system represents a significant advancement in video surveillance, offering advances in speed, precision and robustness that hold promise for enhancing security, traffic management and public space monitoring in smart city infrastructure and other applications.

Detecting multiple moving objects for unusual crowd activities detection in video surveillance system

Unusual crowd activity detection is a challenging problem in surveillance video applications because feature extraction is difficult process in crowded scenes. The main objective of this research work is to detect unusual crowd activities and to detect unusual splits of moving objects. Various methods have been employed to address these challenges. However, there is still a lack of appropriate handling of this problem due to frames having occlusion, noise, and congestion. This paper proposes a novel clustering approach to detect unusual crowd activities. The proposed method consists of five phases including foreground extraction, foreground enhancement, foreground estimation, clustering crowds, and the Unusual Crowd Activities (UCA) model. The UCA model can find unusual crowd activities and unusual splits of moving objects using the Laplacian Matrix formulation. Two public datasets viz. PETS 2009 and UMN dataset are used for evaluating the proposed methodology. To estimate the effectiveness of the proposed work, several unusual event detection methods are compared with the proposed work results. The experimental results revealed that the proposed method gives better results than the existing methods.

Bilingual video captioning model for enhanced video retrieval

Many video platforms rely on the descriptions that uploaders provide for video retrieval. However, this reliance may cause inaccuracies. Although deep learning-based video captioning can resolve this problem, it has some limitations: (1) traditional keyframe extraction techniques do not consider video length/content, resulting in low accuracy, high storage requirements, and long processing times; (2) Arabic language support in video captioning is not extensive. This study proposes a new video captioning approach that uses an efficient keyframe extraction method and supports both Arabic and English. The proposed keyframe extraction technique uses time- and content-based approaches for better quality captions, fewer storage space requirements, and faster processing. The English and Arabic models use a sequence-to-sequence framework with long short-term memory in both the encoder and decoder. Both models were evaluated on caption quality using four metrics: bilingual evaluation understudy (BLEU), metric for evaluation of translation with explicit ORdering (METEOR), recall-oriented understudy of gisting evaluation (ROUGE-L), and consensus-based image description evaluation (CIDE-r). They were also evaluated using cosine similarity to determine their suitability for video retrieval. The results demonstrated that the English model performed better with regards to caption quality and video retrieval. In terms of BLEU, METEOR, ROUGE-L, and CIDE-r, the English model scored 47.18, 30.46, 62.07, and 59.98, respectively, whereas the Arabic model scored 21.65, 36.30, 44.897, and 45.52, respectively. According to the video retrieval, the English and Arabic models successfully retrieved 67% and 40% of the videos, respectively, with 20% similarity. These models have potential applications in storytelling, sports commentaries, and video surveillance.

PARFormer: Transformer-Based Multi-Task Network for Pedestrian Attribute Recognition

Pedestrian attribute recognition (PAR) has received increasing attention because of its wide application in video surveillance and pedestrian analysis. Extracting robust feature representation is one of the key challenges in this task. The existing methods primarily rely on convolutional neural networks (CNNs) as the backbone network for feature extraction. However, these methods mainly focus on small discriminative regions while ignoring the global perspective. To overcome these limitations, we propose PARFormer, a pure transformer-based multi-task PAR network consisting of four modules. In the feature extraction module, we build a transformer-based strong baseline for feature extraction, which achieves competitive results on several PAR benchmarks compared with the existing CNN-based baseline methods. Since the PAR task is vulnerable to environmental factors, we enhance feature robustness in the feature processing module and propose an effective data augmentation strategy named batch random mask (BRM) block to reinforce the attentive feature learning of random patches. Furthermore, we propose a multi-attribute center loss (MACL) to augment the inter-attribute discriminability of feature representations. As viewpoints can affect some specific attributes, in the viewpoint perception module, we propose a multi-view contrastive loss (MVCL) that enables the network to exploit the viewpoint information. In the attribute recognition module, we alleviate the negative-positive imbalance problem to generate the attribute predictions. These modules interact and jointly learn a highly discriminative feature space and supervise the generation of the final features. Extensive experimental results show that the proposed PARFormer network performs well compared to the state-of-the-art methods on several public datasets, including PETA, RAP, and PA100K. Code will be released at https: //github.com/xwf199/PARFormer.

Optimization of the solar energy storage capacity for a monitoring UAV

Unmanned aerial vehicles integrate propulsion systems, communication modules, and sensors, allowing an operator to perform autonomous or remote-controlled flight actions. UAVs provide important advantages for exploring remote locations due to their cost-effectiveness and versatility compared to manned aircraft. However, addressing safety and flight autonomy challenges remains necessary. This paper analyzes and proposes the integration of a photovoltaic solar system to power UAV devices. Through a brief analysis of the aerodynamic model and the wing profile, a consolidation of the solar cells has been achieved without compromising efficiency in-flight maneuvers. Furthermore, an analysis is conducted on the potential of using photovoltaic solar resources in fixed-wing aircraft. The research aims to determine the optimal wing surface area required for video surveillance applications. The current model under discussion is a glider-type system that incorporates two distinct systems, one for video transmission and the other for telemetry data acquisition. An analysis of the battery charge and discharge pattern was carried out using computer simulation tools. This analysis aimed to optimize the battery charging process by integrating photovoltaic cells. The results and conclusions of the tests are described in the final section of the paper.

A chaotic-based watermarking scheme for ensuring integrity of a face recognition system in public large gathering scenario

Secure data transmission is critical in public video surveillance applications such as tracking missing persons in large crowd gatherings. It may help the security personnel to monitor & record the live feeds from multiple cameras. However, this live streaming could be falsified. Various cryptographic techniques have been proposed to ensure the integrity of transmitted images, but continuous attacks still pose serious challenges to ensuring the integrity and authenticity of transmitted images. This work proposes a new chaotic-based watermarking technique to ensure the integrity of the transmitted video frames, and since the frames come from multiple cameras, the scheme should be as simple as possible and as fast as possible, the simplicity manifests in designing a fragile watermarking scheme therefore, the watermark is to be embedded in the spatial domain by direct manipulation of the values of the cover’s pixels. To ensure the security of the proposed scheme, chaotic maps have been deployed to reach an acceptable level of security with minimum computational overhead. The proposed scheme is novel in design in terms of the chaotic maps used and it is tested in challenging large gathering scenario. Experimental results demonstrate that the proposed method successfully locates any falsification in the transmitted images, and the performance of the scheme evaluates different types of attacks. Our testing results in good rate tamper of detection with TPR=99.2005, TNR=99.0777 and with a good quality of imperceptibility for the watermarked image with PSNR=43.82dB, SSIM=00.9955.

Introducing shape priors in Siamese networks for image classification

The efficiency of deep neural networks is increasing, and so is the amount of annotated data required for training them. We propose a solution improving the learning process of a classification network with less labeled data. Our approach is to inform the classifier of the elements it should focus on to make its decision by supplying it with some shape priors. These shape priors are expressed as binary masks, giving a rough idea of the shape of the relevant elements for a given class. We resort to Siamese architecture and feed it with image/mask pairs. By inserting shape priors, only the relevant features are retained. This provides the network with significant generalization power without requiring a specific domain adaptation step. This solution is tested on some standard cross-domain digit classification tasks and on a real-world video surveillance application. Extensive tests show that our approach outperforms the classical classifier by generating a good latent space with less training data. Code is available at https://github.com/halqasir/MG-Siamese.

AI-driven behavior biometrics framework for robust human activity recognition in surveillance systems

The integration of artificial intelligence (AI) into human activity recognition (HAR) in smart surveillance systems has the potential to revolutionize behavior monitoring. These systems analyze an individual’s physiological or behavioral features to continuously monitor and identify any unusual or suspicious activity in video streams, thereby improving security and surveillance measures. Traditional surveillance systems often rely on manual human monitoring, which is resource-intensive, error-prone, and time-consuming. To address these limitations, computer vision-based behavior biometrics has emerged as a solution for secure video surveillance applications. However, implementing automated HAR in real-world scenarios is challenging due to the diversity of human behavior, complex spatiotemporal patterns, varying viewpoints, and cluttered backgrounds. To tackle these challenges, an AI-based behavior biometrics framework is introduced that is based on a dynamic attention fusion unit (DAFU) followed by a temporal-spatial fusion (TSF) network to effectively recognize human activity in surveillance systems. In the first phase of the proposed framework, a lightweight EfficientNetB0 backbone is enhanced by the DAFU to extract human-centric salient features using a unified channel-spatial attention mechanism. In the second phase, the DAFU features with fixed sequence lengths are passed to the proposed TSF network to capture the temporal, spatial, and behavioral dependencies in video data streams. The integration of Echo-ConvLSTM in the TSF further enhances accuracy and robustness by combining temporal dependencies from the echo state network with spatial and temporal dependencies from the convolutional long short-term memory (ConvLSTM). The proposed AI-based behavior biometrics framework is evaluated using four publicly available HAR datasets (UCF101, HMDB51, UCF50, and YouTube Action), yielding higher accuracies of 98.734%, 80.342%, 98.987%, and 98.927%, which demonstrate the superior performance when compared with state-of-the-art (SOTA) methods.

An improved YOLOv5 for real-time human detection in infrared images

<span>The recent prominence of using infrared technology for human detection has garnered attention, particularly for applications in intelligent video surveillance and self-driving systems. This technology offers advantages in adverse weather conditions and night vision. However, within deep learning, the challenge of variable illumination during human detection persists. This study presents a novel method for identifying individuals in thermal images by enhancing the you only look once (YOLOv5s) algorithm. The approach incorporates the Bi-directional feature pyramid network (BiFPN) and the convolutional block attention module (CBAM) to improve the model’s feature integration and extraction capabilities. Evaluation on established thermal imaging datasets confirms the method’s superiority over state-of-the-art convolutional neural networks (CNN) based techniques, achieving remarkable precision 99.1% and recall 96.9%.</span>

Shot classification for human behavioural analysis in video surveillance applications

Human behavior analysis plays a vital role in ensuring security and safety of people in crowded public places against diverse contexts like theft detection, violence prevention, explosion anticipation etc. Analysing human behaviour by classifying of videos in to different shot types helps in extracting appropriate behavioural cues. Shots indicates the subject size within the frame and the basic camera shots include: the close-up, medium shot, and the long shot. If the video is categorised as Close-up shot type, investigating emotional displays helps in identifying criminal suspects by analysing the signs of aggressiveness and nervousness to prevent illegal acts. Mid shot can be used for analysing nonverbal communication like clothing, facial expressions, gestures and personal space. For long shot type, behavioural analysis is by extracting the cues from gait and atomic action displayed by the person. Here, the framework for shot scale analysis for video surveillance applications is by using Face pixel percentage and deep learning based method. Face Pixel ratio corresponds to the percentage of region occupied by the face region in a frame. The Face pixel Ratio is thresholded with predefined threshold values and grouped into Close-up shot, mid shot and long shot categories. Shot scale analysis based on transfer learning utilizes effective pre-trained models that includes AlexNet, VGG Net, GoogLeNet and ResNet. From experimentation, it is observed that, among the pre-trained models used for experimentation GoogLeNet tops with the accuracy of 94.61%.

A new framework for deep learning video based Human Action Recognition on the edge

Nowadays, video surveillance systems are commonly found in most public and private spaces. These systems typically consist of a network of cameras that feed into a central node. However, the processing aspect is evolving towards distributed approaches, leveraging edge-computing. These distributed systems are capable of effectively addressing the detection of people or events at each individual node. Most of these systems, rely on the use of deep-learning and segmentation algorithms which enable them to achieve high performance, but usually with a significant computational cost, hindering real-time execution. This paper presents an approach for people detection and action recognition in the wild, optimized for running on the edge, and that is able to work in real-time, in an embedded platform. Human Action Recognition (HAR) is performed by using a Recurrent Neural Network (RNN), specifically a Long Short-Term Memory (LSTM). The input to the LSTM is an ad-hoc, lightweight feature vector obtained from the bounding box of each detected person in the video surveillance image. The resulting system is highly portable and easily scalable, providing a powerful tool for real-world video surveillance applications (in the wild and real-time action recognition). The proposal has been exhaustively evaluated and compared against other state-of-the-art (SOTA) proposals in five datasets, including four widely used (KTH, WEIZMAN, WVU, IXMAX) and a novel one (GBA) recorded in the wild, that includes several people performing different actions simultaneously. The obtained results validate the proposal, since it achieves SOTA accuracy within a much more complicated video surveillance real scenario, and using a lightweight embedded hardware.