Real Surveillance Scenarios Research Articles

Cross-camera Trajectories Help Person Retrieval in a Camera Network.

We are concerned with retrieving a query person from multiple videos captured by a non-overlapping camera network. Existing methods often rely on purely visual matching or consider temporal constraints but ignore the spatial information of the camera network. To address this issue, we propose a pedestrian retrieval framework based on cross-camera trajectory generation that integrates both temporal and spatial information. To obtain pedestrian trajectories, we propose a novel cross-camera spatio-temporal model that integrates pedestrians' walking habits and the path layout between cameras to form a joint probability distribution. Such a cross-camera spatio-temporal model can be specified using sparsely sampled pedestrian data. Based on the spatio-temporal model, cross-camera trajectories can be extracted by the conditional random field model and further optimised by restricted non-negative matrix factorization. Finally, a trajectory re-ranking technique is proposed to improve the pedestrian retrieval results. To verify the effectiveness of our method, we construct the first cross-camera pedestrian trajectory dataset, the Person Trajectory Dataset, in real surveillance scenarios. Extensive experiments verify the effectiveness and robustness of the proposed method.

Multi-ColorGAN: Few-shot vehicle recoloring via memory-augmented networks

Despite the notable successes of Generative adversarial networks (GANs) achieved to date, applying them to real-world problems still poses significant challenges. In real traffic surveillance scenarios, for the task of generating images of multiple color of truck heads and cars without changing textures and license plates, conditional image generation hardly manipulate the generated images by the color attribute. Image style transfer methods inevitably produce color smearing. Even state-of-the-art methods of disentangled representation learning (e.g. MixNMatch) cannot disentangle colors individually, ensuring that irrelevant factors, such as texture remain the same. To solve this problem, we present an approach called Multi-ColorGAN based on memory-augmented networks for multi-color real vehicle coloring/generation with limited data. In particular, our model could filter out unwanted color changes in specific areas with a simple but effective method called Fusion Module, and generate more natural color images. Experiments on three vehicle image benchmarks and a new truck image dataset are conducted to evaluate the proposed Multi-ColorGAN compared to state-of-the-art.

Deep autoencoder for false positive reduction in handgun detection

In an object detection system, the main objective during training is to maintain the detection and false positive rates under acceptable levels when the model is run over the test set. However, this typically translates into an unacceptable rate of false alarms when the system is deployed in a real surveillance scenario. To deal with this situation, which often leads to system shutdown, we propose to add a filter step to discard part of the new false positive detections that are typical of the new scenario. This step consists of a deep autoencoder trained with the false alarm detections generated after running the detector over a period of time in the new scenario. Therefore, this step will be in charge of determining whether the detection is a typical false alarm of that scenario or whether it is something anomalous for the autoencoder and, therefore, a true detection. In order to decide whether a detection must be filtered, three different approaches have been tested. The first one uses the autoencoder reconstruction error measured with the mean squared error to make the decision. The other two use the k-NN (k-nearest neighbors) and one-class SVMs (support vector machines) classifiers trained with the autoencoder vector representation. In addition, a synthetic scenario has been generated with Unreal Engine 4 to test the proposed methods in addition to a dataset with real images. The results obtained show a reduction in the number of false positives between 22.5% and 87.2% and an increase in the system’s precision of 1.2%-47% when the autoencoder is applied.

Deep hard modality alignment for visible thermal person re-identification

Visible Thermal Person Re-Identification (VTReID) is essentially a cross-modality problem and widely encountered in real night-time surveillance scenarios, which is still in need of vigorous performance improvement. In this work, we design a simple but effective Hard Modality Alignment Network (HMAN) framework to learn modality-robust features. Since current VTReID works do not consider the cross-modality discrepancy imbalance, their models are likely to suffer from the selective alignment behavior. To solve this problem, we propose a novel Hard Modality Alignment (HMA) loss to simultaneously balance and reduce the modality discrepancies. Specifically, we mine the hard feature subspace with large modality discrepancies and abandon the easy feature subspace with small modality discrepancies to make the modality distributions more distinguishable. For mitigating the discrepancy imbalance, we pay more attention on reducing the modality discrepancies of the hard feature subspace than that of the easy feature subspace. Furthermore, we propose to jointly relieve the modality heterogeneity of global and local visual semantics to further boost the cross-modality retrieval performance. This paper experimentally demonstrates the effectiveness of the proposed method, achieving superior performance over the state-of-the-art methods on RegDB and SYSU-MM01 datasets.

Person Re-Identification in Aerial Imagery

Nowadays, with the rapid development of consumer Unmanned Aerial Vehicles (UAVs), visual surveillance by utilizing the UAV platform has been very attractive. Most of the research works for UAV captured visual data are mainly focused on the tasks of object detection and tracking. However, limited attention has been paid to the task of person Re-identification (ReID) which has been widely studied in ordinary surveillance cameras with fixed emplacements. In this paper, to facilitate the research of person ReID in aerial imagery, we collect a large scale airborne person ReID dataset named as Person ReID in Aerial Imagery (PRAI-1581), which consists of 39,461 images of 1581 person identities. The images of the dataset are shot by two DJI consumer UAVs flying at an altitude ranging from 20 to 60 meters above the ground, which covers most of the real UAV surveillance scenarios. In addition, we propose to utilize subspace pooling of convolution feature maps to represent the input person images. Our method can learn a discriminative and compact feature representation for ReID in aerial imagery and can be trained in an end-to-end fashion efficiently. We conduct extensive experiments on the proposed dataset and the experimental results demonstrate that re-identifying persons in aerial imagery is a challenging problem, where our method performs favorably against state of the arts.

Multi-View Vehicle Type Recognition With Feedback-Enhancement Multi-Branch CNNs

Vehicle type recognition (VTR) is a quite common requirement and one of the key challenges in real surveillance scenarios, such as intelligent traffic and unmanned driving. Usually coarse-grained and fine-grained VTRs are applied in different applications, and the challenge from multiple viewpoints is critical for both cases. In this paper, we propose a feedback-enhancement multi-branch CNN (FM-CNN) to solve the challenge in these two cases. The proposed FM-CNN takes three derivatives of an image as input and leverages the advantages of hierarchical details, feedback enhancement, model average, and stronger robustness to translation and mirroring. A single global cross-entropy loss is insufficient to train such a complex CNN and so we add extra branch losses to enhance feedbacks to each branch. Though reusing pre-trained parameters, we propose a novel parameter update method to adapt FM-CNN to task-specific local visual patterns and global information in new datasets. To test the effectiveness of FM-CNN, we create our own multi-view VTR (MVVTR) data set since there are no such data sets available. And, for fine-grained VTR, we use the CompCars data set. Compared with state-of-the-art classification solutions without special preprocessing, the proposed FM-CNN demonstrates better performance in both coarse-grained and fine-grained scenarios. For coarse-grained VTR, it achieves 94.9% Top-1 accuracy on the MVVTR data set. For fine-grained VTR, it achieves 91.0% Top-1 and 97.8% Top-5 accuracies on the CompCars data set.

A Richly Annotated Pedestrian Dataset for Person Retrieval in Real Surveillance Scenarios.

Retrieving specific persons with various types of queries, e.g., a set of attributes or a portrait photo has great application potential in large-scale intelligent surveillance systems. In this paper, we propose a richly annotated pedestrian (RAP) dataset which serves as a unified benchmark for both attribute-based and image-based person retrieval in real surveillance scenarios. Typically, previous datasets have three improvable aspects, including limited data scale and annotation types, heterogeneous data source, and controlled scenarios. Differently, RAP is a large-scale dataset which contains 84928 images with 72 types of attributes and additional tags of viewpoint, occlusion, body parts, and 2589 person identities. It is collected in the real uncontrolled scene and has complex visual variations in pedestrian samples due to the change of viewpoints, pedestrian postures, and cloth appearance. Towards a high-quality person retrieval benchmark, an amount of state-of-the-art algorithms on pedestrian attribute recognition and person re-identification (ReID), are performed for quantitative analysis with three evaluation tasks, i.e., attribute recognition, attribute-based and image-based person retrieval, where a new instance-based metric is proposed to measure the dependency of the prediction of multiple attributes. Finally, some interesting problems, e.g., the joint feature learning of attribute recognition and ReID, and the problem of cross-day person ReID, are explored to show the challenges and future directions in person retrieval.

Video-Based Person Re-Identification via Self Paced Weighting

Person re-identification (re-id) is a fundamental technique to associate various person images, captured by differentsurveillance cameras, to the same person. Compared to the single image based person re-id methods, video-based personre-id has attracted widespread attentions because extra space-time information and more appearance cues that can beused to greatly improve the matching performance. However, most existing video-based person re-id methods equally treatall video frames, ignoring their quality discrepancy caused by object occlusion and motions, which is a common phenomenonin real surveillance scenario. Based on this finding, we propose a novel video-based person re-id method via self paced weighting (SPW). Firstly, we propose a self paced outlier detection method to evaluate the noise degree of video sub sequences. Thereafter, a weighted multi-pair distance metric learning approach is adopted to measure the distance of two person image sequences. Experimental results on two public datasets demonstrate the superiority of the proposed method over current state-of-the-art work.

Anomaly detection with a moving camera using multiscale video analysis

This paper addresses the problem of abandoned object detection in a cluttered environment using a camera moving along a straight track. The developed system compares captured images to a previously recorded reference video, thus requiring proper temporal alignment and geometric registration between the two signals. A real-time constraint is imposed onto the system to allow an effective surveillance capability in practical situations. In this paper, we propose to deal with the simultaneous detection of objects of different sizes using a multiresolution approach together with normalized cross-correlation and a voting step. In order to develop and properly assess the proposed method we designed a database recorded in a real surveillance scenario, consisting of an industrial plant containing a large number of pipes and rotating machines. Also, we have devised a systematic parameter tuning routine that allows the system to be adapted to different scenarios. We have validated it using the designed database. The obtained results are quite effective, achieving real-time, robust abandoned object detection in an industrial plant scenario.

BLACKFACE SURVEILLANCE CAMERA DATABASE FOR EVALUATING FACE RECOGNITION IN LOW QUALITY SCENARIOS

Many face recognition algorithms perform poorly in real life surveillance scenarios because they were tested with datasets that are already biased with high quality images and certain ethnic or racial types. In this paper a black face surveillance camera (BFSC) database was described, which was collected from four low quality cameras and a professional camera. There were fifty (50) random volunteers and 2,850 images were collected for the frontal mugshot, surveillance (visible light), surveillance (IR night vision), and pose variations datasets, respectively. Images were taken at distance 3.4, 2.4, and 1.4 metres from the camera, while the pose variation images were taken at nine distinct pose angles with an increment of 22.5 degrees to the left and right of the subject. Three Face Recognition Algorithms (FRA), a commercially available Luxand SDK, Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA) were evaluated for performance comparison in low quality scenarios. Results obtained show that camera quality (resolution), face-to-camera distance, average recognition time, lighting conditions and pose variations all affect the performance of FRAs. Luxand SDK, PCA and LDA returned an overall accuracy of 97.5%, 93.8% and 92.9% after categorizing the BFSC images into excellent, good and acceptable quality scales.

Robust Face Super-Resolution via Locality-Constrained Low-Rank Representation

Learning-based face super-resolution relies on obtaining accurate a priori knowledge from the training data. Representation-based approaches (e.g., sparse representation-based and neighbor embedding-based schemes) decompose the input images using sophisticated regularization techniques. They give reasonably good reconstruction performance. However, in real application scenarios, the input images are often noisy, blurry, or suffer from other unknown degradations. Traditional face super-resolution techniques treat image noise at the pixel level without considering the underlying image structures. In order to rectify this shortcoming, we propose in this paper a unified framework for representation-based face super-resolution by introducing a locality-constrained low-rank representation (LLR) scheme to reveal the intrinsic structures of input images. The low-rank representation part of LLR clusters an input image into the most accurate subspace from a global dictionary of atoms, while the locality constraint enables recovery of local manifold structures from local patches. In addition, low-rank, sparsity, locality, accuracy, and robustness of the representation coefficients are exploited in LLR via regularization. Experiments on the FEI, CMU face database, and real surveillance scenario show that LLR outperforms the state-of-the-art face super-resolution algorithms (e.g., convolutional neural network-based deep learning) both objectively and subjectively.

Fully-automated person re-identification in multi-camera surveillance system with a robust kernel descriptor and effective shadow removal method

In this paper, a fully-automated person Re-ID (Re-identification) system is proposed for real scenarios of human tracking in non-overlapping camera network. The system includes two phases of human detection and Re-ID. The human ROIs (Regions of Interest) are extracted from human detection phase and then feature extraction is done on these ROIs in order to build human descriptor for Re-ID. Unlike other approaches which deal with manually-cropped human ROIs for person Re-ID, in this system, the person identity is determined based on the human ROIs extracted automatically by a combined method of human detection. Two main contributions are proposed on both phases of human detection and Re-ID in order to enhance the performance of person Re-ID system. First, an effective shadow removal method based on score fusion of density matching is proposed to get better human detection results. Second, a robust KDES (Kernel DEScriptor) is extracted from human ROI for person classification. Additionally, a new person Re-ID dataset is built in real surveillance scenarios from multiple cameras. The experiments on benchmark datasets and our own dataset show that the person Re-ID results using the proposed solutions outperform some of the state-of-the-art methods.

A scalable and flexible framework for smart video surveillance

In the last years, the number of surveillance cameras placed in public locations has increase vastly and as consequence, a huge amount of visual data is generated every minute. In general, this data is analyzed manually, a challenging task which is labor intensive and prone to errors. Therefore, automatic approaches must be employed to enable the processing of the data, so that human operators only need to reason about selected portions. Computer vision problems focused on solving problems in the domain of visual surveillance have been developed aiming at finding accurate and efficient solutions. The main goal of such systems is to analyze the scene focusing on the detection and recognition of suspicious activities performed by humans in the scene, so that the security staff can pay closer attention to these preselected activities. However, these systems are rarely tackled in a scalable manner. Before developing a full surveillance system, several problems have to be solved, which are usually solved individually. However, in a real surveillance scenario, these problems have to be solved in sequence considering only videos as the input. With that in mind, this work proposes a framework for scalable video analysis called Smart Surveillance Framework (SSF) to allow researchers to implement their solutions to the surveillance problems as a sequence of processing modules that communicates through a shared memory.

Robust multiple cameras pedestrian detection with multi-view Bayesian network

Multi-camera pedestrian detection is the challenging problem in the field of surveillance video analysis. However, existing approaches may produce “phantoms” (i.e., fake pedestrians) due to the heavy occlusions in real surveillance scenario, while calibration errors and the diverse heights of pedestrians may also heavily decrease the detection performance. To address these problems, this paper proposes a robust multiple cameras pedestrian detection approach with multi-view Bayesian network model (MvBN). Given the preliminary results obtained by any multi-view pedestrian detection method, which are actually comprised of both real pedestrians and phantoms, the MvBN is used to model both the occlusion relationship and the homography correspondence between them in all camera views. As such, the removal of phantoms can be formulated as an MvBN inference problem. Moreover, to reduce the influence of the calibration errors and keep robust to the diverse heights of pedestrians, a height-adaptive projection (HAP) method is proposed to further improve the detection performance by utilizing a local search process in a small neighborhood of heights and locations of the detected pedestrians. Experimental results on four public benchmarks show that our method outperforms several state-of-the-art algorithms remarkably and demonstrates high robustness in different surveillance scenes.

English

A long-distance imaging system can be strongly affected by atmospheric turbulence. Here a novel method is suggested for justifying the effects of atmospheric distortion on practical images, especially airborne turbulence which can cruelly corrupt a region of interest (ROI). In order to extract precise details about substance behind the distorted layer, a simple and capable frame selection method is proposed to select informative ROIs only from good worth frames. The ROIs in each frame are then registered to further reduce offsets and distortions. The space-varying alteration problem is solved using region-level fusion based on the double density- tree discrete wavelet transform DD-DWT. Finally, for applying double density dual tree wavelet transform is coming to better results. This is capable of estimating the quality in both full and no reference scenarios. The proposed method is shown appreciably to outperform accessible methods, providing enhanced situational attentiveness in a range of real time surveillance scenarios.

Robust object tracking via multi-feature adaptive fusion based on stability: contrast analysis

Object tracking under complex circumstances is a challenging task because of background interference, obstacle occlusion, object deformation, etc. Given such conditions, robustly detecting, locating, and analyzing a target through single-feature representation are difficult tasks. Global features, such as color, are widely used in tracking, but may cause the object to drift under complex circumstances. Local features, such as HOG and SIFT, can precisely represent rigid targets, but these features lack the robustness of an object in motion. An effective method is adaptive fusion of multiple features in representing targets. The process of adaptively fusing different features is the key to robust object tracking. This study uses a multi-feature joint descriptor (MFJD) and the distance between joint histograms to measure the similarity between a target and its candidate patches. Color and HOG features are fused as the tracked object of the joint representation. This study also proposes a self-adaptive multi-feature fusion strategy that can adaptively adjust the joint weight of the fused features based on their stability and contrast measure scores. The mean shift process is adopted as the object tracking framework with multi-feature representation. The experimental results demonstrate that the proposed MFJD tracking method effectively handles background clutter, partial occlusion by obstacles, scale changes, and deformations. The novel method performs better than several state-of-the-art methods in real surveillance scenarios.

Monocular 3-D Gait Tracking in Surveillance Scenes

Gait recognition can potentially provide a noninvasive and effective biometric authentication from a distance. However, the performance of gait recognition systems will suffer in real surveillance scenarios with multiple interacting individuals and where the camera is usually placed at a significant angle and distance from the floor. We present a methodology for view-invariant monocular 3-D human pose tracking in man-made environments in which we assume that observed people move on a known ground plane. First, we model 3-D body poses and camera viewpoints with a low dimensional manifold and learn a generative model of the silhouette from this manifold to a reduced set of training views. During the online stage, 3-D body poses are tracked using recursive Bayesian sampling conducted jointly over the scene's ground plane and the pose-viewpoint manifold. For each sample, the homography that relates the corresponding training plane to the image points is calculated using the dominant 3-D directions of the scene, the sampled location on the ground plane and the sampled camera view. Each regressed silhouette shape is projected using this homographic transformation and is matched in the image to estimate its likelihood. Our framework is able to track 3-D human walking poses in a 3-D environment exploring only a 4-D state space with success. In our experimental evaluation, we demonstrate the significant improvements of the homographic alignment over a commonly used similarity transformation and provide quantitative pose tracking results for the monocular sequences with a high perspective effect from the CAVIAR dataset.

Activity clustering for anomaly detection

This paper aims to address the problem of clustering activities captured in surveillance videos for the applications of online normal activity recognition and anomaly detection. A novel framework is developed for automatic activity modelling and anomaly detection without any manual labelling of the training data set. The framework consists of the following key components: 1 Drawing from natural language processing, we introduce a compact and effective activity representation method as a stochastic sequence of spatio-temporal actions, where we analyse the global structural information of activities using their local action statistics. 2 The natural grouping of activities is discovered through a novel clustering algorithm with unsupervised model selection, named latent Dirichlet Markov clustering LDMC. The approach builds on hidden Markov models HMMs and latent Dirichlet allocation LDA, and overcomes their drawbacks on accuracy, robustness and computational efficiency. 3 A runtime accumulative anomaly measure is introduced to detect abnormal activity, whereas normal activities are recognised when sufficient visual evidence has become available based on an online likelihood ratio test LRT method. This ensures robust and reliable anomaly detection and normal activity recognition at the shortest possible time. Experimental results demonstrate the effectiveness and robustness of our approach using noisy and sparse data sets collected from real surveillance scenarios.

Cooperative situation assessment in a maritime scenario

In large-scale, complex domains such as space defense and security systems, situation assessment and decision making are evolving from centralized models to high-level, net-centric models. In this context, collaboration among the many actors involved in the situation assessment process is critical to achieve a prompt reaction as needed in the operational scenario. In this paper, we propose a multiagent-based approach to situation assessment, where agents cooperate by sharing local information to reach a common and coherent assessment of situations. Specifically, we characterize situation assessment as a classification process based on OWL ontology reasoning, and we provide a protocol for cooperative multiagent situation assessment, which allows the agents to achieve coherent high-level conclusions. We validate our approach in a real maritime surveillance scenario, where our prototype system effectively supports the user in detecting and classifying potential threats; moreover, our distributed solution performs comparably to a centralized method, while preserving independence of decision makers and dramatically reducing the amount of communication required. © 2012 Wiley Periodicals, Inc. © 2012 Wiley Periodicals, Inc.

Human behavior clustering for anomaly detection

This paper aims to address the problem of modeling human behavior patterns captured in surveillance videos for the application of online normal behavior recognition and anomaly detection. A novel framework is developed for automatic behavior modeling and online anomaly detection without the need for manual labeling of the training data set. The framework consists of the following key components. 1) A compact and effective behavior representation method is developed based on spatial-temporal interest point detection. 2) The natural grouping of behavior patterns is determined through a novel clustering algorithm, topic hidden Markov model (THMM) built upon the existing hidden Markov model (HMM) and latent Dirichlet allocation (LDA), which overcomes the current limitations in accuracy, robustness, and computational efficiency. The new model is a four-level hierarchical Bayesian model, in which each video is modeled as a Markov chain of behavior patterns where each behavior pattern is a distribution over some segments of the video. Each of these segments in the video can be modeled as a mixture of actions where each action is a distribution over spatial-temporal words. 3) An online anomaly measure is introduced to detect abnormal behavior, whereas normal behavior is recognized by runtime accumulative visual evidence using the likelihood ratio test (LRT) method. Experimental results demonstrate the effectiveness and robustness of our approach using noisy and sparse data sets collected from a real surveillance scenario.