Large-scale Video Surveillance Research Articles

Joint metric learning of local and global features for vehicle re-identification

Vehicle re-identification (ReID) means to identify the target vehicle in large-scale surveillance videos captured by multiple cameras, where robust and distinctive visual features of vehicles are critical to the performance. Recently, the researchers have approached the problem with attention based models. However, most of these models use strongly-supervised methods, which rely on expensive extra labels, e.g., keypoints(vehicle wheels , logo and lamps) and attributes(e.g., color and type). Therefore, we propose a joint metric learning approach to solve the problem. We present an end-to-end Partition and Fusion Multi-branch Network (PFMN), a novel approach to effectively learn discriminative features without any annotations or additional attributes. For hard samples, which means different vehicles with similar appearance or the same vehicle with different appearances, a novel variant of hard sampling triplet loss is proposed. Based on extensive experiments, we have proved the effectiveness of our proposed method. On the challenging public data sets VeRi-776 and VehicleID, our model outperforms most state-of-the-art algorithms on mAP and rank-1. Especially on mINP, which measures the cost of model retrieval hard samples, we can achieve a significant improvement.

Target Tracking and 3D Trajectory Reconstruction Based on Multicamera Calibration

In traffic scenarios, vehicle trajectories can provide almost all the dynamic information of moving vehicles. Analyzing the vehicle trajectory in the monitoring scene can grasp the dynamic road traffic information. Cross-camera association of vehicle trajectories in multiple cameras can break the isolation of target information between single cameras and obtain the overall road operation conditions in a large-scale video surveillance area, which helps road traffic managers to conduct traffic analysis, prediction, and control. Based on the framework of DBT automatic target detection, this paper proposes a cross-camera vehicle trajectory correlation matching method based on the Euclidean distance metric correlation of trajectory points. For the multitarget vehicle trajectory acquired in a single camera, we first perform 3D trajectory reconstruction based on the combined camera calibration in the overlapping area and then complete the similarity association between the cross-camera trajectories and the cross-camera trajectory update, and complete the trajectory transfer of the vehicle between adjacent cameras. Experiments show that the method in this paper can well solve the problem that the current tracking technology is difficult to match the vehicle trajectory under different cameras in complex traffic scenes and essentially achieves long-term and long-distance continuous tracking and trajectory acquisition of multiple targets across cameras.

Metadata based need-to-know view in large-scale video surveillance systems

Large-scale video surveillance systems are increasingly seen as the answer to problems concerning public safety, law enforcement, and situational awareness in public places. However, the unauthorized use of personal information derived from video data can be harmful. To preserve privacy, it is important to understand what type of personal information is contained in video surveillance data, and how much of that information is essential for an observer to achieve her authorized purpose. The purpose of the observer is described in terms similar to the information extracted by video surveillance systems, so they can be compared. This helps identify what type of information is better suited to control the flow of information to multiple observers, without compromising the privacy of the individuals. This paper presents a privacy-aware and need-to-know access control framework built on fine-grained data properties, extracted from surveillance data, which must conform to the explicitly defined purpose of the observers.

The Design and Implementation of a Dashboard Web-Based Video Surveillance in OpenStack Swift

Abstract The purpose aims at deploying a fully functional OpenStack cluster with different settings and operational environments, followed by an in-depth analysis of its network performance and explanation of what happens behind the scene. Where OpenStack Virtual Machines perform these services for data storage files are stored in an authenticated cloud storage service center. The methodology used presents various scenarios using OpenStack Virtual machines and cloud-based to improve network performance video surveillance backend system architecture is developed with the Sensor web Enablement framework and cloud-based data storage. Data is obtained through simulators on camera/edge devices, file and metadata sharing, and fast cloud base storage. This study aims to compare the performance load resulting from large-scale video surveillance with small ones to show satisfactory performance, scalability, and good reliability from cloud-based storage systems for smart MCS. The most important feature is storage is adjusted dynamically, and there won’t be any worry about space being inadequate or wasted. This paper presents a solution for deploying web applications for CCTV video surveillance, an object cloud storage service system based on the open-source cloud operating system OpenStack Swift. It’s working in the OpenStack swift on IaaS and allocates resources for storage original data by multiple camera surveillance (MCS).

Algorithm for Counting Cars in Large-scale Video Surveillance Systems

The article deals with the problem of counting cars in large-scale video surveillance systems. The proposed method is based on car tracking and counting the number of tracks intersecting the given signal line. We use a distributed tracking algorithm. It reduces the amount of necessary computational resources and increases performance up to realtime by detecting vehicles in a sparse set of frames. We adapted and modified the approach previously proposed for people tracking. Proposed improvement of the speed estimation module and refinement of the motion model reduced the detection frequency by 3 times. The experimental evaluation shows that the proposed algorithm allows reaching an acceptable counting quality with a detection frequency of 3 Hz.

An effective hybrid pruning architecture of dynamic convolution for surveillance videos

The large-scale surveillance videos analysis becomes important as the development of the intelligent city; however, the heavy computational resources necessary for the state-of-the-art deep learning model makes real-time processing hard to be implemented. As the characteristic of high scene similarity generally existing in surveillance videos, we propose an effective compression architecture called dynamic convolution, which can reuse the previous feature maps to reduce the calculation amount; and combine with filter pruning to further speed up the performance. In this paper, we tested the presented method on 45 surveillance videos with various scenes. The experimental results show that the hybrid pruning architecture can reduce up to 80.4% of FLOPs while preserving the precision within 1.34% mAP; furthermore, the method can improve the processing speed up to 2.8 times compared to the traditional Single Shot MultiBox Detection.

Foreground Extraction and Motion Recognition Technology for Intelligent Video Surveillance

With the rapid development of computer technology and network technology, it has become possible to build a large-scale networked video surveillance system. The video surveillance system has become a new type of infrastructure necessary for modern cities. In this paper, the problem of foreground extraction and motion recognition in intelligent video surveillance is studied. The three key sub-problems, namely the extraction of motion foreground in video, the deblurring of motion foreground and the recognition of human motion, are studied and corresponding solutions are proposed. A background modeling technique based on video block is proposed. The background is modeled at the block level, which greatly reduces the spatial complexity of the algorithm. It solves the problem that the traditional Gaussian model (GMM) moving target enters the static state and is integrated into the background process. The target starts to move for a long time and there are ghosts and other problems, which reduce the processing efficiency of the lifting algorithm. The test results on the Weizmann dataset show that the proposed algorithm can achieve high human motion recognition accuracy and recognition with low computational complexity. The rate can reach 100%; the local constrained group sparse representation classification (LGSRC) model is used to classify it. The experimental results on Weizmann, KTH, UCF sports and other test datasets confirm the validity of the algorithm in this chapter. KNN, SRC voting classification accuracy.

ISEE: An Intelligent Scene Exploration and Evaluation Platform for Large-Scale Visual Surveillance

Intelligent video surveillance (IVS) is always an interesting research topic to utilize visual analysis algorithms for exploring richly structured information from big surveillance data. However, existing IVS systems either struggle to utilize computing resources adequately to improve the efficiency of large-scale video analysis, or present a customized system for specific video analytic functions. It still lacks of a comprehensive computing architecture to enhance efficiency, extensibility and flexibility of IVS system. Moreover, it is also an open problem to study the effect of the combinations of multiple vision modules on the final performance of end applications of IVS system. Motivated by these challenges, we develop an Intelligent Scene Exploration and Evaluation (ISEE) platform based on a heterogeneous CPU-GPU cluster and some distributed computing tools, where Spark Streaming serves as the computing engine for efficient large-scale video processing and Kafka is adopted as a middle-ware message center to decouple different analysis modules flexibly. To validate the efficiency of the ISEE and study the evaluation problem on composable systems, we instantiate the ISEE for an end application on person retrieval with three visual analysis modules, including pedestrian detection with tracking, attribute recognition and re-identification. Extensive experiments are performed on a large-scale surveillance video dataset involving 25 camera scenes, totally 587 hours 720p synchronous videos, where a two-stage question-answering procedure is proposed to measure the performance of execution pipelines composed of multiple visual analysis algorithms based on millions of attribute-based and relationship-based queries. The case study of system-level evaluations may inspire researchers to improve visual analysis algorithms and combining strategies from the view of a scalable and composable system in the future.

VU: Edge Computing-Enabled Video Usefulness Detection and its Application in Large-Scale Video Surveillance Systems

In the era of smart and connected communities, video surveillance systems, which typically involve tens to thousands of cameras, have increasingly become prominent components for public safety. In current practice, when a failure occurs in a video surveillance system, the operation and maintenance teams usually spend a substantial amount of time locating and identifying the failure; hence, the fast online response cannot be guaranteed in a large-scale video surveillance system. Meanwhile, the video data that contains potential failures consumes bandwidth that could be used for useful video data. The useless video will waste the scarce bandwidth in the network and storage usage in the cloud. The emergence of edge computing is highly promising in video preprocessing with an edge camera. A video surveillance system is a killer application for edge computing. In this article, we propose an edge computing-enabled video usefulness (i.e., VU) model for large-scale video surveillance systems. We also explore its application, e.g., early failure detection and bandwidth improvement. According to the usefulness of the video data, the VU model can locate a failure and send it to end-users on the fly. In this article, our goals are threefold: 1) proposing a comprehensive VU model. To the best of our knowledge, this is the first work to explore the feasibility of the VU model and to determine VU values in a real application; 2) reducing the mean time to detection (i.e., MTTD) efficiently via edge computing-enabled fast online failure detection approaches; and 3) relieving the network bandwidth for large-scale video surveillance systems. Our experimental results demonstrate the approaches in VU model accurately detect failures that were collected from a video surveillance system with approximately 4000 cameras. The MTTD is substantially shortened by the fast online detection approaches. The video data with the worst VU values is mostly discarded to lessen overload of the network.

AI-Oriented Large-Scale Video Management for Smart City: Technologies, Standards, and Beyond

Deep learning has achieved substantial success in a series of tasks in computer vision. Intelligent video analysis, which can be broadly applied to video surveillance in various smart city applications, can also be driven by such powerful deep learning engines. To practically facilitate deep neural network models in the large-scale video analysis, there are still unprecedented challenges for the large-scale video data management. Deep feature coding, instead of video coding, provides a practical solution for handling the large-scale video surveillance data. To enable interoperability in the context of deep feature coding, standardization is urgent and important. However, due to the explosion of deep learning algorithms and the particularity of feature coding, there are numerous remaining problems in the standardization process. This paper envisions the future deep feature coding standard for the AI oriented large-scale video management, and discusses existing techniques, standards and possible solutions for these open problems.

PROVID: Progressive and Multimodal Vehicle Reidentification for Large-Scale Urban Surveillance

Compared with person reidentification, which has attracted concentrated attention, vehicle reidentification is an important yet frontier problem in video surveillance and has been neglected by the multimedia and vision communities. Since most existing approaches mainly consider the general vehicle appearance for reidentification while overlooking the distinct vehicle identifier, such as the license plate number, they attain suboptimal performance. In this paper, we propose PROVID, a PROgressive Vehicle re-IDentification framework based on deep neural networks. In particular, our framework not only utilizes the multimodality data in large-scale video surveillance, such as visual features, license plates, camera locations, and contextual information, but also considers vehicle reidentification in two progressive procedures: coarse-to-fine search in the feature domain, and near-to-distant search in the physical space. Furthermore, to evaluate our progressive search framework and facilitate related research, we construct the VeRi dataset, which is the most comprehensive dataset from real-world surveillance videos. It not only provides large numbers of vehicles with varied labels and sufficient cross-camera recurrences but also contains license plate numbers and contextual information. Extensive experiments on the VeRi dataset demonstrate both the accuracy and efficiency of our progressive vehicle reidentification framework.

LSTM-based multi-label video event detection

Since large-scale surveillance videos always contain complex visual events, how to generate video descriptions effectively and efficiently without human supervision has become mandatory. To address this problem, we propose a novel architecture for jointly recognizing multiple events in a given surveillance video, motivated by the sequence to sequence network. The proposed architecture can predict what happens in a video directly without the preprocessing of object detection and tracking. We evaluate several variants of the proposed architecture with different visual features on a novel dataset perpared by our group. Moreover, we compute a wide range of quantitative metrics to evaluate this architecture. We further compare it to the popular Support Vector Machine-based visual event detection method. The comparison results suggest that the proposal method can outperform the traditional computer vision pipelines for visual event detection.

A security framework for cloud-based video surveillance system

Utilizing cloud services in running large-scale video surveillance systems is not uncommon. However, special attention should be given to data security and privacy because, typically, data travels over insecure public networks. In this work, we propose an end-to-end security framework for a cloud-based video surveillance system that supports a large number of cameras. Our framework provides mutual authentication, session key management, data confidentiality, and data integrity. Consequently, encrypted video frames can only be sourced from authenticated cameras and only destined to authenticated cloud devices where the integrity of such frames can also be verified against potential change. As video streaming is a very delay-sensitive application, we study different variations of the proposed framework to find security options that achieve the best trade-off between the added delay and the security of the system.

Internet of surveillance: a cloud supported large-scale wireless surveillance system

Large-scale video surveillance systems are among the necessities for securing our life these days. The high bandwidth demand and the large storage requirements are the main challenges in such systems. To face these challenges, the system can be deployed as a multi-tier framework that utilizes different technologies. In such a framework, technologies proposed under the umbrella of the Internet of Things (IoT) can play a significant rule in facing the challenges. In video surveillance, the cameras can be considered as "the things" that are streaming videos to a central processing and storage server (the cloud) through the Internet. Wireless technologies can be used to connect wireless cameras to the surveillance system more conveniently than wired cameras. Unfortunately, wireless communication in general tend to have limited bandwidth that needs careful management to achieve scalability. In this paper, we design and evaluate a reliable IoT-based wireless video surveillance system that provides an optimal bandwidth distribution and allocation to minimize the overall surveillance video distortion. We evaluate our system using NS-3 simulation. The results show that the proposed framework fully utilizes the available cloud bandwidth budget and achieves high scalability.

Implementation of Real-Time Vehicle Tracking in City-Scale Video Network

ABSTRACTTracking unexpected warning vehicles is required for quick response to security incident. To realize real-time vehicle tracking in a large-scale video surveillance network, a geospatial and temporal connection (GSTC) model is introduced to model the connection between videos. The transition time between videos is modeled by a Gaussian mixture model (GMM). With the developed plug-ins based on GSTC and GMM, the video streams of defined geospatial neighbors are automatically called in with the video stream that the object appears in during the tracking process. Experiments show that the ratio of success of real-time tracking is largely increased.

Event-based large scale surveillance video summarization

Recent advances in sensor manufacture and computer vision technologies have simulated the applications of intelligent transportation systems, while a key yet under-addressed issue in these systems is the semantic summarization of large scale surveillance video. The main difficulty of large scale surveillance video summarization arises from the contradiction between the high-degree spatiotemporal redundancies and the limited storage budget. In this paper, we propose a novel approach of large scale surveillance video summarization on the basis of event detection. In the proposed approach, we firstly obtain the trajectories of vehicles and pedestrians in a tracking-by-detection manner, and then detect the abnormal events using the trajectories. Finally, we design a disjoint max-coverage algorithm to generate a summarized sequence with maximum coverage of interested events and minimum number of frames. Compared with traditional key frame-based approaches, our approach enjoys the following favorable features. First, important information can be efficiently extracted from the redundant contents since the approach is event-centric and those interested events contain almost all the important information. Second, abnormal events are successfully detected by combining the Random Forest classifier and the trajectory features. Third, the abnormal events are designed to display, and hence further reduces the compression ratio. Due to the above features, the proposed approach is suitable for different scenarios, ranges from highway to crowded crossings. Experiments on 12 surveillance sequences validate the effectiveness and efficiency of the proposed approach.

Nodes' Adjacency Relation of Video Sensor Network Based on GIS and GPS

As far as the large-scale video surveillance sensor network in urban road and highway, the relay-surveillance on abnormal behavior or particular targets is one of the hot focuses of in recent researches, while the establishment of adjacency relationship of the neighbor sensor nodes is the basis of the sensor scheduling for the relay-surveillance. The topology of a road network is generated according to the road information, which has already existed in the geographic information systems (GIS) regarding the road intersections as nodes and the section between the two intersections as the edge. The initial topology of relay-adjacency relationship between sensors is built by that each video sensor is deployed at the each intersection and the section of the each two intersections is regarded as the basis of adjacency between the each two sensors. When a new video sensor is to be deployed in a section of a road, the related deployed sensors in same section are searched by using the spatial index of GIS based on its GPS information, and then the adjacency relationship between the new sensor and the related ones is generated by using the sorting algorithm according to their GPS information. By using the road network information that has already existed in the GIS system, the algorithm on establishing the relay-adjacency relationship of video sensors is simple and simpler to implement, and it can be used in the construction of sensor relay-surveillance topology such as automatic real-time tracking on abnormal behavior or the analysis of the escape routes and so on in city roads, highway, smarter cities and smarter planet.

Human object annotation for surveillance video forensics

A system that can automatically annotate surveillance video in a manner useful for locating a person with a given description of clothing is presented. Each human is annotated based on two appearance features: primary colors of clothes and the presence of text/logos on clothes. The annotation occurs after a robust foreground extraction stage employing a modified Gaussian mixture model-based approach. The proposed pipeline consists of a preprocessing stage where color appearance of an image is improved using a color constancy algorithm. In order to annotate color information for human clothes, we use the color histogram feature in HSV space and find local maxima to extract dominant colors for different parts of a segmented human object. To detect text/logos on clothes, we begin with the extraction of connected components of enhanced horizontal, vertical, and diagonal edges in the frames. These candidate regions are classified as text or nontext on the basis of their local energy-based shape histogram features. Further, to detect humans, a novel technique has been proposed that uses contourlet transform-based local binary pattern (CLBP) features. In the proposed method, we extract the uniform direction invariant LBP feature descriptor for contourlet transformed high-pass subimages from vertical and diagonal directional bands. In the final stage, extracted CLBP descriptors are classified by a trained support vector machine. Experimental results illustrate the superiority of our method on large-scale surveillance video data.

Large scale continuous visual event recognition using max-margin Hough transformation framework

In this paper we propose a novel method for continuous visual event recognition (CVER) on a large scale video dataset using max-margin Hough transformation framework. Due to high scalability, diverse real environmental state and wide scene variability direct application of action recognition/detection methods such as spatio-temporal interest point (STIP)-local feature based technique, on the whole dataset is practically infeasible. To address this problem, we apply a motion region extraction technique which is based on motion segmentation and region clustering to identify possible candidate “event of interest” as a preprocessing step. On these candidate regions a STIP detector is applied and local motion features are computed. For activity representation we use generalized Hough transform framework where each feature point casts a weighted vote for possible activity class centre. A max-margin frame work is applied to learn the feature codebook weight. For activity detection, peaks in the Hough voting space are taken into account and initial event hypothesis is generated using the spatio-temporal information of the participating STIPs. For event recognition a verification Support Vector Machine is used. An extensive evaluation on benchmark large scale video surveillance dataset (VIRAT) and as well on a small scale benchmark dataset (MSR) shows that the proposed method is applicable on a wide range of continuous visual event recognition applications having extremely challenging conditions.

Detection of cross-channel anomalies

The data deluge has created a great challenge for data mining applications wherein the rare topics of interest are often buried in the flood of major headlines. We identify and formulate a novel problem: cross-channel anomaly detection from multiple data channels. Cross-channel anomalies are common among the individual channel anomalies and are often portent of significant events. Central to this new problem is a development of theoretical foundation and methodology. Using the spectral approach, we propose a two-stage detection method: anomaly detection at a single-channel level, followed by the detection of cross-channel anomalies from the amalgamation of single-channel anomalies. We also derive the extension of the proposed detection method to an online settings, which automatically adapts to changes in the data over time at low computational complexity using incremental algorithms. Our mathematical analysis shows that our method is likely to reduce the false alarm rate by establishing theoretical results on the reduction of an impurity index. We demonstrate our method in two applications: document understanding with multiple text corpora and detection of repeated anomalies in large-scale video surveillance. The experimental results consistently demonstrate the superior performance of our method compared with related state-of-art methods, including the one-class SVM and principal component pursuit. In addition, our framework can be deployed in a decentralized manner, lending itself for large-scale data stream analysis.