Synopsis Generation Research Articles

Spherical video synopsis generation and visualization framework

With the advances in video technology, the advent of spherical video (360° video) recorded using an omnidirectional camera offers a limitless field-of-view (FoV) to the viewers. However, they suffer from the fear of missing out (FOMO) because they can only see a particular FoV at a time. Reviewing a long recorded surveillance video i.e., 24 hours a day is a time-consuming process due to temporal and spatial redundancy. A solution to this problem is to compactly represent the video synopsis by shifting the objects along the time domain. Using a multi-camera setup for surveillance creates blind spots. This problem is solved by using a spherical camera. Therefore, in this paper, we focus on creating and visualizing the video synopsis recorded by the spherical camera. The optimization algorithm plays a key role in condensing the recorded video. Hence, a novel spherical video synopsis optimization framework has been introduced to generate compact videos that eliminate FOMO. The synopsis is generated by shifting objects on the temporal axis and displays them simultaneously by optimizing multiple constraints. It minimizes activity loss, virtual collisions, temporal inconsistencies, and synopsis video length by preserving interactions between objects. The proposed multiobjective optimization includes a new constraint to restrict the number of objects displayed per frame due to the limitation of the human visual system. Direction-based visualization methods have been proposed to improve the viewer’s experience without FOMO. Comparative performance of the proposed framework using the latest metaheuristic optimization algorithms with existing video synopsis optimization algorithms is performed. It is found that chronological disorder ratio and overall virtual collision are minimized effectively through the recent metaheuristics optimization algorithms compared to the related works on video synopsis.

Multiview abnormal video synopsis in real-time

The proliferation of surveillance cameras has resulted in an exponential growth in video data, which has created several challenges for video data analysis, extraction, and storage. Video synopsis is an efficient methodology for analyzing and storing video data because it creates short videos. However, existing video synopsis methods are inappropriate for abnormal behavior situations (e.g., constructing a video synopsis of only abnormal frames emphasizing gun and knife characteristics to predict violent activity). As most existing video synopsis methods rely on the foreground and multiple objects tracking preprocessing, they perform poorly in real-time scenarios, particularly in crowded situations. We proposed a video synopsis approach suitable for multiview cameras to mitigate this problem; it can also be applied to radically diverse crowdedness. A considerable difference exists between the proposed method and existing approaches because it has several unique properties. We first detected abnormal segments from a given video and then tracked and extracted the frames. We can extract the required meaningful data by applying this method, which increases the synopsis generation performance. Second, we created a summary of all the videos. Finally, a stitching algorithm was used to create the synopsis.

An improved tube rearrangement strategy for choice-based surveillance video synopsis generation

Video synopsis is a promising technology that offers easy browsing and indexing of surveillance videos. This article presents an improved video synopsis framework, introducing the inclusion of anomalous tube detection module. The suggested framework performs better than the existing methodologies by offering a flexibility to the user to generate a synopsis video, which is based on user's choice of interest. Traditionally, to generate a synopsis video, the object tubes are temporally shifted for achieving compression. The applied temporal shift incurs a large number of collision artifacts along with temporal chronology violation. To address this issue for producing a visually comfortable synopsis video, the collision and temporal chronology violations are amended through Acceleration/Retardation of object's motion and spatial shift. Collision oriented Acceleration/Retardation and Spatial shift strategies are embedded sequentially in the proposed combined algorithm cSAScO. The unified representation of the proposed cSAScO algorithm combines the individual strength of Simulated Annealing (SA) and Scenario Optimization (ScO) and is employed to the formulated choice-based tube rearrangement problem. The efficacy of the proposed scheme is demonstrated through extensive experiments and its performance compared with that of the benchmark schemes.

GAN based efficient foreground extraction and HGWOSA based optimization for video synopsis generation

Video Synopsis is a smart and efficient solution to summarize a long duration of surveillance video into short. Most of the video synopsis techniques are not suitable to address complex situations like changes in illumination, dynamic background, camera jitter, etc. These techniques firmly depend on the preprocessing results of foreground extraction and multiple objects tracking. Further, the optimization process is a vital phase for the decrement of collision rate among moving objects, where the widely used Simulated Annealing (SA) usually suffers from the issue of slow convergence rate with a high computational overhead. Taking these aforementioned facts into account for feature extraction, we formulate a foreground extraction scheme exploring the concept of multi-frame and multi-scale in Generative Adversarial Network (mFS-GANs). Further, an optimization algorithm is proposed through the hybridization of SA and Grey Wolf Optimizer (GWO), named as, HGWOSA to ensure global optimal result with a low computing overhead. The performance of the proposed scheme is evaluated through extensive simulations and compared with that of the benchmark schemes. The experiments are carried out using some standard surveillance video dataset (ChangeDetection.Net, MIT Surveillance Dataset, and UMN Dataset) and one self-generated surveillance video at IIIT Bhubaneswar. Overall analysis and experimental evaluations demonstrate that our proposed scheme outperforms the other competing schemes in terms of both the quantitative and qualitative measures. Finally, the proposed model can be substantially employed in the generation of off-line video synopsis, which is potentially applicable to video surveillance applications for smart cities.

Demonstration of ScroogeDB

We demonstrate ScroogeDB which aims at minimizing monetary cost of processing aggregation queries in the Cloud. It runs on top of a Cloud database that offers pay-as-you-go query processing where users pay according to the number of bytes processed. ScroogeDB exploits deterministic approximate query processing (DAQ) to achieve monetary savings. That is, ScroogeDB provides deterministic bounds, i.e., bounds that contain the true value with a 100% probability. ScroogeDB creates small synopses of the database and uses these synopses to answer aggregation queries. By rewriting a query on base tables into a query on smaller synopses, we significantly reduce the amount of processed data. We do not pre-compute synopses in advance of an analysis session. Instead, we generate them on the fly, interleaving synopsis generation with query execution. In our demonstration, we show that our system realizes impressive monetary savings with little precision loss. We run our system on top of the Google BigQuery Cloud platform and provide users with a graphical interface that visualizes deterministic bounds. The graphical interface also provides information regarding the generated synopses and how they contribute to monetary savings.

Query-Based Video Synopsis for Intelligent Traffic Monitoring Applications

Synopsis of a long-duration video has many applications in intelligent transportation systems. It can help to monitor traffic with lesser manpower. However, generating meaningful synopsis of a long-duration video recording can be challenging. Often summarized outputs include redundant contents or activities that may not be helpful to the observer. Moving object trajectories are possible sources of information that can be used to generate the synopsis of long-duration videos. The synopsis generation faces challenges due to object tracking, grouping of the trajectories with respect to activity type, object category, and contextual information, and generating smooth synopsis according to a query. In this paper, we propose a method to generate meaningful and smooth synopsis of long-duration videos according to the users’ query. We have tracked moving objects and adopted deep learning to classify the objects into known categories (e.g., car, bike, and pedestrians). We then identify regions in the surveillance scene with the help of unsupervised clustering. Each tube (spatiotemporal object trajectory) is represented by the source and the destination. In the final stage, we take a query from the user and generate the synopsis video by smoothly blending the appropriate tubes over the background frame through energy minimization. The proposed method has been evaluated on two publicly available datasets and our own surveillance datasets. We have compared the method with popular state-of-the-art techniques. The experiments reveal that the proposed method is superior to the existing techniques and it produces visually seamless video synopsis.

Trajectory-Based Surveillance Analysis: A Survey

Due to the advancement of camera hardware and machine learning techniques, video object tracking for surveillance has received noticeable attention from the computer vision research community. Object tracking and trajectory modeling have important applications in surveillance video analysis. For example, trajectory clustering, summarization or synopsis generation, and detection of anomalous or abnormal events in videos are mainly being exploited by the research community. However, barring one research work (which is almost a decade old), there is no recent review that emphasizes the use of video object trajectories, particularly in the perspective of visual surveillance. This paper presents a survey of trajectory-based surveillance applications with a focus on clustering, anomaly detection, summarization, and synopsis generation. The methods reviewed in this paper broadly summarize the abovementioned applications. The main purpose of this survey is to summarize the state-of-the-art video object trajectory analysis techniques used in the indoor and outdoor surveillance.

Comparing synopsis techniques for approximate spatial data analysis

The increasing amount of spatial data calls for new scalable query processing techniques. One of the techniques that are getting attention is data synopsis , which summarizes the data using samples or histograms and computes an approximate answer based on the synopsis. This general technique is used in selectivity estimation, clustering, partitioning, load balancing, and visualization, among others. This paper experimentally studies four spatial data synopsis techniques for three common data analysis problems, namely, selectivity estimation, k-means clustering, and spatial partitioning. We run an extensive experimental evaluation on both real and synthetic datasets of up to 2.7 billion records to study the trade-offs between the synopsis methods and their applicability in big spatial data analysis. For each of the three problems, we compare with baseline techniques that operate on the whole dataset and evaluate the synopsis generation time, the time for computing an approximate answer on the synopsis, and the accuracy of the result. We present our observations about when each synopsis technique performs best.

An improved surveillance video synopsis framework: a HSATLBO optimization approach

Video surveillance cameras capture huge amount of data 24 hours a day. However, most of these videos contain redundant data which make the process difficult for browsing and analysis. A significant amount of research findings have been made in summarization of recorded video, but such schemes do not have much impact on video surveillance applications. On the contrary, video synopsis is a smart technology that preserves all the activities of every single object and projects them concurrently in a condensed time. The energy minimization module in video synopsis framework plays a vital role, which in turn minimizes the activity loss, number of collision and temporal consistency cost. In most of the reported schemes, Simulated Annealing (SA) algorithm is employed to solve the energy minimization problem. However, it suffers from slow convergence rate resulting in a high computational load to the system. In order to mitigate this issue, this article presents an improved energy minimization scheme using hybridization of SA and Teaching Learning based Optimization (TLBO) algorithms. The suggested framework for static surveillance video synopsis generation consists of four computational modules, namely, Object detection and segmentation, Tube formation, Optimization, and finally Stitching and the central focus is on the optimization module. Thus, the present work deals with an improved hybrid energy minimization problem to achieve global optimal solution with reduced computational time. The motivation behind hybridization (HSATLBO) is that TLBO algorithm has the ability to search rigorously, ensuring to reach the optimum solution with less computation. On the contrary, SA reaches the global optimum solution, but it may get disarrayed and miss some critical search points. Exhaustive experiments are carried out and results compared with that of benchmark schemes in terms of minimizing the activity, collision and temporal consistency costs. All the experiments are conducted on five widely used videos taken from standard surveillance video data set (PETS 2001, MIT Surveillance Dataset, ChangeDetection.Net, PETS 2006 and UMN Dataset) as well as one real generated surveillance video from the IIIT Bhubaneswar Surveillance Dataset. To make a fair comparison, additionally, performance of the proposed hybrid scheme to solve video synopsis optimization problem is also compared with that of the other benchmark functions. Experimental evaluation and analysis confirm that the proposed scheme outperforms other state-of-the-art approaches. Finally, the suggested scheme can be easily and reliably deployed in the off-line video synopsis generation.

The incremental Fourier classifier: Leveraging the discrete Fourier transform for classifying high speed data streams

Two major performance bottlenecks with decision tree based classifiers in a data stream environment are the depth of the tree and the update overhead of maintaining leaf node statistics on an instance-wise basis to ensure that classification is consistent with the current state of the data stream. Previous research has shown that classifiers based on Fourier spectra derived from decision trees produce compact array structures that can be searched and maintained much more efficiently than deep tree based structures. However, the key issue of incrementally adapting the spectrum to changes has not been addressed. In this research we present a strategy for incremental maintenance of the Fourier spectrum to changes in concept that take place in data stream environments. Along with the incremental approach we also propose schemes for feature selection and synopsis generation that enable the coefficient array to be refreshed efficiently on a periodic basis. Our empirical evaluation on a number of widely used stream classifiers reveals that the Fourier classifier outperforms them, both in terms of classification accuracy as well as speed of classification.

Surveillance video synopsis generation method via keeping important relationship among objects

To reduce the efforts of human in browsing long surveillance videos, synopsis videos are proposed. Traditional synopsis video generation methods condense most of the activities in the video by simultaneously showing several actions, even when they originally occurred at different times. This inevitably causes ignorance of temporal relationship among objects. For example, two persons walk shoulder to shoulder and they are detected and tracked separately, but in the synopsis they never ‘met’. In this study, a trajectory mapping model is defined, whose energy function includes not only the cost caused by the synopsis video, but that of the original video. In this way, it tries to make the relationship between objects of the original video consistent with that of the synopsis. Finally, the video synopsis is generated by an energy minimisation method. Experiments show that the proposed video synopsis can reduce the spatiotemporal redundancies of the input video as much as possible. Moreover, it can keep the important relationship between objects and maintain the time consistency of important activities.

Selectivity estimation of extended XML query tree patterns based on prime number labeling and synopsis modeling

With the new era of big data and the proliferation of XML documents for representing and exchanging data over the web, selectivity estimation of XML query patterns has become a crucial component of database optimizers. It helps the optimizer choose the best possible plan for query evaluation. Existing selectivity estimators for XML queries can only support basic Query Tree Patterns (QTPs) with logical AND operator. In this paper, we propose a novel approach, called XQuest, for selectivity estimation that supports extended QTPs that may contain logical operators or wildcards. This approach is based on a modified implementation of prime number labeling to construct a structural summary model of the XML data. Subsequently, a simulator of an XML query evaluator runs on the resulting model from the previous stage and aggregates the estimate for each target QTP. We conducted several experiments to study the performance of the proposed approach on three XML benchmark datasets; in terms of synopsis generation time, storage requirements, and estimation accuracy. The results show that the proposed approach can have more accurate estimates with low memory and time requirements. For example, when compared to a Sampling algorithm with the same allocated memory budget, the error rate of the proposed approach never reached 5% whereas it reached 98.5% for the Sampling algorithm.

MVS: A multi-view video synopsis framework

In this paper, we present a framework for generating a synopsis of multi-view videos that are acquired from a surveillance site, indoor or outdoor, using multiple cameras. The synopsis generation is modeled as a scheduling problem that we solve using three separate approaches: table-driven approach, contradictory binary graph coloring (CBGC) approach, and simulated annealing (SA) based approach. An action recognition module is included in the framework to recognize important actions performed by various humans present in the videos. Inclusion of such important actions in the synopsis has helped to reduce its length significantly. The synopsis length is further reduced through a post-processing step that computes the visibility score for each object track using a fuzzy inference system. Among the three proposed schemes, maximum reduction in synopsis length is obtained through the CBGC approach. The stochastic approach using SA, on the other hand, achieves a better trade-off among the multiple optimization criteria. Experimental evaluations on standard datasets demonstrate the efficacy of the proposed framework over its counterparts concerning the reduction in synopsis length and retention of important actions.

Maximum a Posteriori Probability Estimation for Online Surveillance Video Synopsis

To reduce human efforts in browsing long surveillance videos, synopsis videos are proposed. Traditional synopsis video generation applying optimization on video tubes is very time consuming and infeasible for real-time online generation. This dilemma significantly reduces the feasibility of synopsis video generation in practical situations. To solve this problem, the synopsis video generation problem is formulated as a maximum a posteriori probability (MAP) estimation problem in this paper, where the positions and appearing frames of video objects are chronologically rearranged in real time without the need to know their complete trajectories. Moreover, a synopsis table is employed with MAP estimation to decide the temporal locations of the incoming foreground objects in the synopsis video without needing an optimization procedure. As a result, the computational complexity of the proposed video synopsis generation method can be significantly reduced. Furthermore, as it does not require prescreening the entire video, this approach can be applied on online streaming videos.

Video Synopsis Generation Using GPGPU

Video summaries provide a compact video representation preserving the essential activities of the original video, but the summaries may be confusing when mixing different activities together. Summaries Clustered methodology, showing similar activities simultaneously, enables to view much easier and more efficiently. However, it is very time consuming in generating summaries, especially in calculating motion distance and collision cost. To improve the efficiency of generating summaries, a parallel video synopsis generation algorithm is proposed based on GPGPU. The experiment result shows generation efficiency is improved greatly through GPU parallel computing. The acceleration radio can reach at 5.75 when data size is above 1600*960*30000.

Surveillance video synopsis in the compressed domain for fast video browsing

The traditional pixel-domain based video analysis methods have taken dominated places for long. However, due to the rapidly increasing volume and resolution of surveillance video, the desirable fast and scalable browsing encounters significant challenges in terms of efficiency and flexibility. Under this circumstance, operating surveillance video in compressed domain has aroused great concern in academy and industry. In order to perform the intelligent video analysis task on the premise of preserving accuracy and controlling complexity, this paper presents a compressed-domain approach for massive surveillance video synopsis generation, labeling and browsing. The main work and achievements include: (1) a compressed-domain scheme is established to condense the compressed surveillance video and record the synopsis results; (2) a background modeling method via the Motion Vector based Local Binary Pattern (MVLBP) is introduced to extract moving objects in an efficient way; (3) an object flags based synopsis labeling method is proposed to represent the object regions as well as their display modes in a flexible way. Experimental results show that the video analysis system based on this framework can provide not only efficient synopsis generation but also flexible scalable or playback browsing.

Incremental Maintenance of Online Summaries Over Multiple Streams

We propose a novel approach based on predictive quantization (PQ) for online summarization of multiple time-varying data streams. A synopsis over a sliding window of most recent entries is computed in one pass and dynamically updated in constant time. The correlation between consecutive data elements is effectively taken into account without the need for preprocessing. We extend PQ to multiple streams and propose structures for real-time summarization and querying of a massive number of streams. Queries on any subsequence of a sliding window over multiple streams are processed in real time. We examine each component of the proposed approach, prediction, and quantization separately and investigate the space-accuracy trade-off for synopsis generation. Complementing the theoretical optimality of PQ-based approaches, we show that the proposed technique, even for very short prediction windows, significantly outperforms the current techniques for a wide variety of query types on both synthetic and real data sets.

Dynamic Nonuniform Data Approximation in Databases with Haar Wavelet

Data synopsis is a lossy compressed representation of data stored into databases that helps the query optimizer to speed up the query process, e.g. time to retrieve the data from the database. An efficient data synopsis must provide accurate information about the distribution of data to the query optimizer at any point in time. Due to the fact that some data will be queried more often than others, a good data synopsis should consider the use of nonuniform accuracy, e.g. provide better approximation of data that are queried the most. Although, the generation of data synopsis is a critical step to achieve a good approximation of the initial data representation, data synopsis must be updated over time when dealing with time varying data. In this paper, we introduce new Haar wavelet synopses for nonuniform accuracy and time-varying data that can be generated in linear time and space, and updated in sublinear time. We further introduce two linear algorithms, called 2-step and M-Step for the Point-wise approximation problem that clearly outperforms previous algorithms known in literature, and two new algorithm called, Data Mapping and Weight Mapping for the Range-sum approximation problem that, to the best of our knowledge, represent a key research milestone as being the first linear algorithm for arbitrary weights. For both scenarios, we focus not only on the generation of the data synopsis but also on their updates over time. The efficiency of our new data synopses is validated against other linear methods by using both synthetic and real data sets.