Multi-view Video Summarization Research Articles

Multi View Video Summarization Using RNN and SURF Based High Level Moving Object Feature Frames

Multi-View Video summarization is a process to ease the storage consumption that facilitates organized storage, and perform other mainline videos analytical task. This in-turn helps quick search or browse and retrieve the video data with minimum time and without losing crucial data. In static video summarization, there is less challenge in time and sequence issues to rearrange the video-synopsis. The low-level features are easy to compute and retrieve. But for high-level features like event detection, emotion detection, object recognition, face detection, gesture detection, and others requires the comprehension of the video content. This research is to propose an approach to over- come the difficulties in handling the high-level features. The distinguishable contents from the videos are identified by object detection and feature-based area strategy. The major aspect of the proposed solution is to retrieve the attributes of a motion source from a video frame. By dividing the details of the object that are available in the video frame wavelet decomposition are achieved. The motion frequency scoring method records the time of motions in the video. The frequency motion feature of video usage is a challenge given the continuous change of objects shape. Therefore, the object position and corner points are spotted using Speeded Up Robust Features (SURF) feature points. Support vector machine clustering extracts keyframes. The memory-based re- current neural network (RNN) recognizes the object in the video frame and remembers a long sequence. RNN is an artificial neural network where nodes form a temporal relationship. The attention layer in the proposed RNN network extracts the details about the objects in motion. The motion objects identified using the three video clippings is finally summarized using video summarization algorithm. To perform the simulation, MATLAB R 2014b software was used.

Multiview video summarization using video partitioning and clustering

Multiview video summarization plays a crucial role in abstracting essential information form multiple videos of the same location and time. In this paper, we propose a new approach for the multiview summarization. The proposed approach uses the BIRCH clustering algorithm for the first time on the initial set of frames to get rid of the static and redundant. The work presents a new approach for shot boundary detection using frame similarity measures Jaccard and Dice. The algorithm performs effectively synchronized merging of keyframes from all camera-views to obtain the final summary. Extensive experimentation conducted on various datasets suggests that the proposed approach significantly outperforms most of the existing video summarization approaches. To state a few, a 1.5% improvement on video length reduction, 24.28% improvement in compression ratio, and 6.4% improvement in quality assessment ratio is observed on the lobby dataset.

A comprehensive survey of multi-view video summarization

There has been an exponential growth in the amount of visual data on a daily basis acquired from single or multi-view surveillance camera networks. This massive amount of data requires efficient mechanisms such as video summarization to ensure that only significant data are reported and the redundancy is reduced. Multi-view video summarization (MVS) is a less redundant and more concise way of providing information from the video content of all the cameras in the form of either keyframes or video segments. This paper presents an overview of the existing strategies proposed for MVS, including their advantages and drawbacks. Our survey covers the genericsteps in MVS, such as the pre-processing of video data, feature extraction, and post-processing followed by summary generation. We also describe the datasets that are available for the evaluation of MVS. Finally, we examine the major current issues related to MVS and put forward the recommendations for future research1[1]https://github.com/tanveer-hussain/MVS-Survey.

A Study on Multi-view Video Summarization Techniques

With the recent developments in multimedia and their applications in day-to-day life, the summarization of video content is a growing and popular field. The main objective of video summarization is to abstract the important events, or objects from a video to provide an easily interpretable summary. The video-surveillance is a crucial security mechanism of many public and private places like hospitals, universities, banks, etc. These videos provide important information about any intruder or invaders. Although this information is of critical importance, it takes a lot of time and effort to go through hours of long videos in order to find out the significant events. Here, multi-view video summarization comes to rescue as it extracts the most useful events out of the videos and presents it to the user. The multi-view video summarization system provides an advantage over a single view video summarization system by exploiting the complicated intra-view and inter-view correlations, thereby presenting an effective summary of multiple camera views and enriching the video content. It explores various dimensions of the same event and collectively may give more complete information. In this paper, we present a study on multi-view video summarization techniques. We examine work done on various methods of video summarization for videos consisting of many views of the same location as captured by the surveillance cameras positioned at different angles. The final summarized video can be used for the identification of suspicious events, an inspection of post-accidental events, thefts in public places, etc., which will be supportive to the crime department for investigation purposes.

Multiview Summarization and Activity Recognition Meet Edge Computing in IoT Environments

Multiview video summarization (MVS) has not received much attention from the research community due to inter-view correlations and views' overlapping, etc. The majority of previous MVS works are offline, relying on only summary, and require additional communication bandwidth and transmission time, with no focus on foggy environments. We propose an edge intelligence-based MVS and activity recognition framework that combines artificial intelligence with Internet of Things (IoT) devices. In our framework, resource-constrained devices with cameras use a lightweight CNN-based object detection model to segment multiview videos into shots, followed by mutual information computation that helps in a summary generation. Our system does not rely solely on a summary, but encodes and transmits it to a master device using a neural computing stick for inter-view correlations computation and efficient activity recognition, an approach which saves computation resources, communication bandwidth, and transmission time. Experiments show an increase of 0.4 unit in F-measure on an MVS Office dateset and 0.2% and 2% improved accuracy for UCF-50 and YouTube 11 datesets, respectively, with lower storage and transmission times. The processing time is reduced from 1.23 to 0.45 s for a single frame and optimally 0.75 seconds faster MVS. A new dateset is constructed by synthetically adding fog to an MVS dateset to show the adaptability of our system for both certain and uncertain IoT surveillance environments.

Intelligent Embedded Vision for Summarization of Multiview Videos in IIoT

Nowadays, video sensors are used on a large scale for various applications, including security monitoring and smart transportation. However, the limited communication bandwidth and storage constraints make it challenging to process such heterogeneous nature of Big Data in real time. Multiview video summarization (MVS) enables us to suppress redundant data in distributed video sensors settings. The existing MVS approaches process video data in offline manner by transmitting them to the local or cloud server for analysis, which requires extra streaming to conduct summarization, huge bandwidth, and are not applicable for integration with industrial Internet of Things (IIoT). This article presents a light-weight convolutional neural network (CNN) and IIoT-based computationally intelligent (CI) MVS framework. Our method uses an IIoT network containing smart devices, Raspberry Pi (RPi) (clients and master) with embedded cameras to capture multiview video data. Each client RPi detects target in frames via light-weight CNN model, analyzes these targets for traffic and crowd density, and searches for suspicious objects to generate alert in the IIoT network. The frames of each client RPi are encoded and transmitted with approximately 17.02% smaller size of each frame to master RPi for final MVS. Empirical analysis shows that our proposed framework can be used in industrial environments for various applications such as security and smart transportation and can be proved beneficial for saving resources. 1 1 [Online]. Available: https://github.com/tanveer-hussain/Embedded-Vision-for-MVS .

Cloud-Assisted Multiview Video Summarization Using CNN and Bidirectional LSTM

The massive amount of video data produced by surveillance networks in industries instigate various challenges in exploring these videos for many applications, such as video summarization (VS), analysis, indexing, and retrieval. The task of multiview video summarization (MVS) is very challenging due to the gigantic size of data, redundancy, overlapping in views, light variations, and interview correlations. To address these challenges, various low-level features and clustering-based soft computing techniques are proposed that cannot fully exploit MVS. In this article, we achieve MVS by integrating deep neural network based soft computing techniques in a two-tier framework. The first online tier performs target-appearance-based shots segmentation and stores them in a lookup table that is transmitted to cloud for further processing. The second tier extracts deep features from each frame of a sequence in the lookup table and pass them to deep bidirectional long short-term memory (DB-LSTM) to acquire probabilities of informativeness and generates a summary. Experimental evaluation on benchmark dataset and industrial surveillance data from YouTube confirms the better performance of our system compared to the state-of-the-art MVS methods.

F-DES: Fast and Deep Event Summarization

In the multimedia era a large volume of video data can be recorded during a certain period of time by multiple cameras. Such a rapid growth of video data requires both effective and efficient multiview video summarization techniques. The users can quickly browse and comprehend a large amount of audiovisual data. It is very difficult in real-time to manage and access the huge amount of video-content-handling issues of interview dependencies significant variations in illumination and presence of many unimportant frames with low activity. In this paper we propose a local-alignment-based FASTA approach to summarize the events in multiview videos as a solution of the aforementioned problems. A deep learning framework is used to extract the features to resolve the problem of variations in illumination and to remove fine texture details and detect the objects in a frame. Interview dependencies among multiple views of video are then captured via the FASTA algorithm through local alignment. Finally object tracking is applied to extract the frames with low activity. Subjective as well as objective evaluations clearly indicate the effectiveness of the proposed approach. Experiments show that the proposed summarization method successfully reduces the video content while keeping momentous information in the form of events. A computing analysis of the system also shows that it meets the requirement of real-time applications.

Diversity-Aware Multi-Video Summarization

Most video summarization approaches have focused on extracting a summary from a single video; we propose an unsupervised framework for summarizing a collection of videos. We observe that each video in the collection may contain some information that other videos do not have, and thus exploring the underlying complementarity could be beneficial in creating a diverse informative summary. We develop a novel diversity-aware sparse optimization method for multi-video summarization by exploring the complementarity within the videos. Our approach extracts a multi-video summary, which is both interesting and representative in describing the whole video collection. To efficiently solve our optimization problem, we develop an alternating minimization algorithm that minimizes the overall objective function with respect to one video at a time while fixing the other videos. Moreover, we introduce a new benchmark data set, Tour20, that contains 140 videos with multiple manually created summaries, which were acquired in a controlled experiment. Finally, by extensive experiments on the new Tour20 data set and several other multi-view data sets, we show that the proposed approach clearly outperforms the state-of-the-art methods on the two problems-topic-oriented video summarization and multi-view video summarization in a camera network.

Framework for Multi-View Video Summarization on Many core GPU

Framework for Multi-View Video Summarization on Many core GPU

Multi-View Video Summarization Using Bipartite Matching Constrained Optimum-Path Forest Clustering

The task of multi-view video summarization is to efficiently represent the most significant information from a set of videos captured for a certain period of time by multiple cameras. The problem is highly challenging because of the huge size of the data, presence of many unimportant frames with low activity, inter-view dependencies, and significant variations in illumination. In this paper, we propose a graph-theoretic solution to the above problems. Semantic feature in form of visual bag of words and visual features like color, texture, and shape are used to model shot representative frames after temporal segmentation . Gaussian entropy is then applied to filter out frames with low activity. Inter-view dependencies are captured via bipartite graph matching. Finally, the optimum-path forest algorithm is applied for the clustering purpose. Subjective as well as objective evaluations clearly indicate the effectiveness of the proposed approach.

On-Line Multi-View Video Summarization for Wireless Video Sensor Network

Battery lifetime is critical for wireless video sensors. To enable battery-powered wireless video sensors, low-power design is required. In this paper, we consider applying multi-view summarization to wireless video sensors to remove redundant contents such that the compression and transmission power can be reduced. A low-complexity online multi-view video summarization scheme is proposed. Experiments show that the proposed summarization method successfully reduces the video content while keeping important events. A power analysis of the system also shows that a significant amount of energy can be saved.