Video Shot Research Articles

Key-frame-based depth propagation for semi-automatic stereoscopic video conversion

In this paper, we propose a key-frame-based bi-directional depth propagation algorithm for semi-automatic 2D-to-3D stereoscopic video conversion. First, key-frames are identified from each video shot based on color motion-compensation errors to prevent high-motion content between any pair of consecutive key frames. Depths for key-frames are manually assigned or rendered by popular computer tools, and then bi-directionally propagated to non-key-frames there between. Our depth propagation algorithm is featured of a multi-pass error correcting procedure for each frame to prevent depth artifacts from being further propagated to adjacent frames. Our proposed algorithm is advantageous in solving the background occlusion/dis-occlusion problem that degrades the performances of traditional depth propagation algorithms. Experimental results show that our scheme is capable of achieving better results against three prior algorithms in view of the qualities of the estimated depth map (e.g., dis-occluded background and object boundaries) and the synthesized stereo views.

Exploiting scene maps and spatial relationships in quasi-static scenes for video face clustering

Video face clustering is a fundamental step in automatically annotating a video in terms of when and where (i.e., in which video shot and where in a video frame) a given person is visible. State-of-the-art face clustering solutions typically rely on the information derived from visual appearances of the face images. This is challenging because of a high degree of variation in these visual appearances due to factors like scale, viewpoint, head pose and facial expression. As a result, either the generated face clusters are not sufficiently pure, or their number is much higher than that of people appearing in the video. A possible way towards improved clustering performance is to analyze visual appearances of faces in specific contexts and take the contextual information into account when designing the clustering algorithm. In this paper, we focus on the context of quasi-static scenes, in which we can assume that the people's positions in a scene are (quasi-)stationary. We present a novel video clustering algorithm that exploits this property to match faces and efficiently propagate face labels across the scope of viewpoints, scale and level of zoom characterizing different frames and shots of a video. We also present a novel publicly available dataset of manually annotated quasi-static scene videos. Experimental assessment on the latter indicates that exploiting information derived by the scene and the spatial relationships between people can substantially improve the clustering performance compared to the state-of-the-art in the field.

Shot Segmentation for Binocular Stereoscopic Video Based on Spatial–Temporal Feature Clustering

Shot segmentation is the key to content-based analysis, index and retrieval of binocular stereoscopic video. To solve the problem of low accuracy of stereoscopic video shot segmentation in which the segmentation method of 2D video is used to segment monocular video sequence, and the disadvantages of some stereoscopic video shot segmentation methods, a shot segmentation method for binocular stereoscopic video based on spatial---temporal feature clustering (STFC) is proposed. In the method, the features of color and brightness of left video frames in temporal domain as well as the depth feature acquired by matching of left and right frames in spatial domain is extracted. The feature differences between frames are calculated and quantified. Then the clustering of feature differences in three-dimensional space is executed, and the optimization and iteration of the classes are implemented to achieve the division of shot boundary. The experimental results show that the proposed method can effectively solve the problems of error and omission, especially the inaccuracy of smooth shot detection in binocular stereo video shot segmentation when compared with the latest existing algorithm. The higher accuracy of segmentation can be achieved.

How to show the interpreter on screen? The normative organization of visual ecologies in multilingual courtrooms with video links

This paper discusses different aspects of participation in judicial settings which combine a reliance on video conferencing technology, to allow for remote defendants in prison, and the presence of interpreters in the courtroom, to interpret for the former when they do not speak the language of the court. In such a multimedia and multilingual courtroom setting, distinctive participation-related concerns emerge with respect to the face to face multilingual courtroom situation. Based on video recordings of naturally occurring courtroom activities, we discuss a relatively neglected issue regarding interpreters’ visibility in video-mediated courtrooms, and show that the ways in which the interpreter is made visible enact relevant participation frames for him/her. The paper shows how the choice of video shots and where these are placed in sequences of talk-in-interaction displays the participants’ view of the interpreter's role within a continuously evolving multimodal participation frame. While interpreters are usually shown in medium shots, and as much as possible not alone but together with the participants whose talk they are interpreting, there are times when the interpreter is made visible through a close shot. The paper discusses such a deviant case and explains how such a move is made relevant by an unusual configuration of footing in the talk to be interpreted.

Weakly supervised object localization and segmentation in videos

We consider the problem of localizing and segmenting objects in weakly labeled video. A video is weakly labeled if it is associated with a tag (e.g. YouTube videos with tags) describing the main object present in the video. It is weakly labeled because the tag only indicates the presence/absence of the object, but does not give the detailed spatial/temporal location of the object in the video. Given a weakly labeled video, our method can automatically localize the object in each frame and segment it from the background. Our method is fully automatic and does not require any user-input. In principle, it can be applied to a video of any object class. We evaluate our proposed method on a dataset with more than 100 video shots. Our experimental results show that our method outperforms other baseline approaches.

What Is Happening in the Video? —Annotate Video by Sentence

Due to the popularity of online video sharing websites such as YouTube, millions of users have treated online video as a source of information and entertainment. Therefore, video annotation has evoked great interest in the past few years. In this paper, we propose a four-step approach to automatically annotate video shots with sentences. The first step is video preprocessing, converting video shot into a sequence of frame images. The second step is to find related candidate elements of the sentence about the video contents. The main elements in the sentence are objects, events, scenes, and modifiers. These candidate elements are gained by searching for similar images with the video frames in our collected image data sets instead of video data sets. The third step is to select the best elements among these candidate ones by a weighted scoring algorithm. The final step is to construct a sentence with the help of a correlation graph algorithm to analyze the relationships among the best elements. The experimental results indicate that our method is effective to annotate videos with sentences. What is more, the weighted scoring algorithm and the correlation graph algorithm that we propose are efficient in developing the experimental performance.

A spatiotemporal restoration of partial color artifacts in archival films

With the rapid growth in video technology and marketability of all forms of visual media, we encounter the need to exploit archive material more effectively and guarantee picture quality. Old films and video often get corrupted by many artifacts due to extensive usage or improper storage. These defects not only lead to reduction in perceptual quality, but also require large transmission bandwidth. In this paper, we present a novel method to restore color image sequences corrupted by partial color artifact. This degradation visible in old color film/video appear as random color patches in the frames. It reduces the perceptual quality of the video. The proposed approach minimizes a convex function, formulated as rank minimization problem, seeking the temporal correlation among the frames in a video sequence. The main feature of the proposed algorithm is its accurate detection and fast restoration of the artifacts present in a video shot.

Fast Video Shot Boundary Detection Technique based on Stochastic Model

A video is composed of set of shots, where shot is defined as a sequence of consecutive frames captured by one camera without interruption. In video shot transition could be a prompt (hard cut) or gradual (fade, dissolve, and wipe). Shot boundary detection is an essential component of video processing. These boundaries are utilized on many aspect of video processing such as video indexing, and video in demand. In this paper, the authors proposed a new shot boundary detection algorithm. The proposed algorithm detects all type of shot boundaries in a high accuracy. The algorithm is developed based on a global stochastic model for video stream. The proposed stochastic model utilizes the joined characteristic function and consequently the joined momentum to model the video stream. The proposed algorithm is implemented and tested against different types of categorized videos. The proposed algorithm detects cuts fades, dissolves, and wipes transitions. Experimental results show that the algorithm has high performance. The computed precision and recall rates validated its performance.

Spatial and temporal scoring for egocentric video summarization

We present a summarization approach for egocentric video. Given hours of video, the proposed method produces a compact storyboard summary of the camera wearer's day. In contrast to traditional keyframe selection techniques, the resulting summary focuses on the most important video shots which reflect high stable salience, discrimination and representativeness. To accomplish this, we utilize egocentric salience cues, motion cues and a selection model to capture stable salience weight, discriminative weight and representative weight of a video shot respectively. We further combine these weights in a unified framework to predict the importance score of a shot, based on which, important shots are selected for the storyboard. Critically, the approach is neither camera-wearer-specific nor object-specific; that means the learned importance metric need not be trained for a given user or context, and it can predict the importance of shots that have never been seen previously. Experimental results on three video datasets across various genres demonstrate that our proposed approach clearly outperforms several state-of-the-art methods.

“Film!”—The Arab Revolutions and the Filmmaker as Amanuensis

In her recent memoir of the Egyptian revolution Ahdaf Soueif records how one day a woman came up to her on Tahrir Square and started instructing her on how to write down the “national epic” unfolding around them. Taking Soueif’s text as an allegory of the relationship between the individual and the people, I explore how an analogous process is enacted in one particular video shot in Cairo on January 25, 2011. Taken together, these two “texts” suggest the emergence of a distinctively revolutionary aesthetic, which is also a politics of obedience.

Supervised framework for automatic recognition and retrieval of interaction: a framework for classification and retrieving videos with similar human interactions

This study presents supervised framework for automatic recognition and retrieval of interactions (SAFARRIs), a supervised learning framework to recognise interactions such as pushing, punching, and hugging, between a pair of human performers in a video shot. The primary contribution of the study is to extend the vectors of locally aggregated descriptors (VLADs) as a compact and discriminative video encoding representation, to solve the complex class partitioning problem of recognising human interaction. An initial codebook is generated from the training set of video shots, by extracting feature descriptors around the spatiotemporal interest points computed across frames. A bag of action words is generated by encoding the first‐order statistics of the visual words using VLAD. Support vector machine classifiers (1 against all) are trained using these codebooks. The authors have verified SAFARRI's accuracy for classification and retrieval (query by example). SAFARRI is free from tracking or recognition of body parts and capable of identifying the region of interaction in video shots. It gives superior retrieval and classification performances over recently proposed methods, on two publicly available human interaction datasets.

An efficient method for video shot boundary detection and keyframe extraction using SIFT-point distribution histogram

In today’s digital era, there are large volumes of long-duration videos resulting from movies, documentaries, sports and surveillance cameras floating over internet and video databases (YouTube). Since manual processing of these videos are difficult, time-consuming and expensive, an automatic technique of abstracting these long-duration videos are very much desirable. In this backdrop, this paper presents a novel and efficient approach of video shot boundary detection and keyframe extraction, which subsequently leads to a summarized and compact video. The proposed method detects video shot boundaries by extracting the SIFT-point distribution histogram (SIFT-PDH) from the frames as a combination of local and global features. In the subsequent step, using the distance of SIFT-PDH of consecutive frames and an adaptive threshold video shot boundaries are detected. Further, the keyframes representing the salient content of each segmented shot are extracted using entropy-based singular values measure. Thus, the summarized video is then generated by combining the extracted keyframes. The experimental results show that our method can efficiently detect shot boundaries under both abrupt and gradual transitions, and even under different levels of illumination, motion effects and camera operations (zoom in, zoom out and camera rotation). With the proposed method, the computational complexity is comparatively less and video summarization is very compact.

Selecting Video Key Frames Based on Relative Entropy and the Extreme Studentized Deviate Test

This paper studies the relative entropy and its square root as distance measures of neighboring video frames for video key frame extraction. We develop a novel approach handling both common and wavelet video sequences, in which the extreme Studentized deviate test is exploited to identify shot boundaries for segmenting a video sequence into shots. Then, video shots can be divided into different sub-shots, according to whether the video content change is large or not, and key frames are extracted from sub-shots. The proposed technique is general, effective and efficient to deal with video sequences of any kind. Our new approach can offer optional additional multiscale summarizations of video data, achieving a balance between having more details and maintaining less redundancy. Extensive experimental results show that the new scheme obtains very encouraging results in video key frame extraction, in terms of both objective evaluation metrics and subjective visual perception.

Reading the Videos: Temporal Labeling for Crowdsourced Time-Sync Videos Based on Semantic Embedding

Recent years have witnessed the boom of online sharing media contents, which raise significant challenges in effective management and retrieval. Though a large amount of efforts have been made, precise retrieval on video shots with certain topics has been largely ignored. At the same time, due to the popularity of novel time-sync comments, or so-called "bullet-screen comments", video semantics could be now combined with timestamps to support further research on temporal video labeling. In this paper, we propose a novel video understanding framework to assign temporal labels on highlighted video shots. To be specific, due to the informal expression of bullet-screen comments, we first propose a temporal deep structured semantic model (T-DSSM) to represent comments into semantic vectors by taking advantage of their temporal correlation. Then, video highlights are recognized and labeled via semantic vectors in a supervised way. Extensive experiments on a real-world dataset prove that our framework could effectively label video highlights with a significant margin compared with baselines, which clearly validates the potential of our framework on video understanding, as well as bullet-screen comments interpretation.

Video Shot Boundary Detection using Gray Level Cooccurrence Matrix

Objectives : The objective of this paper is to find out the abrupt transitions between consecutive shots in a video with less false detection and high F1 score. Method/Analysis: This paper presents a video shot boundary detection approach using Gray Level Cooccurrence Matrix (GLCM). The proposed system can roughly be divided into feature extraction using GLCM and the application of the abrupt shot boundary detection. In the first step, the frames are converted into gray level and GLCM is calculated from each frame in the video. Secondly, correlation coefficient is calculated from the GLCM of two consecutive frames of the video. A threshold is set to identify the shot boundaries of the video. The proposed system can detect abrupt transitions effectively with less false detection in the uncompressed domain. Findings: The proposed system can able to achieve an average F1 score of 93.51%, which is achieve due to the reduced false detection. Novelty/Improvement: The proposed system uses the GLCM matrix directly instead of calculating the contrast, entropy,etc, i.e., the proposed system is purely based on the correlation of the pixel's co-occurrence .The proposed system also reduced the false detection there by increasing the precision and F1 score.

Weighted multi-view key-frame extraction

The extraction of representative key-frames from video shots is very important in video processing and analysis, since it constitutes the basis for several important tasks such as video shot summarization, browsing and retrieval as well as high-level video segmentation. The extracted key-frames should capture a great percentage of the information of a shot content, while at the same time they should not present similar visual information. Clustering or segmentation methods are usually employed to extract key-frames. A major difficulty is caused by the large variety in the visual content of videos. Thus, using a single image descriptor (color, texture etc) to extract key-frames is not always effective, since there is no single descriptor surpassing the others in all video cases. To tackle this problem, we propose an approach for the weighted fusion of several descriptors that automatically estimates the weight of each descriptor. The weights reflect the relevance of each descriptor for the specific video shot. Moreover, they are used to form a composite similarity matrix as the weighted sum of all the similarity matrices corresponding to the individual descriptors. This matrix is then used as input to a spectral clustering algorithm that partitions shot frames into groups. Finally the medoid frame of each group is selected as key-frame. Numerical experiments using a variety of videos demonstrate that our method is capable of efficiently summarizing video shots regardless of the characteristics of the visual content of a video.

Gaze movement-driven random forests for query clustering in automatic video annotation

In the recent years, the rapid increase of the volume of multimedia content has led to the development of several automatic annotation approaches. In parallel, the high availability of large amounts of user interaction data, revealed the need for developing automatic annotation techniques that exploit the implicit user feedback during interactive multimedia retrieval tasks. In this context, this paper proposes a method for automatic video annotation by exploiting implicit user feedback during interactive video retrieval, as this is expressed with gaze movements, mouse clicks and queries submitted to a content-based video search engine. We exploit this interaction data to represent video shots with feature vectors based on aggregated gaze movements. This information is used to train a classifier that can identify shots of interest for new users. Subsequently, we propose a framework that during testing: a) identifies topics (expressed by query clusters), for which new users are searching for, based on a novel clustering algorithm and b) associates multimedia data (i.e., video shots) to the identified topics using supervised classification. The novel clustering algorithm is based on random forests and is driven by two factors: first, by the distance measures between different sets of queries and second by the homogeneity of the shots viewed during each query cluster defined by the clustering procedure; this homogeneity is inferred from the performance of the gaze-based classifier on these shots. The evaluation shows that the use of aggregated gaze data can be exploited for video annotation purposes.

The Effect of Distance on Moral Engagement: Event Related Potentials and Alpha Power are Sensitive to Perspective in a Virtual Shooting Task.

In a shooting video game we investigated whether increased distance reduces moral conflict. We measured and analyzed the event related potential (ERP), including the N2 component, which has previously been linked to cognitive conflict from competing decision tendencies. In a modified Go/No-go task designed to trigger moral conflict participants had to shoot suddenly appearing human like avatars in a virtual reality scene. The scene was seen either from an ego perspective with targets appearing directly in front of the participant or from a bird's view, where targets were seen from above and more distant. To control for low level visual features, we added a visually identical control condition, where the instruction to “shoot” was replaced by an instruction to “detect.” ERP waveforms showed differences between the two tasks as early as in the N1 time-range, with higher N1 amplitudes for the close perspective in the “shoot” task. Additionally, we found that pre-stimulus alpha power was significantly decreased in the ego, compared to the bird's view only for the “shoot” but not for the “detect” task. In the N2 time window, we observed main amplitude effects for response (No-go > Go) and distance (ego > bird perspective) but no interaction with task type (shoot vs. detect). We argue that the pre-stimulus and N1 effects can be explained by reduced attention and arousal in the distance condition when people are instructed to “shoot.” These results indicate a reduced moral engagement for increased distance. The lack of interaction in the N2 across tasks suggests that at that time point response execution dominates. We discuss potential implications for real life shooting situations, especially considering recent developments in drone shootings which are per definition of a distant view.

Invited Paper] TRECVid Semantic Indexing of Video: A 6-Year Retrospective

Semantic indexing, or assigning semantic tags to video samples, is a key component for content-based access to video documents and collections. The Semantic Indexing task has been run at TRECVid from 2010 to 2015 with the support of NIST and the Quaero project. As with the previous High-Level Feature detection task which ran from 2002 to 2009, the semantic indexing task aims at evaluating methods and systems for detecting visual, auditory or multi-modal concepts in video shots. In addition to the main semantic indexing task, four secondary tasks were proposed namely the localization task, the concept pair task, the no annotation task, and the task. It attracted over 40 research teams during its running period. The task was conducted using a total of 1,400 hours of video data drawn from Internet Archive videos with Creative Commons licenses gathered by NIST. 200 hours of new test data was made available each year plus 200 more as development data in 2010. The number of target concepts to be detected started from 130 in 2010 and was extended to 346 in 2011. Both the increase in the volume of video data and in the number of target concepts favored the development of generic and scalable methods. Over 8 millions shots×concepts direct annotations plus over 20 millions indirect ones were produced by the participants and the Quaero project on a total of 800 hours of development data. Significant progress was accomplished during the period as this was accurately measured in the context of the progress task but also from some of the participants' contrast experiments. This paper describes the data, protocol and metrics used for the main and the secondary tasks, the results obtained and the main approaches used by participants.

CWT and CS Algorithm Based Video Watermarking Using Audio Watermark

To increase the robustness of the scheme, we have developed an efficient video watermarking scheme to embed audio watermark based on optimal location. Before embed the audio watermark in to the digital video, the input video sequence is segmented into shots using discrete cosine transform. The segmented video shots are partitioned addicted to number of frames for the embedding process. In this process the PSNR value calculated for each frames and select the best frame by using CS algorithm. The audio data is utilized as a watermark to embed into the digital video sequence. Initially, the audio data is converted into 9 bit sequence and then it is encrypted. In watermarking embedding process, segmented video data is given to the Coiflets wavelet transform and the encrypted audio watermark data is embedded based on optimal location analysis that is carried out using CS algorithm. Later, the original audio data is extracted from the watermarked video using the designed watermarking extraction procedure. The performance of video watermarking is analyzed in terms of various attacks.