Video Shot Research Articles

Background subtraction via time continuity and texture consistency constraints.

Background subtraction has attracted enormous interest in the field of moving object detection. However, when there are complex scenarios such as illumination changes, dynamic background, and noise, the moving object area obtained by background subtraction often has holes, noise, and shadows. This paper proposes a novel background update model based on matrix factorization, which uses the temporal continuity of video content to solve the problems of holes, noise, and shadows. Moreover, in some cases, the texture consistency of the object is also a factor worth considering. The neighborhood weighed local binary pattern (NWLBP) is introduced to optimize the background update model, which is very effective for suppressing background or foreground shadow. The effectiveness of our method is confirmed by extensive experiments on public data sets and real shot video. Compared with the existing state-of-the-art moving object detection methods, the proposed methods can accurately establish the background model and locate the moving object region robustly.

Synthetic defocus and look-ahead autofocus for casual videography

In cinema, large camera lenses create beautiful shallow depth of field (DOF), but make focusing difficult and expensive. Accurate cinema focus usually relies on a script and a person to control focus in realtime. Casual videographers often crave cinematic focus, but fail to achieve it. We either sacrifice shallow DOF, as in smartphone videos; or we struggle to deliver accurate focus, as in videos from larger cameras. This paper is about a new approach in the pursuit of cinematic focus for casual videography. We present a system that synthetically renders refocusable video from a deep DOF video shot with a smartphone, and analyzes future video frames to deliver context-aware autofocus for the current frame. To create refocusable video, we extend recent machine learning methods designed for still photography, contributing a new dataset for machine training, a rendering model better suited to cinema focus, and a filtering solution for temporal coherence. To choose focus accurately for each frame, we demonstrate autofocus that looks at upcoming video frames and applies AI-assist modules such as motion, face, audio and saliency detection. We also show that autofocus benefits from machine learning and a large-scale video dataset with focus annotation, where we use our RVR-LAAF GUI to create this sizable dataset efficiently. We deliver, for example, a shallow DOF video where the autofocus transitions onto each person before she begins to speak. This is impossible for conventional camera autofocus because it would require seeing into the future.

샷 분석을 통한 대통령선거 TV토론회 연구

본 연구는 대통령선거 후보자 TV토론회에 나타난 영상 샷 구성의 특성을 분석한다. 구체적으로 후보자 TV토론회 샷이 양적으로 어떤 특성을 보이는지, 토론회 포맷에 따른 샷 구성은 어떤 특성을 보이는지 연구한다. 한국 2017년 대통령선거 TV토론회와 미국 2016년 대통령선거 TV토론회 샷을 분석대상으로 삼았다. 분석결과, 샷의 빈도와 유형에 있어서 토론회 간 양적 차이가 있음을 확인할 수 있다. 포맷에 따른 카메라 워킹과 샷의 차이에서 오는 영상 이미지에도 차이가 존재하는 것으로 확인된다. 후보들 간 토론의 비중이나 토론의 긴장감과 상관없이 일대일 샷을 모든 후보에게 획일적으로 동일한 횟수를 부여한다는 것은 역동성을 떨어뜨리는 요인이다. 공정성을 해칠 정도가 아니라면 유연성 있는 샷 구성을 허용하고, 방송사가 자율성을 갖고 샷을 구성할 수 있도록 하는 것이 필요하다.

Video instance search via spatial fusion of visual words and object proposals

Most popular systems for object instance search are based on the bag-of-visual-word model. The inherent weaknesses of this standard model such as quantization error, unstructured representation, burstiness phenomenon are to some extent solved. However, it has a serious problem of searching small objects on a database with cluttered background. In many situations, even the irrelevant objects which share the same texture or shape with a query object get higher score than relevant ones. To overcome this problem, we propose a novel fusion method to significantly boost the accuracy of instance search systems. Firstly, we use the state-of-the-art object detector with denser feature for finding object bounding box and similarity score. Secondly, to exploit the spatial relationship of each visual word with an object proposal, a detected area that might contain a query object, we define three categories of visual word pairs, i.e., discriminative, weak relevant, and context inferred ones. Finally, we propose a new re-ranking scheme with three weighting functions corresponding to the three categories of visual word pairs to compute the final similarity score between a query topic and a video shot. To illustrate the efficiency of the proposed method, we conduct experiments on datasets which have a wide variety of types of query objects. Experimental results on TRECVID Instance Search datasets (INS2013 and INS2014) show the superiority of our proposed method over the state-of-the-art approaches.

Shot boundary detection based on block-wise principal component analysis

With the rapid development of digital video, shot boundary detection (SBD) has attracted much attention since it is the fundamental preprocessing for video indexing, annotation, retrieval, and other content-based operations. However, most state-of-the-art SBD methods are based on the spatial features of video image, and the overall characteristics of video shots are not fully considered. We propose a feature extraction method based on shot characteristics and a more robust SBD process. First, a video is divided into several segments, the segments containing consecutive video frames inside a shot are considered as training segments and others are called candidate segments. Afterward, using block-wise principal component analysis on the training segments, shot eigenspaces are established. The video frames in the candidate segment are then projected onto the corresponding shot eigenspace to extract the feature vectors. Finally, analysis and pattern matching for feature vectors are performed to extract the video shot boundary. Experiments on TRECVID test data demonstrate that the mean values of F1 in cut transition detection and gradual transition (GT) detection of our method are 0.901 and 0.866, respectively, obviously higher than the values of the compared methods, especially in GT detection, thus providing better accuracy in SBD.

ROAM: A Rich Object Appearance Model with Application to Rotoscoping.

Rotoscoping, the detailed delineation of scene elements through a video shot, is a painstaking task of tremendous importance in professional post-production pipelines. While pixel-wise segmentation techniques can help for this task, professional rotoscoping tools rely on parametric curves that offer the artists a much better interactive control on the definition, editing and manipulation of the segments of interest. Sticking to this prevalent rotoscoping paradigm, we propose a novel framework to capture and track the visual aspect of an arbitrary object in a scene, given an initial closed outline of this object. This model combines a collection of local foreground/background appearance models spread along the outline, a global appearance model of the enclosed object and a set of distinctive foreground landmarks. The structure of this rich appearance model allows simple initialization, efficient iterative optimization with exact minimization at each step, and on-line adaptation in videos. We further extend this model by so-called trimaps which serve as an input to alpha-matting algorithms to allow truly seamless compositing. To this end, we leverage local classifiers attached to the roto-curves to define a confidence measure that is well-suited to define trimaps with adaptive band-widths. The resulting trimaps are parametric, temporally consistent and remain fully editable by the artist. We demonstrate qualitatively and quantitatively the merit of this framework through comparisons with tools based on either dynamic segmentation with a closed curve or pixel-wise binary labelling.

Illumination-Invariant Video Cut-Out Using Octagon Sensitive Optimization

This paper presents an effective video cut-out approach, which can be utilized to segment the moving object in video shots. We first introduce the Octagon-Sensitive-Filtering (OSF) and its illumination invariant feature (IIF), which is computed on each pixel of the image via adding contributions from neighboring pixels. We integrate our IIF into the variational model and obtain the seeds during preprocessing to help address large displacement and illumination changes. An effective seed update method based on tracking-then-refinement based on IIF is presented to compensate for location ambiguities, and the strategy is effective to deal with illumination variances and objects deformation. Furthermore, we apply the IIF-based graph-cut to deal with fuzzy boundaries. Multiple experiments on quantitative challenging datasets have shown the robustness, high-quality video cut-out and efficiency of our approach to acute variances of illumination and complex motion.

Deep Reinforcement Learning for Query-Conditioned Video Summarization

Query-conditioned video summarization requires to (1) find a diverse set of video shots/frames that are representative for the whole video, and that (2) the selected shots/frames are related to a given query. Thus it can be tailored to different user interests leading to a better personalized summary and differs from the generic video summarization which only focuses on video content. Our work targets this query-conditioned video summarization task, by first proposing a Mapping Network (MapNet) in order to express how related a shot is to a given query. MapNet helps establish the relation between the two different modalities (videos and query), which allows mapping of visual information to query space. After that, a deep reinforcement learning-based summarization network (SummNet) is developed to provide personalized summaries by integrating relatedness, representativeness and diversity rewards. These rewards jointly guide the agent to select the most representative and diversity video shots that are most related to the user query. Experimental results on a query-conditioned video summarization benchmark demonstrate the effectiveness of our proposed method, indicating the usefulness of the proposed mapping mechanism as well as the reinforcement learning approach.

AN IMPROVED ACTION KEY FRAMES EXTRACTION ALGORITHM FOR COMPLEX COLOUR VIDEO SHOT SUMMARIZATION

Key frame extraction is one of the critical techniques in computer vision fields such as video search, video identification and video forgeries detection. The extracted key frames should be sufficient key frames that preserve main actions in a video with compact representation. The objective of this work is to improve our previous action key frames extraction algorithm (AKF) by adapting a threshold which selects action key frames as final key frames. The threshold adaptation was achieved by using the mean value, the standard deviation, and the L1-norm instead of the comparison of user summaries evaluation method to obtain a fully automatic video summarisation algorithm, and by eliminating the conditions in selecting the final key frames to reduce the complexity of the algorithm. We have validated our proposed Improved AKF on complex colour video shots instead of the simple grey level video shots. The Improved AKF algorithm was able to extract a compact number of action key frames by preventing redundant key frames, reduce processing complexity, and preserve sufficient information about the main actions in a video shot. We then evaluated the Improved AKF algorithm with the-state-of-the-art algorithms in terms of compression ratio using Paul videos and Shih-Tang dataset. The evaluation results showed that the Improved AKF algorithm achieved better compression ratio and retained sufficient information in the extracted action key frames under different testing video shots. Therefore, the improved AKF algorithm is a suitable technique for applications in computer vision fields such as passive object-based video authentication systems

Enhanced Local Directional Pattern based video shot boundary detection and automatic synchronization for STB quality inspection

Enhanced Local Directional Pattern based video shot boundary detection and automatic synchronization for STB quality inspection

ERP/MMR Algorithm for Classifying Topic‐Relevant and Topic‐Irrelevant Visual Shots of Documentary Videos

We propose and evaluate a video summarization method based on a topic relevance model, a maximal marginal relevance (MMR), and discriminant analysis to generate a semantically meaningful video skim. The topic relevance model uses event‐related potential (ERP) components to describe the process of topic relevance judgment. More specifically, the topic relevance model indicates that N400 and P600, which have been successfully applied to the mismatch process of a stimulus and the discourse‐internal reorganization and integration process of a stimulus, respectively, are used for the topic mismatch process of a topic‐irrelevant video shot and the topic formation process of a topic‐relevant video shot. To evaluate our proposed ERP/MMR‐based method, we compared the video skims generated by the ERP/MMR‐based, ERP‐based, and shot boundary detection (SBD) methods with ground truth skims. The results showed that at a significance level of 0.05, the ROUGE‐1 scores of the ERP/MMR method are statistically higher than those of the SBD method, and the diversity scores of the ERP/MMR method are statistically higher than those of the ERP method. This study suggested that the proposed method may be applied to the construction of a video skim without operational intervention, such as the insertion of a black screen between video shots.

Video summarization using event‐related potential responses to shot boundaries in real‐time video watching

Our aim was to develop an event‐related potential (ERP)‐based method to construct a video skim consisting of key shots to bridge the semantic gap between the topic inferred from a whole video and that from its summary. Mayer's cognitive model was examined, wherein the topic integration process of a user evoked by a visual stimulus can be associated with long‐latency ERP components. We determined that long‐latency ERP components are suitable for measuring a user's neuronal response through a literature review. We hypothesized that N300 is specific to the categorization of all shots regardless of topic relevance, N400 is specific for the semantic mismatching process for topic‐irrelevant shots, and P600 is specific for the context updating process for topic‐relevant shots. In our experiment, the N400 component led to more negative ERP signals in response to topic‐irrelevant shots than to topic‐relevant shots and showed a fronto‐central scalp pattern. P600 elicited more positive ERP signals for topic‐relevant shots than for topic‐irrelevant shots and showed a fronto‐central scalp pattern. We used discriminant and artificial neural network (ANN) analyses to decode video shot relevance and observed that the ANN produced particularly high success rates: 91.3% from the training set and 100% from the test set.

Effect of sequential video shot comprehensibility on attentional synchrony: A comparison of children and adults

To comprehend edited video, viewers must infer the meaning conveyed by successive video shots (i.e., continuous video segments separated by edit points, such as camera cuts). The central question here was whether comprehension-related top-down cognitive processes drive eye movements during sequential processing of video montage. Eye movements were recorded as 4 year olds and adults (n = 62) watched a video with the same constituent shots in either normal or random sequence. The key analyses compared eye movements to constituent shots when presented in normal order with those to the same shots presented in random order. The dependent variable was attentional synchrony or the extent to which viewers looked at the same location at the same time, indicating commonality of processing the video. This was calculated as the bivariate contour ellipse area within which points of gaze fell during each video frame. Results indicated that children were more scattered in their gaze locations than adults. Viewers became more similar to each other as normal vignettes unfolded over time; this was especially true in adults and possibly reflects a growing and shared understanding of the content. Conversely, adult attentional synchrony was reduced when watching random shot sequences. Thus, attentional synchrony during normal video viewing is driven not only by salient visual features, such as movement and areas of high contrast, but also, by the unfolding sequential comprehension of video montage, especially in adults. Differences between children and adults indicate that this top-down control of eye movements while watching video changes systematically over development.

Deep Video Stabilization Using Adversarial Networks

AbstractVideo stabilization is necessary for many hand‐held shot videos. In the past decades, although various video stabilization methods were proposed based on the smoothing of 2D, 2.5D or 3D camera paths, hardly have there been any deep learning methods to solve this problem. Instead of explicitly estimating and smoothing the camera path, we present a novel online deep learning framework to learn the stabilization transformation for each unsteady frame, given historical steady frames. Our network is composed of a generative network with spatial transformer networks embedded in different layers, and generates a stable frame for the incoming unstable frame by computing an appropriate affine transformation. We also introduce an adversarial network to determine the stability of apiece of video. The network is trained directly using the pair of steady and unsteady videos. Experiments show that our method can produce similar results as traditional methods, moreover, it is capable of handling challenging unsteady video of low quality, where traditional methods fail, such as video with heavy noise or multiple exposures. Our method runs in real time, which is much faster than traditional methods.

Generative Model Based Video Shot Boundary Detection for Automated Surveillance

Video shot boundary detection (SBD) or video cut detection is one of the fundamental processes of video-processing with respect to semantic understanding, contextual information accumulation, labeling, content-based information retrieval and many more applications, such as video surveillance and monitoring. In this work, the authors have proposed a generative-model based framework for detecting shot boundaries in between the frames of a video segment. To generate a model of shot-boundaries, the authors have applied the concepts of Support Vector Machine to estimate the distance between any two images, and then, have generated a Gaussian Mixture Model from the estimated distances. Next, a Bayesian Estimation process checks the presence of boundaries in between the images by exploiting the Gaussian Mixture-based boundary model. Further, the authors have used the principles of Compressive Sensing to reduce the overhead of boundary detection process without losing of important information.

Gradual transition detection in shot boundary using gradual curve point

Video shot boundary detection (SBD) is a fundamental step in automatic video content analysis. Gradual transition is based on the gradual curves characterised by distribution of average edge information in the sequence of frames. An average edge frame is obtained from optimal edge detector. Features are extracted based on the minimum difference between the sequences of frames. The minimum variance in the local frame sequence is chosen as a gradual curve point. Finally, the performance is computed based on the precision (quality) and recall (quantity) value.

Video Searching and Fingerprint Detection by Using the Image Query and PlaceNet-Based Shot Boundary Detection Method

This work presents a novel shot boundary detection (SBD) method based on the Place-centric deep network (PlaceNet), with the aim of using video shots and image queries for video searching (VS) and fingerprint detection. The SBD method has three stages. In the first stage, we employed Local Binary Pattern-Singular Value Decomposition (LBP-SVD) features for candidate shot boundaries selection. In the second stage, we used the PlaceNet to select the shot boundary by semantic labels. In the third stage, we used the Scale-Invariant Feature Transform (SIFT) descriptor to eliminate falsely detected boundaries. The experimental results show that our SBD method is effective on a series of SBD datasets. In addition, video searching experiments are conducted by using one query image instead of video sequences. The results under several image transitions by using shot fingerprints have shown good precision.

Keyframe extraction from laparoscopic videos based on visual saliency detection

Background and objective: Laparoscopic surgery offers the potential for video recording of the operation, which is important for technique evaluation, cognitive training, patient briefing and documentation. An effective way for video content representation is to extract a limited number of keyframes with semantic information. In this paper we present a novel method for keyframe extraction from individual shots of the operational video.Methods: The laparoscopic video was first segmented into video shots using an objectness model, which was trained to capture significant changes in the endoscope field of view. Each frame of a shot was then decomposed into three saliency maps in order to model the preference of human vision to regions with higher differentiation with respect to color, motion and texture. The accumulated responses from each map provided a 3D time series of saliency variation across the shot. The time series was modeled as a multivariate autoregressive process with hidden Markov states (HMMAR model). This approach allowed the temporal segmentation of the shot into a predefined number of states. A representative keyframe was extracted from each state based on the highest state-conditional probability of the corresponding saliency vector.Results: Our method was tested on 168 video shots extracted from various laparoscopic cholecystectomy operations from the publicly available Cholec80 dataset. Four state-of-the-art methodologies were used for comparison. The evaluation was based on two assessment metrics: Color Consistency Score (CCS), which measures the color distance between the ground truth (GT) and the closest keyframe, and Temporal Consistency Score (TCS), which considers the temporal proximity between GT and extracted keyframes. About 81% of the extracted keyframes matched the color content of the GT keyframes, compared to 77% yielded by the second-best method. The TCS of the proposed and the second-best method was close to 1.9 and 1.4 respectively.Conclusions: Our results demonstrated that the proposed method yields superior performance in terms of content and temporal consistency to the ground truth. The extracted keyframes provided highly semantic information that may be used for various applications related to surgical video content representation, such as workflow analysis, video summarization and retrieval.

Screen recording segmentation to scenes for eye-tracking analysis

In usability studies involving eye-tracking, quantitative analysis of gaze data requires the information about so called scene occurrences. Scene ocurrences are time segments during which the application user interface remains more-less static, so gaze events (e.g., fixations) can be mapped to the particular areas of interest (user interface elements). The scene occurrences typically start and end by user interface changes such as page-to-page transitions, menu expansions, overlay propmts, etc. Normally, one would record such changes programmatically through application logging, yet in many studies, this is not possible. For example, in an early-prototype mobile-app testing, only a camera recording of a smart device screen is often available as evidence. In such cases, analysts must manually annotate the recordings. To reduce the need for manual annotation of scene occurrences, we present an image processing method for segmenting user interface video recordings. The method exploits specific properties of user interface recordings, which greatly differ from real world video shots (for which many segmentation methods exist). The core of our method lies in the use of SSIM and SIFT similarity metrics used on video frames (with several pre-processing and filtering procedures). The main advantage of our method is, that it requires no training data apart from single screenshot example for each scene (to which the recording frames are compared). The method is also able to work with user finger overlays, which are always present in mobile device recordings. We evaluate the accuracy of our method over recordings from several real-life studies and compare it with other image similarity techniques.

A Novel Solution for Scaling Video Shot Boundary Detection Based on Hadoop

Shot Boundary Detection (SBD) is an important step required by CBVIR systems. In order to perform scalable SBD, a MapReduce based solution is proposed. So, instead of handling consecutive frames in a sequential manner, they can be processed in a fully parallel way. Usually, in the sequential case, descriptors of consecutive frames are compared and shot boundaries are detected if significant variations have occurred. It seems simple, but it can take centuries to processes immense multimedia datasets. Then, based on the transitivity of similarity relation, resemblance measurement between distant frames is calculated, and shout boundaries are extracted respectively by Mapper and Reducer routines. The experiment results show that the proposed solution outperforms the sequential traditional methods and can be applied to a large-scale multimedia datasets.