Shot Representation Research Articles

Characters Link Shots: Character Attention Network for Movie Scene Segmentation

Movie scene segmentation aims to automatically segment a movie into multiple story units, i.e., scenes, each of which is a series of semantically coherent and time-continual shots. Previous methods have continued efforts on shot semantic association, but few take into account the impact of different semantics on foreground characters and background scenes in movie shots. In particular, the background scene in the shot can adversely affect scene boundary classification. Motivated by the fact that it is the characters who drive the plot development of a movie scene, we build a Character Attention Network (CANet) to detect movie scene boundaries in a character-centric fashion. To eliminate the background clutter, we extract multi-view character semantics for each shot in terms of human bodies and faces. Furthermore, we equip our CANet with two stages of character attention. The first is Masked Shot Attention (MSA) through selective self-attention over similar temporal contexts from multi-view character semantics to yield an enhanced omni-view shot representation, by which the CANet can better handle the variations of characters in pose and appearance. The second is Key Character Attention (KCA) through temporal-aware attention on character reappearances for Bidirectional Long Short-Term Memory (Bi-LSTM) feature association so that linking shots can be focused on those with recurring key characters. We encourage the proposed CANet in learning boundary-discriminative shot features. Specifically, we formulate a Boundary-Aware circle Loss (BAL) to push far apart CANet-features between adjacent scenes, which is also coupled with the cross-entropy loss to drive CANet-features sensitive to scene boundaries. Experimental results on the MovieNet-SSeg and OVSD datasets show that our method achieves superior performance in temporal scene segmentation compared with state-of-the-art methods.

ITI-CERTH participation to TRECVID 2011

This paper provides an overview of the tasks submitted to TRECVID 2011 by ITI-CERTH. ITICERTH participated in the Known-item search (KIS) as well as in the Semantic Indexing (SIN) and the Event Detection in Internet Multimedia (MED) tasks. In the SIN task, techniques are developed, which combine motion information with existing well-performing descriptors such as SURF, Random Forests and Bag-of-Words for shot representation. In the MED task, the trained concept detectors of the SIN task are used to represent video sources with model vector sequences, then a dimensionality reduction method is used to derive a discriminant subspace for recognizing events, and, nally, SVMbased event classiers are used to detect the underlying video events. The KIS search task is performed by employing VERGE, which is an interactive retrieval application combining retrieval functionalities in various modalities and exploiting implicit user feedback.

Virtual Camera Control Based on Cinematographic Rules of Camera Shot and Camera Angle

This research is focused on developing automatic virtual camera control in 3D video game to support a filmmaking based on machinima. Rule-based virtual camera control defines the camera position and the angle. A virtual camera is developed by representing camera shot and camera angle since it can control its position and rotation automatically. Knowledge acquisition for camera shot and camera angle is conducted by simulating virtual camera to the character based on rules of camera shot which are close-up, medium-shot, and long-shot, and rules of camera angle which are eye-level, low angle and high angle. The constrained areas are represented by defining paths. The paths have the function of track references to define position and rotation for the virtual camera, so the virtual camera can capture character vertically, horizontally, and diagonally. Data of constrained areas are representation of camera shot and camera angle rules. The data are used as knowledge base and rules in the term of perspective packages. The virtual camera is generated using Genetic algorithm by randomizing a number of position and rotation value as initial population, then the system calculates the value to find ideal virtual camera which can fulfil the rules in perspective packages. An implementation of the research is conducted by developing a game in which the automatic virtual camera is generated to produce a short animation with duration of 120 seconds. The rules of transition are implemented using different cut as type of transition in every 4 seconds, which means that there are 30 cut to produce. The result shows that the method used in this research reflects the expectation. All the 30 cuts produced in the game are captured using automatic virtual camera. With frame rate at 40-60 fps, the virtual camera is generated in 2 to 16 generations, with duration from 16 to 37 milliseconds.

Classification of Cinematographic Shots Using Lie Algebra and its Application to Complex Event Recognition

In this paper, we propose a discriminative representation of a video shot based on its camera motion and demonstrate how the representation can be used for high level multimedia tasks like complex event recognition. In our technique, we assume that a homography exists between a pair of subsequent frames in a given shot. Using purely image-based methods, we compute homography parameters that serve as coarse indicators of the ambient camera motion. Next, using Lie algebra, we map the homography matrices to an intermediate vector space that preserves the intrinsic geometric structure of the transformation. The mappings are stacked temporally to generate vector time-series per shot. To extract meaningful features from time-series, we propose an efficient linear dynamical system based technique. The extracted temporal features are further used to train linear SVMs as classifiers for a particular shot class. In addition to demonstrating the efficacy of our method on a novel dataset, we extend its applicability to recognize complex events in large scale videos under unconstrained scenarios. Our empirical evaluations on eight cinematographic shot classes show that our technique performs close to approaches that involve extraction of 3-D trajectories using computationally prohibitive structure from motion techniques.

Interactive Exploration of Surveillance Video through Action Shot Summarization and Trajectory Visualization

We propose a novel video visual analytics system for interactive exploration of surveillance video data. Our approach consists of providing analysts with various views of information related to moving objects in a video. To do this we first extract each object's movement path. We visualize each movement by (a) creating a single action shot image (a still image that coalesces multiple frames), (b) plotting its trajectory in a space-time cube and (c) displaying an overall timeline view of all the movements. The action shots provide a still view of the moving object while the path view presents movement properties such as speed and location. We also provide tools for spatial and temporal filtering based on regions of interest. This allows analysts to filter out large amounts of movement activities while the action shot representation summarizes the content of each movement. We incorporated this multi-part visual representation of moving objects in sViSIT, a tool to facilitate browsing through the video content by interactive querying and retrieval of data. Based on our interaction with security personnel who routinely interact with surveillance video data, we identified some of the most common tasks performed. This resulted in designing a user study to measure time-to-completion of the various tasks. These generally required searching for specific events of interest (targets) in videos. Fourteen different tasks were designed and a total of 120 min of surveillance video were recorded (indoor and outdoor locations recording movements of people and vehicles). The time-to-completion of these tasks were compared against a manual fast forward video browsing guided with movement detection. We demonstrate how our system can facilitate lengthy video exploration and significantly reduce browsing time to find events of interest. Reports from expert users identify positive aspects of our approach which we summarize in our recommendations for future video visual analytics systems.

Diagrams for the Leksell Gamma Knife 4C automatic positioning system accessible shot coordinates

We have developed a graphic representation (APS shot-range diagrams) of achievable shot coordinates with the automatic positioning system (APS) for Gamma Knife stereotactic radiosurgery. We formulated a mathematical relationship between the APS coordinate system and the Leksell frame coordinate system (frame-system) by considering the relative translation and rotation. Using the transformation equations, we calculated the Y-Z coordinate domain in the frame-system (domain B), which corresponds to the known coordinate domain achievable in the APS coordinate system (domain A). APS shot-range diagrams were presented by overlaying of domain B over domain A so that one can easily visualize the achievable shot coordinates with APS for various gamma angles. Domains A and B are identical for a 90° gamma angle, but they are different for other gamma angles. Applications of the diagrams to clinical cases, in which Leksell GammaPlan treatment planning software automatically set some of the shots in a treatment plan as trunnion shots because of the APS coordinate limit, enabled us to avoid the trunnion shots, and consequently to achieve much faster treatment delivery. In conclusion, APS shot-range diagrams can be used for achieving more efficient treatment delivery by avoiding unnecessary trunnion shots with a Gamma Knife unit equipped with the APS.

Content based Video Retrieval using Latent Semantic Indexing and Color, Motion and Edge Features

Optimal efficiency of the retrieval techniques depends on the search methodologies that are used in the video processing system. The use of inappropriate search methodologies may make the processing system ineffective. Hence, an effective video segmentation and retrieval system is an essential prerequisite for searching a relevant video from a huge collection of videos. In this paper we propose a video retrieval system based on the integration of various visual cues. In contrast to key-frame based representation of shot, our approach analyzes all frames within a shot to construct a compact representation of video shot. In feature extraction step we extract quantized color, motion and edge density features. A similarity measure is defined using LSI (Latent semantic indexing) to locate the occurrence of similar video clips in the database. Our approach is able to fully exploit the spatio-temporal contents of the video. Experimental results indicate that the proposed algorithm is effective and outperforms some existing technique. The detailed result analysis and graphs supports the effectiveness and correctness of the system. General Terms Image/Video Processing, Video Retrieval

An Efficient Near-Duplicate Video Shot Detection Method Using Shot-Based Interest Points

We propose a shot-based interest point selection approach for effective and efficient near-duplicate search over a large collection of video shots. The basic idea is to eliminate the local descriptors with lower frequencies among the selected video frames from a shot to ensure that the shot representation is compact and discriminative. Specifically, we propose an adaptive frame selection strategy called furthest point voronoi (FPV) to produce the shot frame set according to the shot content and frame distribution. We describe a novel strategy named reference extraction (RE) to extract the shot interest descriptors from a keyframe with the support of the selected frame set. We demonstrate the effectiveness and efficiency of the proposed approaches with extensive experiments.

INTEGRATION OF COLOR AND MOTION FEATURES FOR VIDEO RETRIEVAL

The usefulness of a video database depends on whether the video of interest can be easily located. In this paper, we propose a video retrieval algorithm based on the integration of several visual cues. In contrast to key-frame based representation of shot, our approach analyzes all frames within a shot to construct a compact representation of video shot. In the video matching step, by integrating the color and motion features, a similarity measure is defined to locate the occurrence of similar video clips in the database. Therefore, our approach is able to fully exploit the spatio-temporal information contained in video. Experimental results indicate that the proposed approach is effective and outperforms some existing technique.

Global Motion Representation of Video Shot Based on Vector Quantization Index Histogram

In this Letter, we propose a novel method of low-level global motion feature description based on Vector Quantization (VQ) index histograms of motion feature vectors (MFVVQIH) for the purpose of video shot retrieval. The contribution lies in three aspects: first, we use VQ to eliminate singular points in the motion feature vector space; second, we utilize the global motion feature vector index histogram of a video shot as the global motion signature; third, video shot retrieval based on index histograms instead of original motion feature vectors guarantees the low computation complexity, and thus assures a real-time video shot retrieval. Experimental results show that the proposed scheme has high accuracy and low computation complexity.

SVM-based Relevance Feedback for semantic video retrieval

This paper presents a novel method for efficient key frame extraction from video shot representation and employs a Support-Vector-Machine-based Relevance Feedback (SVM-RF) to bridging semantic gap between low-level feature and high-level concepts of shots. We introduce a new approach for key frame extraction using a hierarchical approach based on clustering. Using this key frame representation, the most representative key frame is then selected for each shot. Furthermore, our system incorporates user to judge about the result of retrieval and labelled retrieved shot in two groups, relevant and irrelevant. Then, by mean feature of relevant and irrelevant shots train an SVM classifier. In the next step, video database is classified in two groups, relevant and irrelevant shots. Suitable Graphic User Interface (GUI) is shown for capturing RF of user. This process continued until user satisfied with results. The proposed system is checked over collected shots from Trecvid2001 database and home videos include 800 shots of different concepts (10 semantic groups). Experimental results demonstrate the effectiveness of the proposed method.

Segmentation and tracking of multiple video objects

This paper describes a technique that produces a content-based representation of a video shot composed by a background (still) mosaic and one or more foreground moving objects. Segmentation of moving objects is based on ego-motion compensation and on background modelling using tools from robust statistics. Region matching is carried out by an algorithm that operates on the Mahalanobis distance between region descriptors in two subsequent frames and uses singular value decomposition to compute a set of correspondences satisfying both the principle of proximity and the principle of exclusion. The sequence is represented as a layered graph, and specific techniques are introduced to cope with crossing and occlusion. Examples of MPEG-4 (main profile) encoding are reported.

A new key frame representation for video segment retrieval

In this paper, we propose an optimal key frame representation scheme based on global statistics for video shot retrieval. Each pixel in this optimal key frame is constructed by considering the probability of occurrence of those pixels at the corresponding pixel position among the frames in a video shot. Therefore, this constructed key frame is called temporally maximum occurrence frame (TMOF), which is an optimal representation of all the frames in a video shot. The retrieval performance of this representation scheme is further improved by considering the k pixel values with the largest probabilities of occurrence and the highest peaks of the probability distribution of occurrence at each pixel position for a video shot. The corresponding schemes are called k-TMOF and k-pTMOF, respectively. These key frame representation schemes are compared to other histogram-based techniques for video shot representation and retrieval. In the experiments, three video sequences in the MPEG-7 content set were used to evaluate the performances of the different key frame representation schemes. Experimental results show that our proposed representations outperform the alpha-trimmed average histogram for video retrieval.

Layered Representation of a Video Shot with Mosaicing

This paper presents a motion segmentation method useful for representing efficiently a video shot as a static mosaic of the background plus sequences of the objects moving in the foreground. This generates an MPEG-4 compliant, layered representation useful for video coding, editing and indexing. First, a mosaic of the static background is computed by estimating the dominant motion of the scene. This is achieved by tracking features over the video sequence and using a robust technique that discards features attached to the moving objects. The moving objects get removed in the final mosaic by computing the median of the grey levels. Then, segmentation is obtained by taking the pixelwise difference between each frame of the original sequence and the mosaic of the background. To discriminate between the moving object and noise, temporal coherence is exploited by tracking the object in the binarised difference image sequence. The automatic computation of the mosaic and the segmentation procedure are illustrated with real sequences experiments. Examples of coding and content-based manipulation are also shown.

VIDEO CONTENT REPRESENTATION FOR SHOT RETRIEVAL AND SCENE EXTRACTION

In this paper, we present a novel scheme on video content representation by exploring the spatio-temporal information. A pseudo-object-based shot representation containing more semantics is proposed to measure shot similarity and force competition approach is proposed to group shots into scene based on content coherences between shots. Two content descriptors, color objects: Dominant Color Histograms (DCH) and Spatial Structure Histograms (SSH), are introduced. To represent temporal content variations, a shot can be segmented into several subshots that are of coherent content, and shot similarity measure is formulated as subshot similarity measure that serves to shot retrieval. With this shot representation, scene structure can be extracted by analyzing the splitting and merging force competitions at each shot boundary. Experimental results on real-world sports video prove that our proposed approach for video shot retrievals achieve the best performance on the average recall (AR) and average normalized modified retrieval rank (ANMRR), and Experiment on MPEG-7 test videos achieves promising results by the proposed scene extraction algorithm.