Shot Change Research Articles

Enhanced shot boundary detection using video text information

Shot boundary detection is a pre-requisite process for content-based video indexing and retrieval, an application domain that has attracted much research and consumer attentions due to the explosive growth of video data available and accessible to users worldwide. Currently, a number of edge-based techniques have been proposed in the literature for detecting abrupt shot boundaries to avoid the influence of flashlights common in many video types, such as sports, news, entertainment, and interviews videos. However, these techniques are susceptible to miss and false detections of abrupt shot changes. Our study shows that one main reason for these errors is due to the presence of superimposed texts that are common in various video genres. To address this problem, we present in this paper an efficient method that utilizes the edge type - text-edge (edge in text region) or non-text-edge (edge in non-text region) - to reduce erroneous detection due to the sudden appearance and disappearance of superimposed text. Extensive experiments have been conducted and the results show that our proposed method, as compared with other existing methods, can obtain better performance in detecting abrupt shot boundaries and differentiating them from the effects of superimposed text.

Automated Video Segmentation for Lecture Videos

Video, a rich information source, is commonly used for capturing and sharing knowledge in learning systems. However, the unstructured and linear features of video introduce difficulties for end users in accessing the knowledge captured in videos. To extract the knowledge structures hidden in a lengthy, multi-topic lecture video and thus make it easily accessible, we need to first segment the video into shorter clips by topic. Because of the high cost of manual segmentation, automated segmentation is highly desired. However, current automated video segmentation methods mainly rely on scene and shot change detection, which are not suitable for lecture videos with few scene/shot changes and unclear topic boundaries. In this article we investigate a new video segmentation approach with high performance on this special type of video: lecture videos. This approach uses natural language processing techniques such as noun phrases extraction, and utilizes lexical knowledge sources such as WordNet. Multiple linguistic-based segmentation features are used, including content-based features such as noun phrases and discourse-based features such as cue phrases. Our evaluation results indicate that the noun phrases feature is salient.

A MULTI-EXPERT SYSTEM FOR SHOT CHANGE DETECTION IN MPEG MOVIES

Shot Change Detection (SCD) in MPEG coded videos is a complex and still open research problem whose interest is growing up more and more due to the diffusion of Video Databases and Digital Libraries. Techniques providing fully satisfactory performances on complex video domains are not yet available even if a number of proposals exist; such proposals show very often to be complementary in their results. In this context, the Authors investigated the use of Multi-Expert Systems (MES) for approaching the SCD problem. In the present paper, we propose and discuss a strategy to select the SCD techniques to be combined and a method for choosing an effective combining rule. In order to assess the performance of the proposed MES, we set up a database that is significantly wider than the ones commonly used in the field. Experimental results demonstrate that the proposed system performs better than each of the single SCD technique considered.

SUDDEN GLOBAL SPATIAL-TEMPORAL CHANGE DETECTION AND ITS APPLICATIONS

We are watching the news on TV: the change of the background tells us when a new story begins. A glance at the clock and we can clearly see what time it is. These are special spatial-temporal episodes caused by ballistic eye movements and sudden optical changes. In this paper we give a useful definition for generalized sudden global change events, present its main properties and give an algorithm to recognize them in any video-flow. The proposed algorithm is implemented on a standard Cellular Nonlinear Network Universal Machine (CNN-UM). The processing time of the detection is roughly one millisecond on the ACE4k CNN-UM chip in the Aladdin environment. Therefore it can serve as a common function in any online, real-time spatial-temporal algorithm. The sudden global change event can be used to define the instant when the time-dependent part of that algorithm, e.g., homotopy, has to be initialized. The first steps of the multimedia processing, such as searching for key frames — for compression, browsing or indexing — can be helped by the proposed algorithm as a robust video shot change detection method.

On the preview of digital movies

In this paper, a new technique is proposed for the automatic generation of a preview sequence of a feature film. The input video is decomposed into a number of basic components called shots. In this step, the proposed shot change detection algorithm is able to detect both the abrupt and gradual transition boundary. Then, shots are grouped into semantic-related scenes by taking into account the visual characteristics and temporal dynamics of video. Finally, by making use of an empirically motivated approach, the intense-interaction and action scenes are extracted to form the abstracting video. Compared with related works which integrate visual and audio information, our visual-based approach is computationally simple yet effective.

A histogram-based moment-preserving clustering algorithm for video segmentation

Video segmentation is the first step of creating video indices for a video retrieval system. A segmentation algorithm is used to identify shots from video data. In this paper, we propose a histogram-based moment-preserving (HBMP) clustering algorithm for segmenting video data. This algorithm is a hybrid of the shot change detection approach and the clustering approach. The computational results indicate that the proposed algorithm is both effective and efficient with respect to various types of video sequence.

Basic principles of anamorphic composition

A note on critical methodology A t one and the same time the motion picture image is both a flat, two-dimensional pattern of light and dark and an illusory three-dimensional reproduction of the scene that appeared in front of the camera. Critics and audiences almost never notice the two-dimensional pattern, responding instead to the stories that unfold in the illusory three-dimensional world. It takes an unusually assertive image something like the bone tossed into the air in prehistoric times in 2001 that with a cut instantaneously becomes a similarly shaped spaceship in flight to make viewers notice a graphic element as such while a story takes place. When I first began to view anamorphic films to see whether there were any compositional strategies filmmakers might have used in the 1950s and 1960s to organise the wider image, I expected that whatever I might discover would be embodied in the apparent three-dimensional space of the image. Yet as I studied films from those years, I found myself unexpectedly attending to the two-dimensional qualities of their images, especially at moments of shot change and camera repositioning. Increasingly, continuities of line and shape came to define composition in ways I had never expected. Eventually, I came to understand that these two-dimensional qualities served as formal keys to the disposition of three-dimensional elements, And in understanding this, I realised that traditional methods of cinematic analysis were no longer appropriate. Not surprisingly, once I concentrated on the two-dimensional qualities of the image as graphic design, I discovered that the language and assumptions of typographers and graphic designers were much more relevant to understanding anamorphic composition than I ever expected. Most useful of all has been the concept of the grid as a rational and objective regulative system. In books and magazines the grid insures a uniformity of lay-out from page to page. The quartered grid in anamorphic filmmaking provides a similar uniformity of composition from shot to shot. The two dimensional pattern of the grid that underpins the placement text and image on the printed page functions equally well to underpin the placement of a film's apparently three-dimensional elements. Josef Muller-Brockmann's suggestion in Grid Systems in Graphic Design, the standard work on the subject that 'as a controlling system the grid makes it easier to give the surface of space a rational organisation', applies equally well to the motion picture image as to the printed page.2 I explain the experiences that brought me to the ideas discussed in this essay in order to alert readers that what they will encounter here has little to do with traditional ways of discussing cinematic composition. Instead of discussing the placement of camera and actors as a directorial decision, for example, I shall define their placement according to a grid uniformly designed into a film's sets. As one might expect from my unusual approach, my discussion is not based upon written documents. Rather, its conclusions are drawn from the careful examination of more than one hundred anamorphic films released between 1953 and 1965. Those films released before 1960 were filmed primarily in the CinemaScope anamorphic process; the

A mask matching approach for video segmentation on compressed data

Video segmentation provides an easy and efficient way for video retrieval and browsing. A frame is detected as a shot change frame if its content is very different from its previous frames. The process of segmenting videos into shots is usually time consuming due to the large number of frames in the videos. In this paper, we propose a new approach for segmenting videos into shots on MPEG coded video data. This approach detects shot changes by computing the shot change probability for each frame. The MPEG coded video data are only partially decoded such that the time for decoding and processing video data frame by frame and pixel by pixel can be avoided. A set of masks for different types of MPEG coded frames (I, P, and B frames) is defined for the computation of the shot change probability.Experiments based on various parameters are performed to show a 95% of detection rate in average. With further consideration on detecting the dissolve effect, the result is improved to reach an average 98% recall and 96% precision of the detection. A video indexing tool based on this approach was implemented. The results of detected shot changes are kept such that video retrieval and browsing can be provided.

Video segmentation using a histogram-based fuzzy c-means clustering algorithm

The purpose of video segmentation is to segment video sequence into shots where each shot represents a sequence of frames having the same contents, and then select key frames from each shot for indexing. Existing video segmentation methods can be classified into two groups: the shot change detection (SCD) approach for which thresholds have to be pre-assigned, and the clustering approach for which a prior knowledge of the number of clusters is required. In this paper, we propose a video segmentation method using a histogram-based fuzzy c-means (HBFCM) clustering algorithm. This algorithm is a hybrid of the two approaches aforementioned, and is designed to overcome the drawbacks of both approaches. The HBFCM clustering algorithm is composed of three phases: the feature extraction phase, the clustering phase, and the key-frame selection phase. In the first phase, differences between color histogram are extracted as features. In the second phase, the fuzzy c-means (FCM) is used to group features into three clusters: the shot change (SC) cluster, the suspected shot change (SSC) cluster, and the no shot change (NSC) cluster. In the last phase, shot change frames are identified from the SC and the SSC, and then used to segment video sequences into shots. Finally, key frames are selected from each shot. Simulation results indicate that the HBFCM clustering algorithm is robust and applicable to various types of video sequences.

Automatic Shot Change Detection Algorithm Using Multi-stage Clustering for MPEG-Compressed Videos

Automatic shot change detection has been recognized as an important research issue for video classification. This paper proposes an automatic clustering-based algorithm for shot change detection in MPEG-compressed videos with a small number of user-defined parameters. For accurate detection of abrupt and gradual shot changes, the proper selection and extraction of features are important. We first propose a fast edge image extraction scheme in the DCT domain on the basis of AC prediction. Then, by using the features extracted from the edge images and DC images, a two-stage clustering-based algorithm is proposed for shot change detection. In the first stage, the algorithm detects abrupt shot changes by employing two-means clustering on the 2-D feature space of histogram and pixel differences between two neighboring DC frames. In the next stage, it subsequently explores gradual shot changes between two adjacent abrupt shot changes by performing a two-step clustering scheme, which uses multiple features such as an edge energy diagram and several frame difference measures. Simulation results show that the proposed algorithm is fast and accurate.

Shot Change Detection Using Scene-Based Constraint

A key step for managing a large video database is to partition the video sequences into shots. Past approaches to this problem tend to confuse gradual shot changes with changes caused by smooth cam...

Visual Rhythm and Shot Verification

Typical result of an automatic shot change detection algorithm expectedly includes a certain number of undetected shots as well as falsely detected shots. Even though automatic shot change detection algorithms are continuing to improve, the ultimate goal of automatically detecting all shot changes without false alarms may never be achieved. Thus, allowing a human operator to intervene—to review and verify the result of a shot change detection algorithm, to delete falsely detected shots as well as to find undetected shots—may be the most viable approach currently available for increasing the accuracy of the overall shot detection process. For this exact purpose, we propose a shot verifier based on the visual rhythm. The visual rhythm, an abstraction of the video, is a single image, a sub-sampled version of a full video in which the sampling is performed in a pre-determined and in a systematic fashion. It is a representation of the video, which includes the overall content of the video. But most importantly, the visual rhythm contains patterns or visual features that allow the viewer/operator to distinguish and classify many different types of video effects (edits and otherwise): cuts, wipes, dissolves, fades, camera motions, object motions, flashlights, zooms, etc. The different video effects manifest themselves as different patterns on the visual rhythm. Using the visual rhythm, it becomes possible, without sequentially playing the entire video, to find false positive shots as well as undetected shots. Thus, inclusion of the visual rhythm in the shot verification process will aid the operator to verify detected shots as well as to find undetected shots fast and efficiently. Our newly developed shot verifier based on the visual rhythm has been designed for operator efficiency. The design of our shot verifier presented and the usefulness of the visual rhythm during the shot verification process will be demonstrated.

Scene-Based Shot Change Detection and Comparative Evaluation

A key step for managing a large video database is to partition the video sequences into shots. Past approaches to this problem tend to confuse gradual shot changes with changes caused by smooth camera motions. This is in part due to the fact that camera motion has not been dealt with in a more fundamental way. We propose an approach that is based on a physical constraint used in optical flow analysis, namely, the total brightness of a scene point across two frames should remain constant if the change across two frames is a result of smooth camera motion. Since the brightness constraint would be violated across a shot change, the detection can be based on detecting the violation of this constraint. It is robust because it uses only the qualitative aspect of the brightness constraint—detecting a scene change rather than estimating the scene itself. Moreover, by tapping on the significant know-how in using this constraint, the algorithm's robustness is further enhanced. Experimental results are presented to demonstrate the performance of various algorithms. It is shown that our algorithm is less likely to interpret gradual camera motions as shot changes, resulting in a better precision performance than most other algorithms. However, its performance deteriorates under large camera or object motions. A twin-threshold scheme is proposed to improve its robustness.

Shot change detection from MPEG encoded video data

In matching and browsing of video images, it is important that their shot changes are detected efficiently. The detection of shot changes without decoding image data has actively been studied recently, since this is cost effective. However, in conventional methods, noise (e.g., a flash, or a sudden motion) in an image is often erroneously recognized as shot changes, and these reduce the “precision rate;” in addition, slow shot-changes (order of a unit field) are difficult to detect, and they reduce the “recall rate.” This paper proposes an accurate and fast method of counting shot changes in MPEG coded image data. As an example of application of the method, this paper describes a hybrid MPEG browser which recognizes the contents of the images. © 2000 Scripta Technica, Syst Comp Jpn, 31(2): 40–48, 2000

Shot change detection from partially decoded MPEG data

In this paper, we propose a scene decomposition algorithm from MPEG compressed video data. As preprocessing for scene decomposition, partial reconstruction methods of DC images for P- and B-pictures as well as I-pictures directly from an MPEG bitstream are used. As for detection algorithms, we have exploited several methods for detection of abrupt shot changes, dissolves, and wipe transitions using comparison of DC images between frames and coding information such as motion vectors. A method is also proposed for exclusion of undesired detection such as flashlight scenes in order to enhance shot change detection accuracy. More than 95% decomposition accuracy has been obtained in an experiment using a TV program longer than 1 hour. It is also found that in the proposed algorithm, shot change detection can be performed more than 5 times faster than normal playback speed using a 130MIPS workstation. © 1999 Scripta Technica, Syst Comp Jpn, 30(8): 11–22, 1999

A unified approach to shot change detection and camera motion characterization

This paper describes an original approach to partitioning of a video document into shots. Instead of an interframe similarity measure which is directly intensity based, we exploit image motion information, which is generally more intrinsic to the video structure itself. The proposed scheme aims at detecting all types of transitions between shots using a single technique and the same parameter set, rather than a set of dedicated methods. The proposed shot change detection method is related to the computation, at each time instant, of the dominant image motion represented by a two-dimensional affine model. More precisely, we analyze the temporal evolution of the size of the support associated to the estimated dominant motion. Besides, the computation of the global motion model supplies by-products, such as qualitative camera motion description, which we describe in this paper, and other possible extensions, such as mosaicing and mobile zone detection. Results on videos of various content types are reported and validate the proposed approach.

Efficient automatic analysis of camera work and microsegmentation of video using spatiotemporal images

Abstract The shot has been regarded as a fundamental unit for the application of digital manipulation to a video. Various techniques have been developed to detect automatically shot changes. But a sequence shot can be so long and complex that it has to be further decomposed into smaller units for more flexible and detailed manipulation. A sequence shot can be segmented into shot segments, each of which keeps a homogeneous camera motion. Camera work has important significance that reflect the intention of video producers. Camera work analysis and segmentation of a sequence shot into shot segments can help in choosing a representative image for a shot. Following concepts introduced by Tonomura et al. (1993), we propose an efficient method for the automatic detection of camera work changes using spatiotemporal images called X-ray images. We introduce various steps in the spatiotemporal image analysis process which significantly improves its robustness and decreases its computational complexity.

The automatic real-time analysis of film editing and transition effects and its applications

We describe an adaptive method for the automatic analysis of film editing. Unlike previously proposed methods, which were restricted to the detection of cut transitions between sequence shots, our method achieves detection of shot change even when done through smooth transition effects (dissolves, wipes), and identifies classes of transition special effects. This method can be applied in real-time on a flow of digital or analog motion picture data of any resolution. We describe experimental implementations of the method. Applications of such a tool are the temporal analysis of motion picture data to be stored in large databases and the automatic choice of entry points in compressed motion picture. The fact that the analysis can be done in real-time allows for its use at digitizing time, or on the output of a video production site. Temporal analysis and description of motion picture is presently a major time and money bottleneck in the establishment of motion picture databases, and we argue that our method can significantly improve the situation in this field.