Shot Change Detection Research Articles

Exploring Relations in Untrimmed Videos for Self-Supervised Learning

Existing video self-supervised learning methods mainly rely on trimmed videos for model training. They apply their methods and verify the effectiveness on trimmed video datasets including UCF101 and Kinetics-400, among others. However, trimmed datasets are manually annotated from untrimmed videos. In this sense, these methods are not truly unsupervised. In this article, we propose a novel self-supervised method, referred to as Exploring Relations in Untrimmed Videos (ERUV), which can be straightforwardly applied to untrimmed videos (real unlabeled) to learn spatio-temporal features. ERUV first generates single-shot videos by shot change detection. After that, some designed sampling strategies are used to model relations for video clips. The strategies are saved as our self-supervision signals. Finally, the network learns representations by predicting the category of relations between the video clips. ERUV is able to compare the differences and similarities of video clips, which is also an essential procedure for video-related tasks. We validate our learned models with action recognition, video retrieval, and action similarity labeling tasks with four kinds of 3D convolutional neural networks. Experimental results show that ERUV is able to learn richer representations with untrimmed videos, and it outperforms state-of-the-art self-supervised methods with significant margins.

Neuro-mathematical fusion for shot change detection in video sequences

Neuro-mathematical fusion for shot change detection in video sequences

Two Stage Shot Boundary Detection via Feature Fusion and Spatial-Temporal Convolutional Neural Networks

Shot boundary detection is essentially to detect the position of frames where the shot changes. It has been actively studied in video analysis and management for convenience, which becomes a key technique with the rapid proliferation of rich and diverse videos. With respect to the complex characteristics of different shots in varying length and content variation property, in this paper we present a two stage method for shot boundary detection (TSSBD) which distinguishes abrupt shot by fusing color histogram and deep features, and locate gradual shot changes with C3D-based deep analysis. Abrupt shot changes are detected firstly as it occurs between two frames, which divides the complete video into segments containing gradual transitions; Over these video segments, gradual shot change detection is implemented using 3D-convolutional neural network, which classifies clips into specific gradual shot change types; Finally, an effective merging strategy is proposed to locate positions of gradual shot transitions. The experimental analysis illustrates that the proposed progressive method is capable of detecting both abrupt shot transitions and gradual shot transitions accurately.

Shot Change and Stuck Pixel Detection of Digital Video Assets

In this paper, we address two significant components of typical post-production workflows in current entertainment industry: (a) shot change detection and (b) dead- or stuck-pixel detection. The former involves identifying precise temporal boundaries of cinematographic content in order to analyze quality of cuts, while the latter is useful for localizing and correcting spatial anomalies that emerge during the image acquisition step due to faulty sensors. To address the aforementioned, we devise two novel data-driven approaches and demonstrate their portability, scalability, and efficacy through two respective cloud computing-based implementations. Our approaches show promising results on our in-house data set of a large number of movie titles indicating their effectiveness.

Video Shot Boundary Detection using the Scale Invariant Feature Transform and RGB Color Channels

<p>Segmentation of the video sequence by detecting shot changes is essential for video analysis, indexing and retrieval. In this context, a shot boundary detection algorithm is proposed in this paper based on the scale invariant feature transform (SIFT). The first step of our method consists on a top down search scheme to detect the locations of transitions by comparing the ratio of matched features extracted via SIFT for every RGB channel of video frames. The overview step provides the locations of boundaries. Secondly, a moving average calculation is performed to determine the type of transition. The proposed method can be used for detecting gradual transitions and abrupt changes without requiring any training of the video content in advance. Experiments have been conducted on a multi type video database and show that this algorithm achieves well performances.</p>

CUDA based parallel design of a shot change detection algorithm using frame segmentation and object movement

This paper proposes the parallel design of a shot change detection algorithm using frame segmentation and moving blocks. In the proposed approach, the high parallel processing components, such as frame histogram calculation, block histogram calculation, Otsu threshold setting function, frame moving operation, and block histogram comparison, are designed in parallel for NVIDIA GPU. In order to minimize memory access delay time and guarantee fast computation, the output of a GPU kernel becomes the input data of another kernel in a pipeline way using the shared memory of GPU. In addition, the optimal sizes of CUDA processing blocks and threads are estimated through the prior experiments. In the experimental test of the proposed shot change detection algorithm, the detection rate of the GPU based parallel algorithm is the same as that of the CPU based algorithm, but the average of processing time speeds up about 6~8 times.

프레임 블록화와 객체의 이동을 이용한 샷 전환 탐지 알고리즘

This paper proposes a shot change detection algorithm by using frame segmentation and the object changes among moving blocks. In order to detect the rapid moving changes of objects between two consecutive frames, the moving blocks on the diagonal are defined, and their histograms are calculated. When a block of the current frame is compared to the moving blocks of the next frame, the block histograms are used and the threshold of a shot change detection is automatically adjusted by Otsu`s threshold method. The proposed algorithm was tested for the various types of color or gray videos such as films, dramas, animations, and video tapes in National Archives of Korea. The experimental results showed that the proposed algorithm could enhance the detection rate when compared to the studied methods that use brightness, histogram, or segmentation.

Motion Detection in Compressed Video Using Macroblock Classification

In this paper, to detect the moving objects between frames in compressed video and to obtain the best compression video and the noiseless video. We describe a video in which frames by classifying macroblocks (MB), and describe motion estimation (ME), motion vector field (MV) and motion compensation (MC). we propose to classify Macroblocks of each video frame into different classes and use this class information to describe the frame content based on the motion vector. MB class information video applications such as shot change detection, motion discontinuity detection, Outlier rejection for global motion estimation. To reduce the noise and to improve the clarity of the compressed video by using contrast limited adaptive histogram equalization (CLAHE) Algorithm.

Color Mismatch Compensation Method Based on a Physical Model

A new method for detecting and correcting color-timing mismatch in digital film is proposed. This method is based on a physical model which accounts for the absorption of light by the different layers of the film. The whole process consists of five sequential stages: shot change detection, registration, degraded zone detection, and unreliable motion detection and correction. One of the main issues in this paper is how to align the degraded and reference frames in the case of varying illumination conditions. A new transformation, based on the proposed physical model, is derived to make the data term of the optical flow framework robust to color change. The degraded regions are then detected and restored by using the computed flow field. The performance of the proposed method is evaluated through extensive tests on both simulated and real high definition films. The obtained results are very promising and confirm the efficiency of the proposed method.

A Novel Approach for Tracking with Implicit Video Shot Detection

Video shot detection - Shot change detection is an essential step in video content analysis. The field of Video Shot Detection (VSD) is a well exploited area. In the past, there have been numerous approaches designed to successfully detect shot boundaries for temporal segmentation. Robust Pixel Based Method is used to detect shot changes in a video sequence. Tracking algorithm is a time consuming process due to the large amount of data contained in video using the video shot detection the computational cost can be reduced to a great extent by the discarding the frames which are not of any interest for the tracking algorithm. In this paper we present a novel approach of combining the concepts of Video shot detection and Object tracking using particle filter to give us a efficient Tracking algorithm with implicit shot detection. Keywords - Bhattacharyya distance, Local adaptive threshold, Particle filter, Robust Pixel Difference method, Residual Re-sampling, Shot detection. I. INTRODUCTION The rapid development of storage and multimedia technologies has made the retrieval and processing of videos relatively easy. Temporal segmentation is a fundamental step in video processing, and shot change detection is the most basic way to achieve it. However, while hard cuts (abrupt transitions) can be easily detected by finding changes in a color histogram, gradual transitions such as dissolves, fades, and wipes are hard to locate. In practice, however, 99% of all edits fall into one of the following four categories hard cuts, fades, wipes and dissolves. Many shot change detection studies focus on finding low-level visual features, e.g., color histograms and edges, and then locate the spots of changes in those features. We focus on using Robust Pixel Method for shot detection. The conventional shot detection method using pixel wise comparison is not very efficient since it doesn't provide noise tolerance and because of its global thresholding nature. Many scenes involving sudden illumination changes such as lighting etc false trigger a shot change in the conventional method. The robust pixel method used in this paper provides threshold for noise and also locally adaptive threshold which makes it effective in situations where the conventional method fails. Object tracking is an important task in the field of computer vision. It generates the path traced by a specified object by locating its position in each frame of the video sequence. The use of Object tracking is pertinent in many vision applications such as automated surveillance, video indexing, vehicle navigation, motion based recognition, security and defence areas. Occlusion and noise are generally the biggest problems in any target tracking implementation. Tracking algorithms robustness is a measure of how well it continues to track and when it loses its target. Tracking is the observation of person(s) or object(s) on the move and supplying a timely ordered sequence of respective location data to a model under consideration. It is the process of locating a moving human or object over time using a camera. It is based on computer vision. Image registration is the basic step used in tracking application. It is a process that finds the location where a good matching is obtained by matching the template over the searching area of an input image. Registration algorithm fails in complex situations and loses the target in presence of noise, scaling and transformation changes. To address the above mentioned problems, we use a particle filter based tracking method for efficient tracking. Video shot detection and tracking algorithms have both been extensively researched and have been used in real world applications individually. Very less effort has been made to combine the concepts of video shot detection and tracking which can be of great help in real-world as both the technologies complement each other. Combining the two concepts guarantees a computationally quicker, cost effective solution for tracking on large video database with minimal pre-processing. In this paper Section II covers the concepts of Video shot detection using robust pixel difference method and also demonstrates it effectiveness with results. Section III focuses on concepts of tracking algorithm with results. Section IV elaborates the method of combining the two approaches where tracking algorithm is initiated after every shot change hence serving its final purpose of computationally efficient shot detection cum tracking system.

Detection of Video Shot Changes Based on Image Matching

In order to improve the supervision of TV commercials and reduce the illegal ones, it is necessary to develop a system to detect TV commercials in real time. One of the key points is to detect video shot changes. We put forward a novel method of image matching for shot change detection: firstly, a feature coding is introduced for describing image local gray level distribution; secondly, image matching is performed according to the feature coding; thirdly, shot changes can be detected. A preliminary system of video shot change detection is developed. Experiments show that the algorithm is better than traditional algorithms and can well meet the need of TV Video shot cut detection.

A stereoscopic video conversion scheme based on spatio-temporal analysis of MPEG videos

Abstract In this article, an automatic stereoscopic video conversion scheme which accepts MPEG-encoded videos as input is proposed. Our scheme is depth-based, relying on spatio-temporal analysis of the decoded video data to yield depth perception cues, such as temporal motion and spatial contrast, which reflect the relative depths between the foreground and the background areas. Our scheme is shot-adaptive, demanding that shot change detection and shot classification be performed for tuning of algorithm or parameters that are used for depth cue combination. The above-mentioned depth estimation is initially block-based, followed by a locally adaptive joint trilateral upsampling algorithm to reduce the computing load significantly. A recursive temporal filter is used to reduce the possible depth fluctuations (and also artifacts in the synthesized images) resulting from wrong depth estimations. The traditional Depth-Image-Based-Rendering algorithm is used to synthesize the left- and right-view frames for 3D display. Subjective tests show that videos converted by our scheme provide comparable perceived depth and visual quality with those converted from the depth data calculated by stereo vision techniques. Also, our scheme is shown to outperform the well-known TriDef software in terms of human’s perceived 3D depth. Based on the implementation by using “OpenMP” parallel programming model, our scheme is capable of executing in real-time on a multi-core CPU platform.

Macroblock Classification Method for Video Applications Involving Motions

In this paper, a macroblock classification method is proposed for various video processing applications involving motions. Based on the analysis of the Motion Vector field in the compressed video, we propose to classify Macroblocks of each video frame into different classes and use this class information to describe the frame content. We demonstrate that this low-computation-complexity method can efficiently catch the characteristics of the frame. Based on the proposed macroblock classification, we further propose algorithms for different video processing applications, including shot change detection, motion discontinuity detection, and outlier rejection for global motion estimation. Experimental results demonstrate that the methods based on the proposed approach can work effectively on these applications.

A Martingale Framework for Detecting Changes in Data Streams by Testing Exchangeability

In a data streaming setting, data points are observed sequentially. The data generating model may change as the data are streaming. In this paper, we propose detecting this change in data streams by testing the exchangeability property of the observed data. Our martingale approach is an efficient, nonparametric, one-pass algorithm that is effective on the classification, cluster, and regression data generating models. Experimental results show the feasibility and effectiveness of the martingale methodology in detecting changes in the data generating model for time-varying data streams. Moreover, we also show that: 1) An adaptive support vector machine (SVM) utilizing the martingale methodology compares favorably against an adaptive SVM utilizing a sliding window, and 2) a multiple martingale video-shot change detector compares favorably against standard shot-change detection algorithms.

A template-based baseball video scene classification using efficient playfield segmentation

In this paper, we present an effective and efficient framework for baseball video scene classification. The results of scene classification can be able to provide the ground for baseball video abstraction and high-level event extraction. In general, most conventional approaches are shot-based, which shot change detection and key-frame extraction are necessary prerequisite procedures. On the contrary, we propose a frame-based approach. In our scene classification framework, an efficient playfield segmentation technique is proposed, and then the reduced field maps are utilized as scene templates. Because the shot change detection and the key-frame extraction are not required in proposed method, the new framework is very simple and efficient. The experimental results have demonstrated that the effectiveness of our proposed framework for baseball videos scene classification, and it can be easily extended the template-based approach to other kinds of sports videos.

An adaptive shot change detection algorithm and its implementation on portable multimedia player

Shot change detection is a main technique for automatic temporal segmentation of video, which requires adaptive adjustment of thresholds and real-time operation in actual applications. To date, few researches have been conducted on real-time shot change detection to be applied to hardware terminals with low performance such as PMPs (portable media player) and cellular phones. In this paper, we propose an adaptive real-time shot change detection algorithm and implement it on an actual PMP. The proposed algorithm consists of the following elements: sub-sampling, weighting variance, and adaptive thresholds. Our experiments obtained the detection rate of about 94.4% in overall accuracy, demonstrating a higher detection rate and a computational amount 1/256 less than the conventional algorithms which have optimal fixed thresholds adjusted by humans for each video. We verified the real-time operation of shot change detection by implementing our algorithm on the PMP of Homecast, a PMP company. The proposed algorithm will be helpful in searching video data on portable media players such as PMPs and cellular phones.

Wipe scene-change detector based on visual rhythm spectrum

To store and retrieve large-scale video data sets effectively, the process of shot-change detection is an essential step. In this paper, we propose an automatic shot-change detection algorithm based on Visual Rhythm Spectrum. The Visual Rhythm Spectrum contains distinctive patterns or visual features for many different types of video effects. For the improvement of detection speed, the proposed algorithm is executed by using the partial data of digital compressed video. The proposed detection algorithm can be universally applied to various kinds of shot-change categories such as scene-cuts and wipes. The developed wipe detector is implemented and tested with real video sequences containing a variety of wipe types and lengths. It is shown by simulations that the proposed detection algorithm outperforms other existing approaches.

Video Classification and Shot Detection for Video Retrieval Applications

Appropriate organization of video databases is essential for pertinent indexing and retrieval of visual information. This paper proposes a new feature called Block Intensity Comparison Code (BICC) for video classification and an unsupervised shot change detection algorithm to detect the shot changes in a video stream using autoassociative neural network (AANN) which makes retrieval problems much simpler. BICC represents the average block intensity difference between blocks of a frame. A novel AANN misclustering rate (AMR) algorithm is used to detect the shot transitions. The experiments demonstrate the effectiveness of the proposed methods.

激しい動きやカメラワークを含む映像における動き推定に基づくショット切り替え検出法(ITS画像処理,映像メディア及び一般)

激しい動きやカメラワークを含む映像における動き推定に基づくショット切り替え検出法(ITS画像処理,映像メディア及び一般)

Video Classification and Shot Detection for Video Retrieval Applications

Appropriate organization of video databases is essential for pertinent indexing and retrieval of visual information. This paper proposes a new feature called Block Intensity Comparison Code (BICC) for video classification and an unsupervised shot change detection algorithm to detect the shot changes in a video stream using autoassociative neural network (AANN) which makes retrieval problems much simpler. BICC represents the average block intensity difference between blocks of a frame. A novel AANN misclustering rate (AMR) algorithm is used to detect the shot transitions. The experiments demonstrate the effectiveness of the proposed methods.