Shot Boundary Research Articles

Two Stage Shot Boundary Detection via Feature Fusion and Spatial-Temporal Convolutional Neural Networks

Shot boundary detection is essentially to detect the position of frames where the shot changes. It has been actively studied in video analysis and management for convenience, which becomes a key technique with the rapid proliferation of rich and diverse videos. With respect to the complex characteristics of different shots in varying length and content variation property, in this paper we present a two stage method for shot boundary detection (TSSBD) which distinguishes abrupt shot by fusing color histogram and deep features, and locate gradual shot changes with C3D-based deep analysis. Abrupt shot changes are detected firstly as it occurs between two frames, which divides the complete video into segments containing gradual transitions; Over these video segments, gradual shot change detection is implemented using 3D-convolutional neural network, which classifies clips into specific gradual shot change types; Finally, an effective merging strategy is proposed to locate positions of gradual shot transitions. The experimental analysis illustrates that the proposed progressive method is capable of detecting both abrupt shot transitions and gradual shot transitions accurately.

Enhanced Local Directional Pattern based video shot boundary detection and automatic synchronization for STB quality inspection

Enhanced Local Directional Pattern based video shot boundary detection and automatic synchronization for STB quality inspection

ERP/MMR Algorithm for Classifying Topic‐Relevant and Topic‐Irrelevant Visual Shots of Documentary Videos

We propose and evaluate a video summarization method based on a topic relevance model, a maximal marginal relevance (MMR), and discriminant analysis to generate a semantically meaningful video skim. The topic relevance model uses event‐related potential (ERP) components to describe the process of topic relevance judgment. More specifically, the topic relevance model indicates that N400 and P600, which have been successfully applied to the mismatch process of a stimulus and the discourse‐internal reorganization and integration process of a stimulus, respectively, are used for the topic mismatch process of a topic‐irrelevant video shot and the topic formation process of a topic‐relevant video shot. To evaluate our proposed ERP/MMR‐based method, we compared the video skims generated by the ERP/MMR‐based, ERP‐based, and shot boundary detection (SBD) methods with ground truth skims. The results showed that at a significance level of 0.05, the ROUGE‐1 scores of the ERP/MMR method are statistically higher than those of the SBD method, and the diversity scores of the ERP/MMR method are statistically higher than those of the ERP method. This study suggested that the proposed method may be applied to the construction of a video skim without operational intervention, such as the insertion of a black screen between video shots.

A semantic-based video scene segmentation using a deep neural network

Video scene segmentation is very important research in the field of computer vision, because it helps in efficient storage, indexing and retrieval of videos. Achieving this kind of scene segmentation cannot be done by just calculating the similarity of low-level features presented in the video; high-level features should also be considered to achieve a better performance. Even though much research has been conducted on video scene segmentation, most of these studies failed to semantically segment a video into scenes. Thus, in this study, we propose a Deep-learning Semantic-based Scene-segmentation model (called DeepSSS) that considers image captioning to segment a video into scenes semantically. First, the DeepSSS performs shot boundary detection by comparing colour histograms and then employs maximum-entropy-applied keyframe extraction. Second, for semantic analysis, using image captioning that benefits from deep learning generates a semantic text description of the keyframes. Finally, by comparing and analysing the generated texts, it assembles the keyframes into a scene grouped under a semantic narrative. That said, DeepSSS considers both low- and high-level features of videos to achieve a more meaningful scene segmentation. By applying DeepSSS to data sets from MS COCO for caption generation and evaluating its semantic scene-segmentation task results with the data sets from TRECVid 2016, we demonstrate quantitatively that DeepSSS outperforms other existing scene-segmentation methods using shot boundary detection and keyframes. What’s more, the experiments were done by comparing scenes segmented by humans and scene segmented by the DeepSSS. The results verified that the DeepSSS’ segmentation resembled that of humans. This is a new kind of result that was enabled by semantic analysis, which was impossible by just using low-level features of videos.

A vague set approach for identifying shot transition in videos using multiple feature amalgamation

Shot boundary detection (SBD) is the preliminary and most significant step in Content Based Video Retrieval (CBVR). As such the effectiveness of a CBVR system depends heavily on reliable detection of shot boundaries. In this work, a simple yet effective technique for amalgamating several distance features extracted from video frames has been proposed. The aim here is to develop a technique which is able to produce a better distance feature from the existing ones by hybridizing several distance metrics. In the proposed model, any number of distance features can be incorporated and fused together. The resultant feature is not only more robust but also immune to features which are inefficient. Robustness of the proposed method is tested by combining several low performing features with the more efficient ones. Several statistical amalgamation functions are also tested for determining the most efficient one in terms of F1 score. The power of vague sets has been harnessed to detect the shot boundaries effectively using the resultant distance feature. The proposed method is proved to be effective by means of the results obtained, which show that multiple feature amalgamation can lead to a hybrid distance feature which performs better than the best feature incorporated for SBD. The proposed technique is analyzed using ANOVA. A comparison with the other existing methods portray the efficacy of the proposed approach. This method can also be applied for other research problems where several features are to be fused together for producing superior results than the ones obtained by individual methods.

Curvelet transform based feature extraction and selection for multimedia event classification

Multimedia event classification has been one of the major endeavors in video event analysis. For event identification, feature extraction plays a critical role, merely distinguishing the right features becomes a challenging job. So, in this paper, feature extraction and selection for video event detection are proposed. For video event detection or classification, identification of object structure and its motion are a basic needs. So, for object detection, curvelet features are considered, due to its high directional selectivity and high anisotropic properties. Next, an algorithm for Shot Boundary Detection (SBD) called Motion based SBD (MSBD) is proposed to identify the shot boundaries. Also, to make the users to access the queried event in less time, an algorithm for representing few representative frames containing whole video content is proposed. To make an event search efficient, object-based features from dominant features are extracted and to enhance the features much more efficient and dominant, feature selection is performed using the ranking method. Lastly, an SVM classifier with RBF kernel is used for event classification. The proposed work is experimented using Columbia Consumer Video (CCV) dataset and evaluated using mean Average Precision (mAP) and find that, it outperforms various other existing methods.

Generative Model Based Video Shot Boundary Detection for Automated Surveillance

Video shot boundary detection (SBD) or video cut detection is one of the fundamental processes of video-processing with respect to semantic understanding, contextual information accumulation, labeling, content-based information retrieval and many more applications, such as video surveillance and monitoring. In this work, the authors have proposed a generative-model based framework for detecting shot boundaries in between the frames of a video segment. To generate a model of shot-boundaries, the authors have applied the concepts of Support Vector Machine to estimate the distance between any two images, and then, have generated a Gaussian Mixture Model from the estimated distances. Next, a Bayesian Estimation process checks the presence of boundaries in between the images by exploiting the Gaussian Mixture-based boundary model. Further, the authors have used the principles of Compressive Sensing to reduce the overhead of boundary detection process without losing of important information.

Motion and illumination defiant cut detection based on Weber features

The spontaneous proliferation of video data necessitates implementing hierarchical structures for various content management applications. Temporal video segmentation is the key towards such management. To address the problem of temporal segmentation, the current communication exploits the concept of psychological behaviour of the human visual system. Towards this goal an abrupt cut detection scheme has been proposed based on Weber's law which provides a strong spatial correlation among the neighbouring pixels. Thus, the authors provide a robust and unique solution for abrupt shot boundary detection when the frames are affected partially or fully by flashlight, fire and flicker, high motion associated with an object or camera. Further, they have devised a model for generating an automatic threshold, taking into account the statistics of the feature vector which quadrates itself with the variation in the contents of the video. The effectiveness of the proposed framework is validated by exhaustive comparison with few contemporary and recent approaches by using benchmark datasets TRECVID 2001, TRECVID 2002, TRECVID 2007 and some publicly available videos. The results obtained give credence to the remarkable improvement in the performance while preserving a good trade-off between missed hits and false hits as compared to the state-of-the-art methods.

Gradual transition detection in shot boundary using gradual curve point

Video shot boundary detection (SBD) is a fundamental step in automatic video content analysis. Gradual transition is based on the gradual curves characterised by distribution of average edge information in the sequence of frames. An average edge frame is obtained from optimal edge detector. Features are extracted based on the minimum difference between the sequences of frames. The minimum variance in the local frame sequence is chosen as a gradual curve point. Finally, the performance is computed based on the precision (quality) and recall (quantity) value.

Video Searching and Fingerprint Detection by Using the Image Query and PlaceNet-Based Shot Boundary Detection Method

This work presents a novel shot boundary detection (SBD) method based on the Place-centric deep network (PlaceNet), with the aim of using video shots and image queries for video searching (VS) and fingerprint detection. The SBD method has three stages. In the first stage, we employed Local Binary Pattern-Singular Value Decomposition (LBP-SVD) features for candidate shot boundaries selection. In the second stage, we used the PlaceNet to select the shot boundary by semantic labels. In the third stage, we used the Scale-Invariant Feature Transform (SIFT) descriptor to eliminate falsely detected boundaries. The experimental results show that our SBD method is effective on a series of SBD datasets. In addition, video searching experiments are conducted by using one query image instead of video sequences. The results under several image transitions by using shot fingerprints have shown good precision.

Parameter Analysis of the Jensen-Shannon Divergence for Shot Boundary Detection in Streaming Media Applications

Shot boundary detection is an integral part of multimedia, be it video management or video processing. Multiple boundary detection techniques have been developed throughout the years, but are only applicable to very specific instances. The Jensen-Shannon divergence (JSD) is one such a technique that can be implemented to detect the shot boundaries in digital videos. This paper investigates the use of the JSD algorithm to detect shot boundaries in streaming media applications. Furthermore, the effects of the various parameters used by the JSD technique, on the accuracy of the detected boundaries, are quantified by the recall and precision metrics all the while keeping track of how they affect the execution time.

A Comparison of Corner Feature Detectors for Video Abrupt Shot Detection

Comparison of feature detectors and evaluation of their performance is very important in computer vision. A new algorithm is proposed in this paper to compare the performance of four corner feature detectors based on abrupt shot boundary detection. The proposed algorithm consists of two stages: feature vectors generation where corner detector for all video frames is computed to obtain the descriptor feature vectors, and features matching where the number of matching features between two successive frames is calculated. The corner feature detectors used in this paper are BRISK, Harries, MinEigen, and FAST. Experimental results indicate that the proposed algorithm using MinEigen features detector provides better performance than other features detectors where the average value of recall, precision, and F measure is 0.99083, 0.98808, and 0.98875 for selected testing videos respectively. The results also show that the FAST is superior to others feature detectors when considering execution time.

A Novel Solution for Scaling Video Shot Boundary Detection Based on Hadoop

Shot Boundary Detection (SBD) is an important step required by CBVIR systems. In order to perform scalable SBD, a MapReduce based solution is proposed. So, instead of handling consecutive frames in a sequential manner, they can be processed in a fully parallel way. Usually, in the sequential case, descriptors of consecutive frames are compared and shot boundaries are detected if significant variations have occurred. It seems simple, but it can take centuries to processes immense multimedia datasets. Then, based on the transitivity of similarity relation, resemblance measurement between distant frames is calculated, and shout boundaries are extracted respectively by Mapper and Reducer routines. The experiment results show that the proposed solution outperforms the sequential traditional methods and can be applied to a large-scale multimedia datasets.

IMPULSIVE INTERMODAL CYBER BULLYING RECOGNITION FROM PUBLIC NETS

Cyberbullying has grown as an important societal challenge nowadays. The Cyberbullying affects both in terms of psychological and emotional means of a person. So there is a need to devise a method to detect and prevent cyberbullying in social networks. Most of the existing cyberbullying methods involves only text detection and few methods are available for analysing the visual detection. In this proposed work is going to detect multimodel cyberbullying such as audio, video, image along with text in the social networks. The cyberbully image will be detected using the computer vision algorithm which includes two methods like Image Similarity and Optical Character Recognition (OCR). The cyberbully video will be detected using the Shot Boundary detection algorithm where the video will be broken into frames and analysed using various methods in it. The proposed framework also support to identify the cyberbully audio in the social network. Finally the cyberbully data will be classified into Physical bullying, Social bullying and Verbal bullying using classifiers.

Efficient non-local means denoising for image sequences with dimensionality reduction

The aim of this paper is to improve both accuracy and computational efficiency of non-local means video (NLMV) denoising algorithm. A technique of principal component analysis (PCA) is used to reduce the heavy dimensionality of patches. A pre-processing step of shot boundary detection is used to split the video sequence into different shots having content-wise similar frames. Further PCA is computed globally for these shots. To speed-up the denoising process, weights are computed in reduced subspace. In the proposed method, we modify the original histogram difference (HD) technique such that content-wise similar frames are separated more systematically and accurately. We have achieved improvement with respect to accuracy and computational speed compared to standard NLM. Moreover, qualitative and quantitative comparisons show that the proposed method is consistently superior compared to that of NLM and some of its variants.

Detection and classification of wipe transitions in sport videos in presence of object motion

Transition detection is the necessary step in retrieval and investigation of videos on the basis of contents. Since last two decades, most of the researchers are involved in developing algorithms for detection of gradual transitions. However, the features of all gradual transitions are different and hence this issue still needs to be addressed precisely. After identifying this issue, an integrated shot boundary detection method is proposed in this paper. Wipe transition is the effect which is mostly used in the video making industries .Since there are different types of wipe transitions, it becomes very difficult to detect such transitions. Due to complexity in detection of wipes due to noise, object and camera motion earlier methods have less focus on this issue. In this paper an efficient wipe detection method is presented which gives better results even in the presence of object and camera movements.

Effective Video Shot Boundary Detection and Keyframe Selection using Soft Computing Techniques

The amount of video data generated and made publicly available has been tremendously increased in today's digital era. Analyzing these huge video repositories require effective and efficient content-based video analysis systems. Shot boundary detection and Keyframe extraction are the two major tasks in video analysis. In this direction, a method for detecting abrupt shot boundaries and extracting representative keyframe from each video shot is proposed. These objectives are achieved by incorporating the concepts of fuzzy sets and intuitionistic fuzzy sets. Shot boundaries are detected using coefficient of correlation on fuzzified frames. Further, probabilistic entropy measures are computed to extract the keyframe within fuzzified frames of a shot. The keyframe representative of a shot is the frame with highest entropy value. To show the efficacy of the proposed methods two benchmark datasets are used (TRECVID and Open Video Project). The proposed methods outperform when compared with some of state-of-the-art shot boundary detection and keyframe extraction methods.

Methods and Challenges in Shot Boundary Detection: A Review.

The recent increase in the number of videos available in cyberspace is due to the availability of multimedia devices, highly developed communication technologies, and low-cost storage devices. These videos are simply stored in databases through text annotation. Content-based video browsing and retrieval are inefficient due to the method used to store videos in databases. Video databases are large in size and contain voluminous information, and these characteristics emphasize the need for automated video structure analyses. Shot boundary detection (SBD) is considered a substantial process of video browsing and retrieval. SBD aims to detect transition and their boundaries between consecutive shots; hence, shots with rich information are used in the content-based video indexing and retrieval. This paper presents a review of an extensive set for SBD approaches and their development. The advantages and disadvantages of each approach are comprehensively explored. The developed algorithms are discussed, and challenges and recommendations are presented.

Content-based video retrieval in historical collections of the German Broadcasting Archive

The German Broadcasting Archive (DRA) maintains the cultural heritage of radio and television broadcasts of the former German Democratic Republic (GDR). The uniqueness and importance of the video material stimulates a large scientific interest in the video content. In this paper, we present an automatic video analysis and retrieval system for searching in historical collections of GDR television recordings. It consists of video analysis algorithms for shot boundary detection, concept classification, person recognition, text recognition and similarity search. The performance of the system is evaluated from a technical and an archival perspective on 2,500 h of GDR television recordings.

Automatic Detection of Video Shot Boundary in Social Media Using a Hybrid Approach of HLFPN and Keypoint Matching

Shot boundary detection (SBD) is an important and fundamental step in video content analysis such as content-based video indexing, browsing, and retrieval. In this paper, a hybrid SBD method is presented by integrating a high-level fuzzy Petri net (HLFPN) model with keypoint matching. The HLFPN model with histogram difference is executed as a predetection. Next, the speeded-up robust features (SURF) algorithm that is reliably robust to image affine transformation and illumination variation is used to figure out all possible false shots and the gradual transition based on the assumption from the HLFPN model. The top-down design can effectively lower down the computational complexity of SURF algorithm. The proposed approach has increased the precision of SBD and can be applied in different types of videos.