Content-based Video Indexing Research Articles

Video shot boundary detection based on frames objects comparison and scale-invariant feature transform technique

The most popular source of data on the Internet is video which has a lot of information. Automating the administration, indexing, and retrieval of movies is the goal of video structure analysis, which uses content-based video indexing and retrieval. Video analysis requires the ability to recognize shot changes since video shot boundary recognition is a preliminary stage in the indexing, browsing, and retrieval of video material. A method for shot boundary detection (SBD) is suggested in this situation. This work proposes a shot boundary detection system with three stages. In the first stage, multiple images are read in temporal sequence and transformed into grayscale images. Based on correlation value comparison, the number of redundant frames in the same shots is decreased, from this point on, the amount of time and computational complexity is reduced. Then, in the second stage, a candidate transition is identified by comparing the objects of successive frames and analyzing the differences between the objects using the standard deviation metric. In the last stage, the cut transition is decided upon by matching key points using a scale-invariant feature transform (SIFT). The proposed system achieved an accuracy of 0.97 according to the F-score while minimizing time consumption.

RECOGNITION OF CHARACTER FROM VIDEO SUBTITLES

An important task in content based video indexing is to extract text information from videos. The challenges involved in text extraction and recognition are variation of illumination on each video frame with text, the text present on the complex background and different font size of the text. Using various image processing algorithms like morphological operations, blob detection and histogram of oriented gradients the character recognition of video subtitles is implemented. Segmentation, feature extraction and classification are the major steps of character recognition. Several experimental results are shown to demonstrate the performance of the proposed algorithm.

Unsupervised Video Segmentation using Quantization

Abstract: Unsupervised video segmentation is a challenging task in computer vision that involves dividing a video into meaningful segments without any labeled data or prior knowledge of the video content. One approach to achieving this is the use of quantization, which involves clustering similar image patches or video frames into discrete groups based on their visual features.Beta-VAE is a type of variational autoencoder (VAE) that is capable of learning disentangled representations of data. It can be applied to video segmentation using quantization, allowing for more effective segmentation and analysis of complex video datasets.The use of Beta-VAE and quantization is essential for several reasons. Firstly, it allows for the automatic analysis of large video datasets without the need for manual annotation, which can be time-consuming and expensive. Secondly, it enables the detection and tracking of objects and events in videos, which has applications in surveillance, robotics, and autonomous driving. Finally, it can be used for content-based video indexing and retrieval, which is crucial for video search and recommendation systems.The advantage of using Beta-VAE in video segmentation is that it can learn disentangled representations of video frames, which separates the data into meaningful factors of variation. This leads to more accurate segmentation, as the model can distinguish between different objects and events in the video, even if they have similar visual features.In conclusion, the use of Beta-VAE and quantization is necessary for advancing the field of computer vision and improving the analysis and understanding of videos. It has numerous applications in various industries and can contribute to the development of more effective and efficient video analysis tools.

Indexed Dataset from YouTube for a Content-Based Video Search Engine

Numerous researches on content-based video indexing and retrieval besides video search engines are tied to a large-scaled video dataset. Unfortunately, reduction in open-sourced datasets resulted in complications for novel approaches exploration. Although, video datasets that index video files located on public video streaming services have other purposes, such as annotation, learning, classification, and other computer vision areas, with little interest in indexing public video links for purpose of searching and retrieval. This paper introduces a novel large-scaled dataset based on YouTube video links to evaluate the proposed content-based video search engine, gathered 1088 videos, that represent more than 65 hours of video, 11,000 video shots, and 66,000 unmarked and marked keyframes, 80 different object names used for marking. Moreover, a state-of-the-art features vector, and combinational-based matching, beneficial to the accuracy, speed, and precision of the video retrieval process. Any video record in the dataset is represented by three features: temporal combination vector, object combination vector with shot annotations, and 6 keyframes, sideways with other metadata. Video classification for the dataset was also imposed to expand the efficiency of retrieval of video-based queries. A two-phased approach has been used based on object and event classification, storing video records in aggregations related to feature vectors extracted. While object aggregation stores video records with the maximal occurrence of extracted object/concept from all shots, event aggregation classify based on groups according to the number of shots per video. This study indexed 58 out of 80 different object/concept categories, each has 9 shot number groups.

A systematic review on content-based video retrieval

Content-based video retrieval and indexing have been associated with intelligent methods in many applications such as education, medicine and agriculture. However, an extensive and replicable review of the recent literature is missing. Moreover, relevant topics that can support video retrieval, such as dimensionality reduction, have not been surveyed. This work designs and conducts a systematic review to find papers able to answer the following research question: “what segmentation, feature extraction, dimensionality reduction and machine learning approaches have been applied for content-based video indexing and retrieval?”. By applying a research protocol proposed by us, 153 papers published from 2011 to 2018 were selected. As a result, it was found that strategies for cut-based segmentation, color-based indexing, k-means based dimensionality reduction and data clustering have been the most frequent choices in recent papers. All the information extracted from these papers can be found in a publicly available spreadsheet. This work also indicates additional findings and future research directions.

Fast Temporal Video Segmentation Based on Krawtchouk-Tchebichef Moments

With the increasing growth of multimedia data, the current real-world video sharing websites are being huge in repository size, more specifically video databases. This growth necessitates to look for superior techniques in processing video because video contains a lot of useful information. Temporal video segmentation (TVS) is considered essential stage in content-based video indexing and retrieval system. TVS aims to detect boundaries between successive video shots. TVS algorithm design is still challenging because most of the recent methods are unable to achieve fast and robust detection. In this regard, this paper proposes a TVS algorithm with high precision and recall values, and low computation cost for detecting different types of video transitions. The proposed algorithm is based on orthogonal moments which are considered as features to detect transitions. To increase the speed of the TVS algorithm as well as the accuracy, fast block processing and embedded orthogonal polynomial algorithms are utilized to extract features. This utilization will lead to extract multiple local features with low computational cost. Support vector machine (SVM) classifier is used to detect transitions. Specifically, the hard transitions are detected by the trained SVM model. The proposed algorithm has been evaluated on four datasets. In addition, the performance of the proposed algorithm is compared to several state-of-the-art TVS algorithms. Experimental results demonstrated that the proposed algorithm performance improvements in terms of recall, precision, and F1-score are within the ranges (1.31 - 2.58), (1.53 - 4.28), and (1.41 - 3.03), respectively. Moreover, the proposed method shows low computation cost which is 2% of real-time.

Methods and Challenges in Shot Boundary Detection: A Review.

The recent increase in the number of videos available in cyberspace is due to the availability of multimedia devices, highly developed communication technologies, and low-cost storage devices. These videos are simply stored in databases through text annotation. Content-based video browsing and retrieval are inefficient due to the method used to store videos in databases. Video databases are large in size and contain voluminous information, and these characteristics emphasize the need for automated video structure analyses. Shot boundary detection (SBD) is considered a substantial process of video browsing and retrieval. SBD aims to detect transition and their boundaries between consecutive shots; hence, shots with rich information are used in the content-based video indexing and retrieval. This paper presents a review of an extensive set for SBD approaches and their development. The advantages and disadvantages of each approach are comprehensively explored. The developed algorithms are discussed, and challenges and recommendations are presented.

Automatic Detection of Video Shot Boundary in Social Media Using a Hybrid Approach of HLFPN and Keypoint Matching

Shot boundary detection (SBD) is an important and fundamental step in video content analysis such as content-based video indexing, browsing, and retrieval. In this paper, a hybrid SBD method is presented by integrating a high-level fuzzy Petri net (HLFPN) model with keypoint matching. The HLFPN model with histogram difference is executed as a predetection. Next, the speeded-up robust features (SURF) algorithm that is reliably robust to image affine transformation and illumination variation is used to figure out all possible false shots and the gradual transition based on the assumption from the HLFPN model. The top-down design can effectively lower down the computational complexity of SURF algorithm. The proposed approach has increased the precision of SBD and can be applied in different types of videos.

Exploiting visual saliency for assessing the impact of car commercials upon viewers

Content based video indexing and retrieval (CBVIR) is a lively area of research which focuses on automating the indexing, retrieval and management of videos. This area has a wide spectrum of promising applications where assessing the impact of audiovisual productions emerges as a particularly interesting and motivating one. In this paper we present a computational model capable to predict the impact (i.e. positive or negative) upon viewers of car advertisements videos by using a set of visual saliency descriptors. Visual saliency provides information about parts of the image perceived as most important, which are instinctively targeted by humans when looking at a picture or watching a video. For this reason we propose to exploit visual information, introducing it as a new feature which reflects high-level semantics objectively, to improve the video impact categorization results. The suggested salience descriptors are inspired by the mechanisms that underlie the attentional abilities of the human visual system and organized into seven distinct families according to different measurements over the identified salient areas in the video frames, namely population, size, location, geometry, orientation, movement and photographic composition. Proposed approach starts by computing saliency maps for all the video frames, where two different visual saliency detection frameworks have been considered and evaluated: the popular graph based visual saliency (GBVS) algorithm, and a state-of-the-art DNN-based approach. Then, frame-level salience descriptors are extracted from these maps. Next, pooled statistics are used to collapse the obtained frame-level values into video-level descriptors. Finally, a Logistic regression classifier is built upon the subset of video-level features resulting from a feature selection stage. Experimental validation, conducted on a publicly available corpus of 138 commercials collected from YouTube, shows that the proposed salience descriptors are indicative of the impact upon viewers and achieve a similar performance when compared to a method purely based on aesthetics. Besides, the combined approach, exploiting both saliency and aesthetics together, ultimately results in better performance than what can be achieved individually. In addition, the seven families of salience descriptors defined are also compared in terms of classification performance. Finally, a similar study is also performed targeting the distinct pooling techniques used in the video-level feature computation.

A Multi-stage Method for Chinese Text Detection in News Videos

With the rapid increase of on-line video resources, there is an urgent demand for text detection and recognition technologies to build content-based video indexing and retrieval systems. Chinese news video texts contain highly condensed and rich information, but the low resolution of videos on the Internet and the complexity of Chinese character structures bring challenges for text detection. In this paper, we present a multi-stage scheme for Chinese news video text detection. We propose an improved Stroke Width Transform (SWT) method by incorporating text color consistency constraint for candidate text blocks generation. Then we use “divide and conquer” strategy to distinguish candidate text blocks into three sub-spaces according to their geometric shapes and size. For each sub-space, a neural network is designed to filter the candidates into text or non-text blocks. Finally, the text blocks are merged into text lines based on the stroke width, color and other heuristic information. Experimental results on self-collected Chinese news video dataset and ICDAR 2013 dataset show that the proposed method is effective to detect both news video captions and scene texts.

A unified approach to content-based indexing and retrieval of digital videos from television archives

This work addresses the development of a unified approach to content-based indexing and retrieval of digital videos fromtelevision archives. The proposed approach has been designed to deal with arbitrary television genres, making it suitablefor various applications. To achieve this goal, the main steps of a content-based video retrieval system are addressed in thiswork, namely: video segmentation, key-frame extraction, content-based video indexing and the video retrieval operation itself.Video segmentation is addressed as a typical TV broadcast structuring problem, which consists in automatically determiningthe boundaries of each broadcasted program (like movies, news, among others) and inter-program (for instance, commercials).Specifically, to segment the videos, Electronic Program Guide (EPG) metadata is combined with the detection of two specialcues, namely, audio cuts (silence) and dark monochrome frames. On the other hand, a color histogram-based approach performskey-frame extraction. Video indexing and retrieval are accomplished by using hashing and k-d tree methods, while visualsignatures containing color, shape and texture information are estimated for the key-frames, by using image and frequencydomain techniques. Experimental results with the dataset of a multimedia information system especially developed for managingtelevision broadcast archives demonstrate that our approach works efficiently, retrieving videos in 0.16 seconds on average andachieving recall, precision and F1 measure values, as high as 0.76, 0.97 and 0.86 respectively.

Research on the Content-Based Video Indexing for Smart Grid

Data has become the fundamental resource by the emerging new services such as cloud computing, internet of things and social network. In the electric power applications, the video data mining plays an important role in the intelligent data analysis. With growth of video data in such an amazing speed, the information retrieval is becoming more and more important. This paper focuses on the analysis of the content-based video retrieval and proposes the design of a uniformed search engine system. The system is oriented to the retrieval of both the unstructured video contents and structured tags, which helps to achieve the integration of the heterogeneity data resources. In this paper, a retrieval framework is discussed and several problems are addressed.

A comprehensive review of significant researches on content based indexing and retrieval of visual information

Developments in multimedia technologies have paved way for the storage of huge collections of video documents on computer systems. It is essential to design tools for content-based access to the documents, so as to allow an efficient exploitation of these collections. Content based analysis provides a flexible and powerfulway to access video data when compared with the other traditional video analysis techniques. The area of content based video indexing and retrieval (CBVIR), focusing on automating the indexing, retrieval and management of video, has attracted extensive research in the last decade. CBVIR is a lively area of research with enduring acknowledgments from several domains. Herein a vital assessment of contemporary researches associated with the content-based indexing and retrieval of visual information. In this paper, we present an extensive review of significant researches on CBVIR. Concise description of content based video analysis along with the techniques associated with the content based video indexing and retrieval is presented.

A Survey on Visual Content-Based Video Indexing and Retrieval

Video indexing and retrieval have a wide spectrum of promising applications, motivating the interest of researchers worldwide. This paper offers a tutorial and an overview of the landscape of general strategies in visual content-based video indexing and retrieval, focusing on methods for video structure analysis, including shot boundary detection, key frame extraction and scene segmentation, extraction of features including static key frame features, object features and motion features, video data mining, video annotation, video retrieval including query interfaces, similarity measure and relevance feedback, and video browsing. Finally, we analyze future research directions.

Content-based video search on peer-to-peer networks

In this paper, we address the efficiency issue of the content-based video indexing over peer-to-peer (P2P) networks. The traditional video index is improved to suit with the P2P computational model. The algorithms to perform the video query based on the content similarity in the P2P environment are proposed. Also, the algorithms to handle the node joining, departure, index entry insertion are proposed. Furthermore, the load balancing approach based on the proposed algorithms is proposed. From the experiment results, our proposed approach outperforms a naive approach, which directly applies the P2P model with full replication, when a number of P2P nodes to be joined, as well as a number of videos to be inserted, is increased. Meanwhile, the efficiency of our approach in terms of the query answering is bounded by linear complexity. Moreover, our proposed load balancing approach is much more efficient than the naive approach in all experiments.

Sequence Multi-Labeling: A Unified Video Annotation Scheme With Spatial and Temporal Context

Automatic video annotation is a challenging yet important problem for content-based video indexing and retrieval. In most existing works, annotation is formulated as a multi-labeling problem over individual shots. However, video is by nature informative in spatial and temporal context of semantic concepts. In this paper, we formulate video annotation as a sequence multi-labeling (SML) problem over a shot sequence. Different from many video annotation paradigms working on individual shots, SML aims to predict a multi-label sequence for consecutive shots in a global optimization manner by incorporating spatial and temporal context into a unified learning framework. A novel discriminative method, called sequence multi-label support vector machine (SVM <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SML</sup> ), is accordingly proposed to infer the multi-label sequence for a given shot sequence. In SVM <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SML</sup> , a joint kernel is employed to model the feature-level and concept-level context relationships (i.e., the dependencies of concepts on the low-level features, spatial and temporal correlations of concepts). A multiple-kernel learning (MKL) algorithm is developed to optimize the kernel weights of the joint kernel as well as the SML score function. To efficiently search the desirable multi-label sequence over the large output space in both training and test phases, we adopt an approximate method to maximize the energy of a binary Markov random field (BMRF). Extensive experiments on TRECVID'05 and TRECVID'07 datasets have shown that our proposed SVM <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SML</sup> gains superior performance over the state-of-the-art.

Particle swarm optimisation based video abstraction

Video abstraction is a basic step for intelligent access to video and multimedia databases which facilitates content-based video indexing, retrieving and browsing. This paper presents a new video abstraction scheme. The proposed method relies on two stages. First, video is divided into short segments. Second, keyframes in each segment are selected using particle swarm optimisation. A group of experiments show that the proposed technique is promising in regards to selecting the most significant keyframes despite a sustainment in overhead processing.

On Evaluating Video Object Segmentation Quality: A Perceptually Driven Objective Metric

The task of extracting objects in video sequences emerges in many applications such as object-based video coding (e.g., MPEG-4) and content-based video indexing and retrieval (e.g., MPEG-7). The MPEG-4 standard provides specifications for the coding of video objects, but does not address the problem of how to extract foreground objects in image sequences. Therefore, for specific applications, evaluating the quality of foreground/background segmentation results is necessary to allow for an appropriate selection of segmentation algorithms and for tuning their parameters for optimal performance. Many segmentation algorithms have been proposed along with a number of evaluation criteria. Nevertheless, formal psychophysical experiments evaluating the quality of different video foreground object segmentation results have not yet been conducted. In this paper, a generic framework for both subjective and objective segmentation quality evaluation is presented. An objective quality assessment method for segmentation evaluation is derived on the basis of perceptual factors through subjective experiments. The performance of the proposed method is shown on different state-of-the-art foreground/background segmentation algorithms and our method is compared to other objective methods which do not include perceptual factors. Moreover, on the basis of subjective results, weighting strategies are introduced into the proposed metric to meet the specificity of different segmentation applications e.g., video compression, video surveillance and mixed reality. Experimental results confirm the efficiency of the proposed approach.

An intelligent strategy for the automatic detection of highlights in tennis video recordings

Professional sport is a vital part of many peoples’ lives and sport is a multi billion industry in its own right. Professional coaches, sporting professionals and sports journalists often use video recordings of sporting events as a tool in their daily work. With the rapid growth of sport video recordings, there is an emerging demand for a sophisticated sports content-based video indexing systems. Video recordings of sporting event broadcasts often last several hours including commercial breaks. It is therefore time-consuming and difficult to locate key segments containing highlights. This study proposes the use of computational intelligence strategies and information from both the audio and video domains to automatically locate sporting highlight on personal video playback devices. Tennis is chosen as the target domain but the strategy should generalize to other sports as well. Experimental results show that both the mean precision and recall rates are higher than 89%, hence, the proposed approach is capable of effectively detecting tennis highlights in sports broadcasts.

Hierarchical Indexing Structure for Efficient Similarity Search in Video Retrieval

With the rapid increase in both centralized video archives and distributed WWW video resources, content-based video retrieval is gaining its importance. To support such applications efficiently, content-based video indexing must be addressed. Typically, each video is represented by a sequence of frames. Due to the high dimensionality of frame representation and the large number of frames, video indexing introduces an additional degree of complexity. In this paper, we address the problem of content-based video indexing and propose an efficient solution, called the ordered VA-file (OVA-file) based on the VA-file. OVA-file is a hierarchical structure and has two novel features: 1) partitioning the whole file into slices such that only a small number of slices are accessed and checked during k nearest neighbor (kNN) search and 2) efficient handling of insertions of new vectors into the OVA-file, such that the average distance between the new vectors and those approximations near that position is minimized. To facilitate a search, we present an efficient approximate kNN algorithm named ordered VA-LOW (OVA-LOW) based on the proposed OVA-file. OVA-LOW first chooses possible OVA-slices by ranking the distances between their corresponding centers and the query vector, and then visits all approximations in the selected OVA-slices to work out approximate kNN. The number of possible OVA-slices is controlled by a user-defined parameter delta. By adjusting delta, OVA-LOW provides a trade-off between the query cost and the result quality. Query by video clip consisting of multiple frames is also discussed. Extensive experimental studies using real video data sets were conducted and the results showed that our methods can yield a significant speed-up over an existing VA-file-based method and (distance with high query result quality. Furthermore, by incorporating temporal correlation of video content, our methods achieved much more efficient performance