Automatic Video Indexing Research Articles

Comparison between Recurrent Networks and Temporal Convolutional Networks Approaches for Skeleton-Based Action Recognition.

Action recognition plays an important role in various applications such as video monitoring, automatic video indexing, crowd analysis, human-machine interaction, smart homes and personal assistive robotics. In this paper, we propose improvements to some methods for human action recognition from videos that work with data represented in the form of skeleton poses. These methods are based on the most widely used techniques for this problem—Graph Convolutional Networks (GCNs), Temporal Convolutional Networks (TCNs) and Recurrent Neural Networks (RNNs). Initially, the paper explores and compares different ways to extract the most relevant spatial and temporal characteristics for a sequence of frames describing an action. Based on this comparative analysis, we show how a TCN type unit can be extended to work even on the characteristics extracted from the spatial domain. To validate our approach, we test it against a benchmark often used for human action recognition problems and we show that our solution obtains comparable results to the state-of-the-art, but with a significant increase in the inference speed.

Towards wide-scale continuous gesture recognition model for in-depth and grayscale input videos

In recent years, gesture recognition in video sequences has aroused growing interest in the fields of computer vision and behavioral understanding, for example in the control of robots and video games, in the field of video surveillance, automatic video indexing or content-based video retrieval. Processing large-scale continuous gesture data with in-depth, grayscale input videos remains a primary challenge for academic researchers. A wide range of recognition models have been proposed to solve this problem but have not proven their great performance. The main contribution of this article to address this problem is to segment the sequences of continuous gestures into isolated gestures, using the average of the velocity information calculated on the basis of the estimate of the deep optical flow, and to extract a set of relevant descriptors, called characteristics. signature, in order to characterize different intensities and spatial information describing the location, speed and orientation of movement. Finally, to transmit to a linear SVM the characteristics built for the depth and gray scale sequences, for each isolated segment for its classification. The experimental study carried out on the various standard data collections namely KTH, Chalearn and Weizmann, on our model and on the main models that we have studied in the literature, as well as the analysis of the results, which we obtained, clearly show the limits of these studied models and confirms the performance of our model as well as efficiency in terms of precision, recall and robustness.

Video Index Point Detection and Extraction Framework Using Custom YoloV4 Darknet Object Detection Model

The trend of learning from videos instead of documents has increased. There could be hundreds and thousands of videos on a single topic, with varying degrees of context, content, and depth of the topic. The literature claims that learners are nowadays less interested in viewing a complete video but prefers the topic of their interests. This develops the need for indexing of video lectures. Manual annotation or topic-wise indexing is not new in the case of videos. However, manual indexing is time-consuming due to the length of a typical video lecture and intricate storylines. Automatic indexing and annotation are, therefore, better and efficient solutions. This research aims to identify the need for automatic video indexing for better information retrieval and ease users navigating topics inside a video. The automatically identified topics are referred to as “Index Points.” 137-layer YoloV4 Darknet Neural Network creates a custom object detector model. The model is trained on approximately 6000 video frames and then tested on a suite of 50 videos of around 20 hours of run time. Shot Boundary detection is performed using Structural Similarity fused with a Binary Search Pattern algorithm which outperformed the state-of-the-art SSIM technique by reducing the processing time to approximately 21% and providing around 96% accuracy. Generation of accurate index points in terms of true positives and false negatives is detected through precision, recall, and F1 score, which varies between 60–80% for every video. The results show that the proposed algorithm successfully generates a digital index with reasonable accuracy in topic detection.

Automated laparoscopic colorectal surgery workflow recognition using artificial intelligence: Experimental research.

Identifying laparoscopic surgical videos using artificial intelligence (AI) facilitates the automation of several currently time-consuming manual processes, including video analysis, indexing, and video-based skill assessment. This study aimed to construct a large annotated dataset comprising laparoscopic colorectal surgery (LCRS) videos from multiple institutions and evaluate the accuracy of automatic recognition for surgical phase, action, and tool by combining this dataset with AI. A total of 300 intraoperative videos were collected from 19 high-volume centers. A series of surgical workflows were classified into 9 phases and 3 actions, and the area of 5 tools were assigned by painting. More than 82 million frames were annotated for a phase and action classification task, and 4000 frames were annotated for a tool segmentation task. Of these frames, 80% were used for the training dataset and 20% for the test dataset. A convolutional neural network (CNN) was used to analyze the videos. Intersection over union (IoU) was used as the evaluation metric for tool recognition. The overall accuracies for the automatic surgical phase and action classification task were 81.0% and 83.2%, respectively. The mean IoU for the automatic tool segmentation task for 5 tools was 51.2%. A large annotated dataset of LCRS videos was constructed, and the phase, action, and tool were recognized with high accuracy using AI. Our dataset has potential uses in medical applications such as automatic video indexing and surgical skill assessments. Open research will assist in improving CNN models by making our dataset available in the field of computer vision.

Wildlife recognition in nature documentaries with weak supervision from subtitles and external data

We propose a weakly supervised framework for domain adaptation in a multi-modal context for multi-label classification. This framework is applied to annotate objects such as animals in a target video with subtitles, in the absence of visual demarcators. We start from classifiers trained on external data (the source, in our setting – ImageNet), and iteratively adapt them to the target dataset using textual cues from the subtitles. Experiments on a challenging dataset of wildlife documentaries validate the framework, with a final F1 measure of approximately 70%, which significantly improves over the results of a state-of-the-art approach, that is, applying classifiers trained on ImageNet without adaptation. The methods proposed here take us a step closer to object recognition in the wild and automatic video indexing.

Video segmentation using minimum ratio similarity measurement

Video segmentation plays an essential role in digital video processing, pattern recognition, security, video conferencing, etc. The convenience of the video is based on its content which is still impossible. One major challenging task of automatic video indexing is automatic detection of video shots. In this paper, a new algorithm is proposed to detect the shot boundary by using the minimum ratio similarity measurement between the characteristic features of two consecutive frames. Where, diverse parameters are calculated for each frame that creates a feature vector of size 40. The system performance is measured in terms of metric parameters. Also, a comparative study with alternative algorithms such as rapid cut detection, histogram-based method, etc. is done. Results suggest that the precision performance of the algorithm is independent of the nature of the video. The F-measure performance comparison shows that the proposed algorithm is the best with maximum average value and minimum standard deviation.

Video structure analysis for content-based indexing and categorisation of TV sports news

Text and video seem to be totally different digital media. Nevertheless, similarly to text, video is also hierarchically structured. The knowledge of the structures of textual data as well as of video data allows for more efficient indexing. The analogies of text and video structures are observed and discussed. Next, the juxtaposition is presented of two indexing processes, i.e., of text and video indexing based on the content analysis of their structure units. Then, different possible strategies of video indexing are also considered. Several frameworks of automatic detection and content-based categorisation of video shots and scenes have already been proposed in scientific literature. The experiments mainly concern with the classification of TV news, the categorisation of the sport events in TV sports news, the detection of people or special events etc. It has been observed that many sport videos have repetitive structure patterns. In the new indexing procedures implemented in the AVI - the Automatic Video Indexer - the knowledge of the structure of video shots and scenes allowed to reduce significantly the number of frames analysed in a given process.

Finding a needle in a haystack: an interactive video archive explorer for professional video searchers

Professional video searchers typically have to search for particular video fragments in a vast video archive that contains many hours of video data. Without having the right video archive exploration tools, this is a difficult and time consuming task that induces hours of video skimming. We propose the video archive explorer, a video exploration tool that provides visual representations of automatically detected concepts to facilitate individual and collaborative video search tasks. This video archive explorer is developed by employing a user-centred methodology, which ensures that the tool is more likely to fit to the end user needs. A qualitative evaluation with professional video searchers shows that the combination of automatic video indexing, interactive visualisations and user-centred design can result in an increased usability, user satisfaction and productivity.

GRADUAL TRANSITION DETECTION FOR VIDEO PARTITIONING USING MORPHOLOGICAL OPERATORS

Temporal segmentation of video data for partitioning the sequence into shots is a prerequisite in many applications: automatic video indexing and editing, old flm restoration, perceptual coding, etc. The detection of abrupt transitions or cuts has been thoroughly studied in previous works. In this paper we present a scheme to identify the most common gradual transitions, i.e., dissolves and wipes, which relies on mathematical morphology operators. The approach is restricted to fast techniques which require low computation (without motion estimation and adapted to compressed sequences) and are able to cope with random brightness variations (often occurring in old flms). The present study illustrates how the morphological operators can be used to analyze temporal series for detecting particular events, either working directly on the 1D signal or building an intermediate 2D image from the 1D signals to take advantage of the spatial operators.

A Divide-And-Rule Scheme For Shot Boundary Detection Based on SIFT

Shot boundary detection is an important fundamental process toward automatic video indexing, retrieval, editing, etc. After a critical review of most approaches seeking to solve this problem, we propose a novel shot boundary detection. To improve the performance of the algorithm and reduce the computational cost, frames that are clearly not shot boundaries are first removed from the original video. After that, a novel SIFT keypoint matching algorithm based on SVM is proposed, which is used to capture the changing statistics of different kinds of shot transitions so as to identify, not only abrupt transitions, but also gradual transitions(fade, dissolve, wipe) accordingly. At last, our system use different algorithms for different kinds of shot transitions to help us to get a better solution for shot boundary detection problem. Numerical experiments in the evaluation of TRECVID and a variety of film videos demonstrate that our method is capable of accurately detecting shot transitions, and could greatly reduce the computational cost.

A Personal Videocasting System with Intelligent TV Browsing for a Practical Video Application Environment

In this paper, a video broadcasting system between a home-server-type device and a mobile device is proposed. The home-server-type device can automatically extract semantic information from video contents, such as news, a soccer match, and a baseball game. The indexing results are utilized to convert the original video contents to a digested or arranged format. From the mobile device, a user can make recording requests to the home-server-type devices and can then watch and navigate recorded video contents in a digested form. The novelty of this study is the actual implementation of the proposed system by combining the actual IT environment that is available with indexing algorithms. The implementation of the system is demonstrated along with experimental results of the automatic video indexing algorithms. The overall performance of the developed system is compared with existing state-of-the-art personal video recording products.

Effective Fades and Flashlight Detection Based on Accumulating Histogram Difference

Scene change detection is a fundamental step in automatic video indexing, browsing and retrieval. Fade in and fade out are two kinds of gradually changing scenes which are difficult to be detected in comparison with the abruptly changing scenes. The salient character of flashlight effect is the luminance change, which is caused by abrupt appearance or disappearance of the illumination source. Performance of shot boundary detection is not satisfactory for the video sequences containing flashlights, if no flashlight discrimination strategy is adopted. In this paper, an effective fades and flashlight detection method is proposed for both the compressed and uncompressed videos, based on the accumulating histogram difference (AHD). This fades detection method is proposed in terms of their mathematical models. AHDs of all the two consecutive frames during fades transitions can be classified into six cases. The flashlight detection method is proposed based on the AHD and the energy variation characters. AHD and energy variation characters for the starting and ending frames of a flashlight have certain regularities, which can also be expressed by cases. Thus the fades and flashlight detection problems are converted into cases matching ones. Experimental results on several test video sequences with different bit rates show the effectiveness of the proposed AHD based fades and flashlight detection method

A new method for the characterization of telecine effect in video sequences

A great deal of the audio–visual content used for broadcasting and automatic video indexing comes from film. This film content has to undergo a conversion of its frame rate, from 24 fps , the characteristic frame rate of film recording, to 25 fps or 30 fps which are the frame rates used in TV broadcasting. This conversion is carried out by a telecine and consists of a periodic repetition of fields. In many applications the need arises to reverse this conversion ( inverse telecine). This paper presents a statistical method for automatic removal of the repeated fields present in the video sequence. The method consists of two steps: the first is to determine, using an energy detector, if the sequence has been telecined. Then, the two parameters necessary for removing the telecine effect are determined: the system for which the film has been telecined (for NTSC or PAL system) and the location of one repeated field.

Temporal video segmentation and classification of edit effects

The process of shot break detection is a fundamental component in automatic video indexing, editing and archiving. This paper introduces a novel approach to the detection and classification of shot transitions in video sequences including cuts, fades and dissolves. It uses the average inter-frame correlation coefficient and block-based motion estimation to track image blocks through the video sequence and to distinguish changes caused by shot transitions from those caused by camera and object motion. We present a number of experiments in which we achieve better results compared with two established techniques.

Indización automática de vídeo

After an initial discussion of the problems presented by image and video indexing as compared to text indexing, the authors describe some of the ba- sic techniques for automatic video indexing. The content-based retrieval paradigm and some automatic methods for segmentation and key-frame identification are further described. Certain low-level parameters for identifying an image are also introduced. The authors discuss the drawbacks of such automatic methods based solely on the image and give examples from projects using accompanying information as well, such as audio and captions. The article concludes by pointing out that the use of standards, like Mpeg-7, can pro- mote the development of new and richer applications based on video.

Detection of Gradual Transitions on Intensity-Time Distributions using Partial Linear Indicator

Video partitioning, which segments a video sequence into a meaningful group, is a basic technique for automatic video indexing and video editing. In this paper, we present a statistical analysis of video transitions in intensity-time distributions and suggest an implementation using partial linear indicator for fast video partitioning. Since it is very difficult to detect the gradual transitions of intensity variations, such as dissolve, fade and wipe, out of camera panning and object moving using a single measure, we analyze intensity variations through time frames and formalize the editing effects to the intensity distributions using variations of means and variances. Based on the idea that the intensity change introduced by editings, as opposed to motions, is more uniform, a fast method to extract the patterns of gradual transitions within motions is described. This method detects the gradually changed editing patterns and classifies the patterns into analyzed editing classes, even if there are simultaneous motions. The experimental results of our method present the effectiveness in detecting gradual transitions and the efficiency in processing time.

An automatic video indexing method based on shot classification

AbstractWith the recent increase in the number of news videos and in their value, their automatic indexing has been anticipated. Among various proposals for this problem, the use of textual data has been actively considered, and some applications of this type have actually been developed. Nonetheless, most of these approaches are textual data oriented, and a few of them consider the correspondences between their indexes (keywords) and image contents which are essentials for an image database.This paper proposes an automatic video indexing method which takes advantage of the characteristic of news videos: similar images have similar contents. The method tags indexes selectively according to typical image content. The method has been applied to actual news videos, and indexing has been successful for between 25% and 93% of “typical shots,” and for between 75% and 100% of “captionized shots.” © 2001 Scripta Technica, Syst Comp Jpn, 32(9): 32–41, 2001

Video Segmentation in the Wavelet Compressed Domain

Automatic video indexing is an important feature in the design of a video database. Recently, compressed domain techniques have become popular due to their inherent advantages of efficiency and lower complexity. On the other hand, wavelet transform has emerged as a powerful tool for efficient compression of visual information. A variety of wavelet-based video compression techniques have been reported in the literature. However, there has been little work done in the area of video indexing in the wavelet domain. In this paper, we present video segmentation techniques in a wavelet-based compression framework. These techniques employ wavelet coefficients, their distribution, and the motion vectors estimated by the associated video coder. Simulation results show that the proposed techniques provide a good indexing performance at a low complexity.

Long transition analysis for digital video sequences

Video indexing is a technique used to extract objects within a video sequence and index them so that they can be used for future retrieval. In many video sequences, special effects such as fade and wipe are incorporated, and in some cases, it is desirable to be able to include objects within such effects for indexing. In this paper, we introduce an automatic process that determines the type of transition and extracts information from it, so that this information can be used in object extraction. Such a process consists of four stages: shot boundary refinement, shot type determination, frame reconstruction for soft transitions, and shot classification for hard transitions. In this paper, we will give the implementation, timing, and performance analysis for each stage. Long transition analysis bridges the gap between shot boundary detection and object tracking and smoothes the process of automatic video indexing for video databases.

Algebraic retrieval of fragmentarily indexed video

When dealing with long video data, the task of identifying and indexing all meaningful subintervals that become answers to some queries is infeasible. It is infeasible not only when done by hand but even when done by using latest automatic video indexing techniques. Whether manually or automatically, it is only fragmentary video intervals that we can identify in advance of any database usage. Our goal is to develop a framework for retrieving meaningful intervals from such fragmentarily indexed video data. We propose a set of algebraic operations that includes ourglue join operations, with which we can dynamically synthesize all the intervals that are conceivably relevant to a given query. In most cases, since these operations also produce irrelevant intervals, we also define variousselection operations that are useful in excluding them from the answer set. We also show the algebraic properties possessed by those operations, which establish the basis of an algebraic query optimization.