Temporal Video Segmentation Research Articles

A novel optical flow-based representation for temporal video segmentation

A novel optical flow-based representation for temporal video segmentation

Robust Complex Behaviour Modeling at 90Hz

Modeling complex crowd behaviour for tasks such as rare event detection has received increasing interest. However, existing methods are limited because (1) they are sensitive to noise often resulting in a large number of false alarms; and (2) they rely on elaborate models leading to high computational cost thus unsuitable for processing a large number of video inputs in real-time. In this paper, we overcome these limitations by introducing a novel complex behaviour modeling framework, which consists of a Binarized Cumulative Directional (BCD) feature as representation, novel spatial and temporal context modeling via an iterative correlation maximization, and a set of behaviour models, each being a simple Bernoulli distribution. Despite its simplicity, our experiments on three benchmark datasets show that it significantly outperforms the state-of-the-art for both temporal video segmentation and rare event detection. Importantly, it is extremely efficient — reaches 90Hz on a normal PC platform using MATLAB.

Temporal video segmentation: detecting the end-of-act in circus performance videos

The segmentation into acts of a circus performance video is challenging as the content has similar characteristics to other performance videos but is quite different from movies, TV programs, and home videos. Segmentation is useful as a long duration circus show usually contains several shorter segments that are acts. We propose a new method for detecting end-of-act within circus performance videos. Unlike other temporal video segmentation methods, this method does not rely on shot detection techniques and uses audio and video content analysis separately. First is audio content analysis, for detecting applause on the circus audio stream. Second is image analysis. The applause is further analyzed to test whether this applause occurs at the end-of-act. An end-of-act is detected, if the image(s) before and after the applause are different or there are black frames just after the applause. Otherwise, it is not the end-of-act. The experiment to detect end-of-act on Circus Oz performance videos achieved a 92.27 % recall and 49.05 % precision, providing useful clues that assist human annotators to segment circus video into acts.

Analysis of facial motions by means of RGB-D data

A temporal segmentation procedure that considers facial expression variations is presented in this paper. It is examined whether it is possible to use the Kinect sensor for improving temporal video segmentation by using information on scene depth in addition to color information. The presented video segmentation procedure that considers the information on scene depth makes it possible to secure invariance with respect to angle and lighting.

Online Kernel Slow Feature Analysis for Temporal Video Segmentation and Tracking

Slow feature analysis (SFA) is a dimensionality reduction technique which has been linked to how visual brain cells work. In recent years, the SFA was adopted for computer vision tasks. In this paper, we propose an exact kernel SFA (KSFA) framework for positive definite and indefinite kernels in Krein space. We then formulate an online KSFA which employs a reduced set expansion. Finally, by utilizing a special kind of kernel family, we formulate exact online KSFA for which no reduced set is required. We apply the proposed system to develop a SFA-based change detection algorithm for stream data. This framework is employed for temporal video segmentation and tracking. We test our setup on synthetic and real data streams. When combined with an online learning tracking system, the proposed change detection approach improves upon tracking setups that do not utilize change detection.

Linear-time subspace clustering via bipartite graph modeling.

We present a linear-time subspace clustering approach that combines sparse representations and bipartite graph modeling. The signals are modeled as drawn from a union of low-dimensional subspaces, and each signal is represented by a sparse combination of basis elements, termed atoms, which form the columns of a dictionary matrix. The sparse representation coefficients are arranged in a sparse affinity matrix, which defines a bipartite graph of two disjoint sets: 1) atoms and 2) signals. Subspace clustering is obtained by applying low-complexity spectral bipartite graph clustering that exploits the small number of atoms for complexity reduction. The complexity of the proposed approach is linear in the number of signals, thus it can rapidly cluster very large data collections. Performance evaluation of face clustering and temporal video segmentation demonstrates comparable clustering accuracies to state-of-the-art at a significantly lower computational load.

Adequation and hardware implementation of the color structure descriptor for real-time temporal video segmentation

In this paper, we focused on the study of the MPEG-7 color structure descriptor (CSD) for real-time temporal video segmentation. A new hardware architecture of the CSD was proposed. In the aim of optimizing this implementation in terms of hardware resources and execution time, different algorithm transformations have been tested for the considered application. The CSD was applied for different quantization levels in the HMMD color space and for different grey levels, with and without a frame skip. We have demonstrated that the use of a low number of quantization levels and a frame skip can significantly reduce the complexity and assure a better computing performance while preserving a satisfactory level of accuracy in terms of shot boundary detection rate. This is useful for implementation on resource-constrained hardware platform and multiprocessing applications. The performances of the proposed architecture were evaluated for different quantization levels to show the effect on occupied hardware resources and execution time. The comparative study demonstrates the effectiveness of the proposed architecture, which can operate in real-time video and without restriction on image size. In this work we have also designed a system on chip (SOC) for real-time video summarization based on the CSD. The proposed SOC integrating the CSD module was developed using a platform based on a Xilinx Virtex5 FPGA. A complete demonstration, including CSD extraction, shot boundary detection and key-frames visualization, was realized.

Temporal video segmentation by event detection: A novelty detection approach

Temporal segmentation of videos into meaningful image sequences containing some particular activities is an interesting problem in computer vision. We present a novel algorithm to achieve this semantic video segmentation. The segmentation task is accomplished through event detection in a frame-by-frame processing setup. We propose using one-class classification (OCC) techniques to detect events that indicate a new segment, since they have been proved to be successful in object classification and they allow for unsupervised event detection in a natural way. Various OCC schemes have been tested and compared, and additionally, an approach based on the temporal self-similarity maps (TSSMs) is also presented. The testing was done on a challenging publicly available thermal video dataset. The results are promising and show the suitability of our approaches for the task of temporal video segmentation.

PTU-016 Wireless Capsule Endoscope Localisation Based On Visual Odometry

IntroductionThe localisation of a wireless capsule endoscope (WCE) within the small-bowel is typically performed by wearable radiofrequency sensors triangulation. The accuracy of this approach is low.1 Only a few approaches...

A novel unsupervised approach for minimally-invasive video segmentation

Temporal segmentation of laparoscopic video is the first step toward identifying anomalies and interrupts, recognizing actions, annotating video and assessing the surgeons' learning curve. In this paper, a novel approach for temporal segmentation of minimally-invasive videos (MIVS) is proposed. Illumination variation, shadowing, dynamic backgrounds and tissue respiratory motion make it challenging to extract information from laparoscopic videos. These challenges if not properly addressed could increase the errors of data extraction modules. Therefore, in MIVS, several data sets are extracted from laparoscopic videos using different methods to alleviate error effects of data extraction modules on MIVS performance. Each extracted data set is segmented temporally with Genetic Algorithm (GA) after outlier removal. Three different cost functions are examined as objective function of GA. The correlation coefficient is calculated between objective values of the solutions visited by GA and their corresponding performance measures. Performance measures include detection rate, recognition rate and accuracy. Cost functions having negative correlations with all mentioned performance measures are selected. Finally, a multi-objective GA is executed on the data sets to optimize the selected cost functions. MIVS is tested on laparoscopic videos of varicocele and ureteropelvic junction obstruction surgeries collected from hasheminejad kidney center. Experimental results demonstrate that MIVS outperforms the state-of-the-art methods in terms of accuracy, detection rate and recognition rate.

Shot boundary detection in videos using Graph Cut Sets

The Shot Boundary Detection (SBD) is an early step for most of the video applications involving understanding, indexing, characterization, or categorization of video. The SBD is temporal video segmentation and it has been an active topic of research in the area of content based video analysis. The research efforts have resulted in a variety of algorithms. The major methods that have been used for shot boundary detection include pixel intensity based, histogram-based, edge-based, and motion vectors based, technique. Recently researchers have attempted use of graph theory based methods for shot boundary detection. The proposed algorithm is one such graph based model and employs graph partition mechanism for detection of shot boundaries. Graph partition model is one of the graph theoretic segmentation algorithms, which offers data clustering by using a graph model. Pair-wise similarities between all data objects are used to construct a weighted graph represented as an adjacency matrix (weighted similarity matrix) that contains all necessary information for clustering. Representing the data set in the form of an edge-weighted graph converts the data clustering problem into a graph partitioning problem. The algorithm is experimented on sports and movie videos and the results indicate the promising performance.

Intensity-driven dissolve detection adapted to synthetic video contents

We approach the problem around several main directions of video temporal segmentation and propose an intensity-based dissolve detection approach that is able to perform on animated video contents. It uses the hypothesis that during a dissolve the amount of fading-out and fading-in pixels should be significant compared with other visual transitions. We use this information as a visual discontinuity function. Instead of just applying a global threshold to filter these values, as most of the existing approaches do, we use a twin-thresholding approach and the shape analysis of the discontinuity measure. This allows us to reduce false detections caused by steep intensity fluctuations as well as to retrieve dissolves caught up in other visual transitions (e.g., caused by movement, color effects, etc.). Experimental tests conducted on more than 452 dissolve transitions show that whether classic approaches tend to fail, the proposed method is able to provide good performance achieving average precision and recall ratios above 94% and 79.6%, respectively.

Content-Based Management Service for Medical Videos

Development of health information technology has had a dramatic impact to improve the efficiency and quality of medical care. Developing interoperable health information systems for healthcare providers has the potential to improve the quality and equitability of patient-centered healthcare. In this article, we describe an automated content-based medical video analysis and management service that provides convenience and ease in accessing the relevant medical video content without sequential scanning. The system facilitates effective temporal video segmentation and content-based visual information retrieval that enable a more reliable understanding of medical video content. The system is implemented as a Web- and mobile-based service and has the potential to offer a knowledge-sharing platform for the purpose of efficient medical video content access.

Video analysis for identifying human operation difficulties and faucet usability assessment

As the world struggles to cope with a growing elderly population, concerns of how to preserve independence are becoming increasingly acute. A major hurdle to independent living is the inability to use everyday household objects. This work aims to automate the assessment of product usability for the elderly population using the tools of computer vision and machine learning. A novel video analysis technique is presented that performs temporal segmentation of video containing human–product interaction and automatically identifies time segments in which the human has difficulties in operating the product. The method has applications in the automatic assessment of the usability of various product designs via measuring the frequency of operation difficulties. The approach is applied to a case study of water faucet design for the older adult population with dementia. Experiments in the automatic analysis of a large database of real-world recorded videos confirm the effectiveness of the approach in providing valid temporal segmentation (accuracy 88.1%) and in the correct estimation of the relative advantage (or disadvantage) of one design over another in terms of operation difficulties in performing various actions.

Video Segmentation and Structuring for Indexing Applications

This paper introduces a complete framework for temporal video segmentation. First, a computationally efficient shot extraction method is introduced, which adopts the normalized graph partition approach, enriched with a non-linear, multiresolution filtering of the similarity vectors involved. The shot boundary detection technique proposed yields high precision (90%) and recall (95%) rates, for all types of transitions, both abrupt and gradual. Next, for each detected shot, the authors construct a static storyboard by introducing a leap keyframe extraction method. The video abstraction algorithm is 23% faster than existing techniques for similar performances. Finally, the authors propose a shot grouping strategy that iteratively clusters visually similar shots under a set of temporal constraints. Two different types of visual features are exploited: HSV color histograms and interest points. In both cases, the precision and recall rates present average performances of 86%.

Dominant sets based movie scene detection

Multimedia indexing and retrieval has become a challenging topic in organizing huge amount of multimedia data. This problem is not a trivial task for large visual databases; hence, segmentation into low- and high-level temporal video segments might improve the realization of this task. In this paper, we introduce a weighted undirected graph-based movie scene detection approach to detect semantically meaningful temporal video segments. The method is based on the idea of finding the dominant scene of the video according to the selected low-level feature. The proposed method starts from obtaining the most reliable solution first and exploit each solution in the subsequent steps recursively. The dominant movie scene boundary, which can be the highest probability to be the correct one, is determined and this scene boundary information is also exploited in the subsequent steps. We handle two partitioning strategies to determine the boundaries of the remaining scenes. One is a tree-based strategy and the other is an order-based strategy. The proposed dominant sets based movie scene detection method is compared with the graph-based video scene detection methods presented in literature.

Automatic and fast temporal segmentation for personalized news consuming

Automatic news program segmentation and classification becomes a hot topic, which reorganizes the news program according to the news' topics, and provides the on-demand services to mobile consumers or Internet/home TV consumers. This paper presents a personalized news consuming system, including the system architecture, consumption steps and key techniques. Then, focused on the core technique, i.e., video temporal segmentation, the automatic video temporal segmentation method is proposed, evaluated and compared with existing ones. Experimental results show that the proposed scheme is computational efficient and gets higher correct detection rate. These properties make it a suitable choice for the personalized news consuming system.

Time-Delayed Correlation Analysis for Multi-Camera Activity Understanding

We propose a novel approach to understanding activities from their partial observations monitored through multiple non-overlapping cameras separated by unknown time gaps. In our approach, each camera view is first decomposed automatically into regions based on the correlation of object dynamics across different spatial locations in all camera views. A new Cross Canonical Correlation Analysis (xCCA) is then formulated to discover and quantify the time delayed correlations of regional activities observed within and across multiple camera views in a single common reference space. We show that learning the time delayed activity correlations offers important contextual information for (i) spatial and temporal topology inference of a camera network; (ii) robust person re-identification and (iii) global activity interpretation and video temporal segmentation. Crucially, in contrast to conventional methods, our approach does not rely on either intra-camera or inter-camera object tracking; it thus can be applied to low-quality surveillance videos featured with severe inter-object occlusions. The effectiveness and robustness of our approach are demonstrated through experiments on 330 hours of videos captured from 17 cameras installed at two busy underground stations with complex and diverse scenes.

Feature extraction and clustering for dynamic video summarisation

In this paper an effective dynamic video summarisation algorithm is presented using audio-visual features extracted from videos. Audio, colour and motion features are dynamically fused using an adaptively weighting mechanism. Dissimilarities of temporal video segments are formulated using the extracted features before these segments are clustered using a fuzzy c-means algorithm with an optimally determined cluster number. The experimental results demonstrate the ability of the proposed algorithm to automatically summarise the videos with good performance.

Automatic video temporal segmentation based on multiple features

This paper investigates automatic video temporal segmentation techniques, also named shot boundary detection (SBD) techniques. Firstly, the existing SBD algorithms are reviewed in detail. Then, a new SBD algorithm is proposed aiming to obtain fast and accurate detection, and its performances are evaluated and compared with existing works. This algorithm computes the frame difference/similarity by such simple features as pixel difference and histogram difference, adopts motion-based difference to resist camera or object movements in the same shot and uses the flash detection to avoid false positives caused by light changes or flashes. The adopted features are computational efficient, and the combination of various features improve the detection accuracy. These properties make the algorithm suitable for real-time applications, such as broadcasted news segmentation.