Large Video Dataset Research Articles

Contribution of color in saliency model for videos

Much research has been concerned with the contribution of the low-level features of a visual scene to the deployment of visual attention. Bottom-up saliency models have been developed to predict the location of gaze according to these features. So far, color besides intensity, contrast and motion is considered as one of the primary features in computing bottom-up saliency. However, its contribution in guiding eye movements when viewing natural scenes has been debated. We investigated the contribution of color information in a bottom-up visual saliency model. The model efficiency was tested using the experimental data obtained on 45 observers who were eye-tracked while freely exploring a large dataset of color and grayscale videos. The two datasets of recorded eye positions, for grayscale and color videos, were compared with a luminance-based saliency model (Marat et al. Int J Comput Vis 82:231–243, 2009). We incorporated chrominance information to the model. Results show that color information improves the performance of the saliency model in predicting eye positions.

A Study on the Power Consumption of H.264/AVC-Based Video Sensor Network

There is an increasing interest in using video sensor networks (VSNs) as an alternative to existing video monitoring/surveillance applications. Due to the limited amount of energy resources available in VSNs, power consumption efficiency is one of the most important design challenges in VSNs. Video encoding contributes to a significant portion of the overall power consumption at the VSN nodes. In this regard, the encoding parameter settings used at each node determine the coding complexity and bitrate of the video. This, in turn, determines the encoding and transmission power consumption of the node and the VSN overall. Therefore, in order to calculate the nodes’ power consumption, we need to be able to estimate the coding complexity and bitrate of the video. In this paper, we modeled the coding complexity and bitrate of the H.264/AVC encoder, based on the encoding parameter settings used. We also propose a method to reduce the model estimation error for videos whose content changes within a specified period of time. We have conducted our experiments using a large video dataset captured from real-life applications in the analysis. Using the proposed model, we show how to estimate the VSN power consumption for a given topology.

Near-Duplicate Subsequence Matching Between the Continuous Stream and Large Video Dataset

In this paper, we study the subsequence matching problem of near-duplicate video detection. In particular, we address the application of monitoring a continuous stream with a large video dataset. To achieve real-time response and high accuracy, we propose a novel framework containing two characteristics . First, the subsequence matching is transformed to a 2-D Hough space projection of pairwise frame similarities between two subsequences. We present an approximate Hough transform that replaces the 2-D Hough space with a 1-D Hough space. The near-duplicate subsequence detection can be deemed to be the voting and searching in the 1-D Hough space with a lower time complexity. Second, a coarse-to-fine matching strategy is incorporated in the proposed framework. The coarse level matching selects the candidate videos based on the time-decay hit frequency between the query stream and dataset videos. The fine level matching applies the approximate Hough transform to detect the near-duplicate subsequences coexisting within the query stream and candidate videos. Several state-of-the-art methods are implemented for comparison. Experimental results show our framework outperforms in terms of accuracy and efficiency.

Structural Signatures for Passenger Vehicle Classification in Video

This paper focuses on a challenging pattern recognition problem of significant industrial impact, i.e., classifying vehicles from their rear videos as observed by a camera mounted on top of a highway with vehicles traveling at high speed. To solve this problem, this paper presents a novel feature called structural signature. From a rear-view video, a structural signature recovers the vehicle side profile information, which is crucial in its classification. As a vehicle moves away from a camera, its surfaces deform differently based on their relative orientation to the camera. This information is used to extract the structure of a vehicle, which captures the relative orientation of vehicle surfaces and the road surface. This paper presents a complete system that computes structural signatures and uses them for classification of passenger vehicles into sedans, pickups, and minivans/sport utility vehicles in highway videos. It analyzes the performance of the proposed system on a large video data set.

Using objective ground-truth labels created by multiple annotators for improved video classification: A comparative study

We address the problem of predicting category labels for unlabeled videos in a large video dataset by using a ground-truth set of objectively labeled videos that we have created. Large video databases like YouTube require that a user uploading a new video assign to it a category label from a prescribed set of labels. Such category labeling is likely to be corrupted by the subjective biases of the uploader. Despite their noisy nature, these subjective labels are frequently used as gold standard in algorithms for multimedia classification and retrieval. Our goal in this paper is NOT to propose yet another algorithm that predicts labels for unseen videos based on the subjective ground-truth. On the other hand, our goal is to demonstrate that the video classification performance can be improved if instead of using subjective labels, we first create an objectively labeled ground-truth set of videos and then train a classifier based on such a ground-truth so as to predict objective labels for the set of unlabeled videos.With regard to how we generate the objectively-labeled ground-truth dataset, we base it on the notion that when a video is labeled by a panel of diverse individuals, the majority opinion rendered by the panel may be taken to be the objective opinion. In this manner, using judgments provided by multiple human annotators, we have collected objective labels for a ground-truth dataset consisting of randomly-selected 1000 videos from the TinyVideos database that contains roughly 52,000 videos from YouTube (courtesy of Karpenko and Aarabi [1]).Through a fourfold cross-validation experiment on the ground-truth set, we demonstrate that the objective labels have a superior consistency compared to the subjective labels when used for video classification. We show that this claim is valid for several different kinds of feature sets that one can use to compare videos and with two different types of classifiers that one can use for label prediction.Subsequently, we use the ground-truth dataset of 1000 videos to predict the objective category labels of the remaining 51,000 videos. We compare the objective labels thus determined with the subjective labels provided by the video uploaders and qualitatively argue for the more informative nature of the objective labels.

Guest editorial: selected papers from ICIMCS 2011

International Conference on Internet Multimedia Computing and Services (ICIMCS) is an annual conference sponsored by ACM SIGMM China Chapter. The conference is especially interested in the latest technologies and applications that deal with the web-scale processing and management of heterogeneous data from the Internet for multimedia computing and service. ICIMCS 2011 held in Chengdu, China— the ancient hometown of lovely panda. The conference has attracted around 80 participants, including researchers from academia and industries across ten countries/regions, for sharing their recent works in the topics ranging from visual information analysis and mining, query processing and search, multimedia privacy and security. This special issue comprises the extended versions of five papers, including two best papers, and three papers from the regular and special sessions of ICIMCS 2011. These papers cover key issues in multimedia computing, including the leveraging of Internet resources for multi-modality fusion and visual classifier learning, preprocessing and indexing of million-scale Internet data, and sharing and streaming of Internet videos. Some of these techniques also demonstrate applications for emerging Internet services, such as video recommendation and product search system, by processing and modeling the heterogeneous forms of resources associated with multimedia data. The first two papers are about the web-scale processing of Internet data. The paper entitled ‘‘Video Recommendation over Multiple Information Sources’’, presented by Meng Wang and his colleagues from National University of Singapore, proposes a unified framework that explores heterogeneous information sources for video recommendation. The framework, based on multi-task SVM learning, aggregates multiple ranked lists generated from personal data, social network, and video metadata into an optimized list for recommendation. The framework is experimented on a large video dataset composed of 1-month social activities happened on Facebook and YouTube websites by 76 users. The second paper entitled ‘‘Multi-label Multiinstance Learning with Missing Object Tags’’, presented by Yi Shen and his colleagues from University of North Carolina at Charlotte, proposes a web-scale learning of object classifiers for free from a collection of user-tagged Flickr images as many as 10 millions. Particularly, the paper addresses three important issues toward fully automatic learning: scalable filtering of spam tags by distributed image clustering; joint modeling of loose tags and missing tags by multiple instance learning that is capable of performing tag prediction; structural learning that takes into account the object relationship to train discriminant classifiers. The next two papers address the search and mining of visual instances. The paper entitled ‘‘Combining Global and Local Matching of Multiple Features for Precise Item Image Retrieval’’, co-authored by Haojie Li and his colleagues from Dalian University of Technology and Nanjing C.-W. Ngo (&) City University of Hong Kong, Hong Kong, Hong Kong e-mail: cscwngo@cityu.edu.hk

Action unit classification using active appearance models and conditional random fields

In this paper, we investigate to what extent modern computer vision and machine learning techniques can assist social psychology research by automatically recognizing facial expressions. To this end, we develop a system that automatically recognizes the action units defined in the facial action coding system (FACS). The system uses a sophisticated deformable template, which is known as the active appearance model, to model the appearance of faces. The model is used to identify the location of facial feature points, as well as to extract features from the face that are indicative of the action unit states. The detection of the presence of action units is performed by a time series classification model, the linear-chain conditional random field. We evaluate the performance of our system in experiments on a large data set of videos with posed and natural facial expressions. In the experiments, we compare the action units detected by our approach with annotations made by human FACS annotators. Our results show that the agreement between the system and human FACS annotators is higher than 90% and underlines the potential of modern computer vision and machine learning techniques to social psychology research. We conclude with some suggestions on how systems like ours can play an important role in research on social signals.

Exploiting Visual-Audio-Textual Characteristics for Automatic TV Commercial Block Detection and Segmentation

Automatic TV commercial block detection (CBD) and commercial block segmentation (CBS) are two key components of a smart commercial digesting system. In this paper, we focus our research on CBD and CBS by the means of collaborative exploitation of visual-audio-textual characteristics embedded in commercials. Rather than utilizing exclusively visual-audio characteristics like most previous works, an abundance of textual characteristics associated with commercials are fully exploited. Additionally, Tri-AdaBoost, an interactive ensemble learning manner, is proposed to form a consolidated semantic fusion across visual, audio, and textual characteristics. In order to segment a detected commercial block into multiple individual commercials, additional informative descriptors including textual characteristics are introduced to boost the robustness in the detection of frame marked with product information (FMPI). Together with the characteristics of audio spectral variation pointer and silent position, FMPI can provide a kind of complementary representation architecture to model the similarity of intra-commercial and the dissimilarity of inter-commercial. Experiments are conducted on a large video dataset from both China central television (CCTV) channels and TRECVID'05, and promising experimental results show the effectiveness of the proposed scheme.

Saliency, Attention, Active Visual Search, and Picture Scanning

How is a complex visual scene processed? How is the selection of one particular location in a visual scene accomplished? Does it involve bottom-up, sensory-driven cues or top-down world knowledge expectations or both? How the decision is made when to terminate a fixation and move the gaze? How the decision is made where to direct the gaze in order to take the next sample? The goal of the special issue is to attempt to answer these questions by advancing our understanding of the state-of-the-art on bottom-up and top-down approaches to active visual search and picture scanning. The manner in which attention is involved was considered a highly relevant topic to the special issue. The special issue starts with a paper by Mital, Smith, Hill, and Henderson who investigated the contribution of lowto mid-level visual features to gaze location during free viewing of a large dataset of videos ranging in content and length. Their results showed that temporal features such as flicker, motion, and their respective contrasts were the most predictive of gaze location. Motion and mid-level visual features may influence gaze allocation in dynamic scenes, but remain unclear whether this influence is involuntary or due to correlations with higher-order factors such as scene semantics. Fuchs, Ansorge, Redies, and Leder investigated experimentally whether visual salience attracts attention in a bottom-up manner. Their results from participants viewing abstract and depictive paintings as well as photographs in free viewing and target-search experimental setups showed that local feature contrasts in color, luminance, and orientation drive attention in all conditions that a characteristic short-lived temporal profile of the salience-driven effect on fixations exists. Urban, Follet, Chamaret, Meur, and Baccino investigated the importance of spatial scale to predict visual attention. They proposed a fast attentional model and studied which frequency band predicts the best fixation locations during free viewing task. They found that medium frequencies (0.7–1.3 cycles per degree) globally allowed the best prediction of attention, with variability among categories. Fixation locations were found to be more predictable using medium to high frequencies in man-made street scenes and low to medium frequencies in natural landscape scenes. Foulsham and Underwood tested the hypothesis of whether visual saliency can be overridden in a search task with saccades being planned on the basis of target features, rather than being captured by saliency. Their results from three experimental conditions seem to indicate that saliency effects on search may have nothing to do with bottom-up saccade guidance. Humphrey and Underwood explored whether scanpath similarity could be improved by verbalising one’s memory of the picture and whether the previously observed domainspecific advantage was still present when no bottom-up information was available. Results showed recognition accuracy to be increased by post-stimulus verbalisation, and experts demonstrated an advantage for stimuli that contained domain-relevant information. Saliency influenced both verbal feedback and eye movements, but was moderated by domain expertise. Scanpaths were more J. G. Taylor (&) Department of Mathematics, King’s College, London WC2R 2LS, UK e-mail: taymore2002@aol.co.uk

Clustering of Gaze During Dynamic Scene Viewing is Predicted by Motion

Where does one attend when viewing dynamic scenes? Research into the factors influencing gaze location during static scene viewing have reported that low-level visual features contribute very little to gaze location especially when opposed by high-level factors such as viewing task. However, the inclusion of transient features such as motion in dynamic scenes may result in a greater influence of visual features on gaze allocation and coordination of gaze across viewers. In the present study, we investigated the contribution of low- to mid-level visual features to gaze location during free-viewing of a large dataset of videos ranging in content and length. Signal detection analysis on visual features and Gaussian Mixture Models for clustering gaze was used to identify the contribution of visual features to gaze location. The results show that mid-level visual features including corners and orientations can distinguish between actual gaze locations and a randomly sampled baseline. However, temporal features such as flicker, motion, and their respective contrasts were the most predictive of gaze location. Additionally, moments in which all viewers’ gaze tightly clustered in the same location could be predicted by motion. Motion and mid-level visual features may influence gaze allocation in dynamic scenes, but it is currently unclear whether this influence is involuntary or due to correlations with higher order factors such as scene semantics.

Extracting Objects and Events from MPEG Videos for Highlight-based Indexing and Retrieval

Automatic recognition of highlights from videos is a fundamental and challenging problem for content-based indexing and retrieval applications. In this paper, we propose techniques to solve this problem using knowledge supported extraction of semantics, and compressed-domain processing is employed for efficiency. Firstly, knowledgebased rules are utilized for shot detection on extracted DCimages, and statistical skin detection is applied for human object detection. Secondly, through filtering outliers in motion vectors, improved detection of camera motions like zooming, panning and tilting are achieved. Video highlight high-level semantics are then automatically extracted via low-level analysis in the detection of human objects and camera motion events, and finally these highlights are taken for shot-level annotation, indexing and retrieval. Results using a large test video data set have demonstrated the accuracy and robustness of the proposed techniques.

Integrating Illumination, Motion, and Shape Models for Robust Face Recognition in Video

The use of video sequences for face recognition has been relatively less studied compared to image-based approaches. In this paper, we present an analysis-by-synthesis framework for face recognition from video sequences that is robust to large changes in facial pose and lighting conditions. This requires tracking the video sequence, as well as recognition algorithms that are able to integrate information over the entire video; we address both these problems. Our method is based on a recently obtained theoretical result that can integrate the effects of motion, lighting, and shape in generating an image using a perspective camera. This result can be used to estimate the pose and structure of the face and the illumination conditions for each frame in a video sequence in the presence of multiple point and extended light sources. We propose a new inverse compositional estimation approach for this purpose. We then synthesize images using the face model estimated from the training data corresponding to the conditions in the probe sequences. Similarity between the synthesized and the probe images is computed using suitable distance measurements. The method can handle situations where the pose and lighting conditions in the training and testing data are completely disjoint. We show detailed performance analysis results and recognition scores on a large video dataset.

Hyper-rectangle based segmentation and clustering of large video data sets

Video information processing has been one of great challenging areas in the database community since it needs huge amount of storage space and processing power. In this paper, we investigate the problem of clustering large video data sets that are collections of video clips as foundational work for the subsequent processing such as video retrieval. A video clip, a sequence of video frames, is represented by a multidimensional data sequence, which is partitioned into video segments considering temporal relationship among frames, and then similar segments of the clip are grouped into video clusters . We present the effective video segmentation and clustering algorithm which guarantees the clustering quality to such an extent that satisfies predefined conditions, and show its effectiveness via experiments on various video data sets.