Low-level Audio-visual Features Research Articles

Impact of Affective Multimedia Content on the Electroencephalogram and Facial Expressions

Most of the research in the field of affective computing has focused on detecting and classifying human emotions through electroencephalogram (EEG) or facial expressions. Designing multimedia content to evoke certain emotions has been largely motivated by manual rating provided by users. Here we present insights from the correlation of affective features between three modalities namely, affective multimedia content, EEG, and facial expressions. Interestingly, low-level Audio-visual features such as contrast and homogeneity of the video and tone of the audio in the movie clips are most correlated with changes in facial expressions and EEG. We also detect the regions associated with the human face and the brain (in addition to the EEG frequency bands) that are most representative of affective responses. The computational modeling between the three modalities showed a high correlation between features from these regions and user-reported affective labels. Finally, the correlation between different layers of convolutional neural networks with EEG and Face images as input provides insights into human affection. Together, these findings will assist in (1) designing more effective multimedia contents to engage or influence the viewers, (2) understanding the brain/body bio-markers of affection, and (3) developing newer brain-computer interfaces as well as facial-expression-based algorithms to read emotional responses of the viewers.

Driven to distraction: A lack of change gives rise to mind wandering

How does the dynamic structure of the external world direct attention? We examined the relationship between event structure and attention to test the hypothesis that narrative shifts (both theoretical and perceived) negatively predict attentional lapses. Self-caught instances of mind wandering were collected while 108 participants watched a 32.5 min film called The Red Balloon. We used theoretical codings of situational change and human perceptions of event boundaries to predict mind wandering in 5-s intervals. Our findings suggest a temporal alignment between the structural dynamics of the film and mind wandering reports. Specifically, the number of situational changes and likelihood of perceiving event boundaries in the prior 0–15 s interval negatively predicted mind wandering net of low-level audiovisual features. Thus, mind wandering is less likely to occur when there is more event change, suggesting that narrative shifts keep attention from drifting inwards.

Arousal Recognition Using Audio-Visual Features and FMRI-Based Brain Response

As the indicator of emotion intensity, arousal is a significant clue for users to find their interested content. Hence, effective techniques for video arousal recognition are highly required. In this paper, we propose a novel framework for recognizing arousal levels by integrating low-level audio-visual features derived from video content and human brain's functional activity in response to videos measured by functional magnetic resonance imaging (fMRI). At first, a set of audio-visual features which have been demonstrated to be correlated with video arousal are extracted. Then, the fMRI-derived features that convey the brain activity of comprehending videos are extracted based on a number of brain regions of interests (ROIs) identified by a universal brain reference system. Finally, these two sets of features are integrated to learn a joint representation by using a multimodal deep Boltzmann machine (DBM). The learned joint representation can be utilized as the feature for training classifiers. Due to the fact that fMRI scanning is expensive and time-consuming, our DBM fusion model has the ability to predict the joint representation of the videos without fMRI scans. The experimental results on a video benchmark demonstrated the effectiveness of our framework and the superiority of integrated features.

Learning Computational Models of Video Memorability from fMRI Brain Imaging.

Generally, various visual media are unequally memorable by the human brain. This paper looks into a new direction of modeling the memorability of video clips and automatically predicting how memorable they are by learning from brain functional magnetic resonance imaging (fMRI). We propose a novel computational framework by integrating the power of low-level audiovisual features and brain activity decoding via fMRI. Initially, a user study experiment is performed to create a ground truth database for measuring video memorability and a set of effective low-level audiovisual features is examined in this database. Then, human subjects' brain fMRI data are obtained when they are watching the video clips. The fMRI-derived features that convey the brain activity of memorizing videos are extracted using a universal brain reference system. Finally, due to the fact that fMRI scanning is expensive and time-consuming, a computational model is learned on our benchmark dataset with the objective of maximizing the correlation between the low-level audiovisual features and the fMRI-derived features using joint subspace learning. The learned model can then automatically predict the memorability of videos without fMRI scans. Evaluations on publically available image and video databases demonstrate the effectiveness of the proposed framework.

A framework for video event classification by modeling temporal context of multimodal features using HMM

Semantic high-level event recognition of videos is one of most interesting issues for multimedia searching and indexing. Since low-level features are semantically distinct from high-level events, a hierarchical video analysis framework is needed, i.e., using mid-level features to provide clear linkages between low-level audio-visual features and high-level semantics. Therefore, this paper presents a framework for video event classification using temporal context of mid-level interval-based multimodal features. In the framework, a co-occurrence symbol transformation method is proposed to explore full temporal relations among multiple modalities in probabilistic HMM event classification. The results of our experiments on baseball video event classification demonstrate the superiority of the proposed approach.

Movie abstraction via the progress of the storyline

An appropriate movie abstraction is helpful for movie producers to promote the progress of the storyline as well as for the audiences to capture the theme of the movie before watching the full-length movie. Most existing movie abstraction schemes rely heavily on video content only, which may not deliver ideal results because of the semantic gap between computer calculated low-level audiovisual features and human used high-level perceptual understanding. In this study, the authors incorporate script into movie content understanding and present a new movie abstraction approach via the progress of the storyline, which is the soul of a film that actually catches the audiences' attention. The authors first segment the movie scenes by analysis of the movie script. Then the authors conduct storyline analysis using the attention analysis and audiovisual features. Given the transition intensity values, the authors calculate the storyline progress score and adopt this as the criterion to generate movie abstraction. The promising experimental results demonstrate that the analysis of storyline evolution is an effective approach for the abstraction and understanding of movie content.

Multimodal genre classification of TV programs and YouTube videos

This paper presents an automatic video genre classification system, which utilizes several low level audio-visual features as well as cognitive and structural information, and in case of web videos...

Determination of emotional content of video clips by low-level audiovisual features

Affective analysis of video content has greatly increased the possibilities of the way we perceive and deal with media. Different kinds of strategies have been tried, but results are still opened to improvements. Most of the problems come from the lack of standardized test set and real affective models. In order to cope with these issues, in this paper we describe the results of our work on the determination of affective models for evaluation of video clips using audiovisual low-level features. The affective models were developed following two classes of psychological theories of affect: categorial and dimensional. The affective models were created from real data, acquired through a series of user experiments. They reflect the affective state of a viewer after watching a certain scene from a movie. We evaluate the detection of Pleasure, Arousal and Dominance coefficients as well as the detection rate of six affective categories. For this end, two Bayesian network topologies are used, a Hidden Markov Model and an Autoregressive Hidden Markov Model. The measurements were done using audio-only models, video-only models and fused models. Fusion is done using two different methods, a Decision Level Fusion and Feature Level Fusion. All tests were conducted using localized affective models, both categorial and dimensional. Results are presented in terms of detection rate and accuracy for affective families, affective dimensions and probabilistic networks. Arousal was the best detected dimension, followed by dominance and pleasure.

Indexing of Fictional Video Content for Event Detection and Summarisation

This paper presents an approach to movie video indexing that utilises audiovisual analysis to detect important and meaningful temporal video segments, that we term events. We consider three event classes, corresponding to dialogues, action sequences, and montages, where the latter also includes musical sequences. These three event classes are intuitive for a viewer to understand and recognise whilst accounting for over 90% of the content of most movies. To detect events we leverage traditional filmmaking principles and map these to a set of computable low-level audiovisual features. Finite state machines (FSMs) are used to detect when temporal sequences of specific features occur. A set of heuristics, again inspired by filmmaking conventions, are then applied to the output of multiple FSMs to detect the required events. A movie search system, named MovieBrowser, built upon this approach is also described. The overall approach is evaluated against a ground truth of over twenty-three hours of movie content drawn from various genres and consistently obtains high precision and recall for all event classes. A user experiment designed to evaluate the usefulness of an event-based structure for both searching and browsing movie archives is also described and the results indicate the usefulness of the proposed approach.

Investigation on unsupervised clustering algorithms for video shot categorization

Automatic categorization of video shots is the first and necessary step for organizing a long video stream into high-level scenes. However, existing techniques on video shot categorization still suffer from the problem of semantic gap between low-level audio-visual features and high-level semantic concepts. To bridge the gap, current researchers have been making efforts on the characterizations of: (1) spatio-temporal coherence among shots, and (2) bipartite correlation between descriptive features and shot categories. In the most recent works, spectral clustering methods and information-theoretic co-clustering (ITCC) have been actively studied and used to solve the above two issues, respectively. In this paper, we investigate the effectiveness of the two algorithms on video shot categorization. The comparison is examined in terms of estimating number of clusters and classification accuracies, where the K-means clustering algorithm is used as the benchmark. Experiments on 4-h sports videos show that both algorithms perform better than K-means. While the ITCC algorithm has advantages in estimating the number of clusters, the spectral clustering is better concerning the classification accuracy.

Modeling individual and group actions in meetings with layered HMMs

We address the problem of recognizing sequences of human interaction patterns in meetings, with the goal of structuring them in semantic terms. The investigated patterns are inherently group-based (defined by the individual activities of meeting participants, and their interplay), and multimodal (as captured by cameras and microphones). By defining a proper set of individual actions, group actions can be modeled as a two-layer process, one that models basic individual activities from low-level audio-visual (AV) features,and another one that models the interactions. We propose a two-layer hidden Markov model (HMM) framework that implements such concept in a principled manner, and that has advantages over previous works. First, by decomposing the problem hierarchically, learning is performed on low-dimensional observation spaces, which results in simpler models. Second, our framework is easier to interpret, as both individual and group actions have a clear meaning, and thus easier to improve. Third, different HMMs can be used in each layer, to better reflect the nature of each subproblem. Our framework is general and extensible, and we illustrate it with a set of eight group actions, using a public 5-hour meeting corpus. Experiments and comparison with a single-layer HMM baseline system show its validity.

Knowledge representation and semantic annotation of multimedia content

Knowledge representation and semantic annotation of multimedia documents typically have been pursued in two different directions. Previous approaches have focused either on low-level descriptors, such as dominant colour, or on the semantic content dimension and corresponding manual annotations, such as person or vehicle. Here, a knowledge infrastructure and an experimentation platform for semantic annotation to bridge the two directions are presented. Ontologies are being extended and enriched to include low-level audiovisual features and descriptors. Additionally, a tool that allows for linking low-level MPEG-7 visual descriptions to ontologies and annotations is presented. Thus, ontologies that include prototypical instances of high-level domain concepts together with a formal specification of the corresponding visual descriptors are constructed. This infrastructure is exploited by a knowledge-assisted analysis framework that may handle problems such as segmentation, tracking, feature extraction and matching in order to classify scenes, identify and label objects and thus automatically create the associated semantic metadata.

Computational media aesthetics: finding meaning beautiful

Content management's future is bright. Innovative media management, annotation, delivery, and navigation services will enrich online shopping, help-desk services, and anytime-anywhere training over wireless devices. Semantics-based annotations will break the traditional linear manner of accessing and browsing media and will support vignette-oriented access of audio and video. This can lead to new offerings of customized media management utilities for various market segments such as online education and training, advertising, news networks, and broadcasting studios. However, the semantic gap between the rich meaning that users want when they query and browse media and the shallowness of the content descriptions that we can actually compute is weakening today's automatic content-annotation systems. This is a crucial obstacle that we must overcome to achieve that bright future. A serious need exists to develop algorithms and technologies that can annotate content with deep semantics and establish semantic connections between media's form and function, for the first time letting users access indexed media and navigate content in unforeseeable and surprising ways. To address these underlying problems,, we advocate an approach that markedly departs from existing methods' based on detecting and annotating low-level audio-visual features. To go beyond representing what a video or movie directly shows, we postulate that we must analyze and interpret the content's visual, aural, and emotional impact. Our contention is that we must understand compositional and aesthetic media principles to guide content analysis.