Low-level Audio Research Articles

Saliency Prediction in Uncategorized Videos Based on Audio-Visual Correlation

Substantial research has been done in saliency modeling to make intelligent machines that can perceive and interpret their surroundings and focus only on the salient regions in a visual scene. But existing spatio-temporal saliency models either treat videos as merely image sequences excluding any audio information or are unable to cope with inherently varying content. Based on the hypothesis that an audiovisual saliency model will perform better than traditional spatio-temporal saliency models, this work aims to provide a generic preliminary audio/video saliency model. This is achieved by augmenting visual saliency map with an audio saliency map computed by synchronizing low-level audio and visual features. The proposed model was evaluated using different criteria against eye fixations data for a publicly available video dataset DIEM. The evaluation results show that the model outperforms two state-of-the-art visual spatio-temporal saliency models supporting our hypothesis that an audiovisual model performs better in comparison to a visual model for natural uncategorized videos.

Searching, Navigating, and Recommending Movies through Emotions: A Scoping Review

Movies offer viewers a broad range of emotional experiences, providing entertainment, and meaning. Following the PRISMA-ScR guidelines, we reviewed the literature on digital systems designed to help users search and browse movie libraries and offer recommendations based on emotional content. Our search yielded 83 eligible documents (published between 2000 and 2021). We identified 22 case studies, 34 empirical studies, 26 proof of concept, and one theoretical paper. User transactions (e.g., ratings, tags) were the preferred source of information. The documents examined approached emotions from both a categorical (n=35) and dimensional (n=18) perspectives, and nine documents offer a combination of both approaches. Although there are several authors mentioned, the references used are frequently dated, and 12 documents do not mention author or model used. We identified 61 words related to emotion or affect. Documents presented on average 1.36 positive terms and 2.64 negative terms. Sentiment analysis (n=31) is frequently used for emotion identification, followed by subjective evaluations (n=15), movie low-level audio and visual features (n = 11), and face recognition technologies (n=8). We discuss limitations and offer a brief review of current emotion models and research.

Supplemental Material for Dissecting the Groove Experience: The Role of Instrumentation, Low-Level Audio Features, and Personal Musical Background

Supplemental Material for Dissecting the Groove Experience: The Role of Instrumentation, Low-Level Audio Features, and Personal Musical Background

Multimodal soccer highlight identification using a sparse subset of frames integrating long-term sliding windows

The massive growth of audiences eager for sport content has substantially increased workers’ demand in this profitable segment. Highlight identification is vital for summarizing football matches. Decision support tools can significantly reduce the number of company employees required to tackle such a task, widely benefiting workforce resource allocation. This paper discusses the development of an automatic football highlight detector. The proposed system exploits discriminative low-level audio and video features extracted from a compact set of irregularly time–spaced frames that integrate a long-term sliding window. A new mixed wrapper-probabilistic algorithm leverages a cost-effective selection of the most significant frames submitted to a robust multi-frame consensus classification scheme. By considering a comprehensive database integrating 30 full matches, the proposed approach achieves a highlight identification rate of 100% (including all annotated goals), conjugated with a match-time compression rate of about 94%, when employing a Random Forest classifier.

Variation in Multitrack Mixes: Analysis of Low-level Audio Signal Features

To further the development of intelligent music production tools, towards generating mixes that would realistically be created by a human mix-engineer, it is important to understand what kind of mixes can be created, and are typically created, by human mix-engineers. This paper presents an analysis of 1501 mixes, over 10 different songs, created by mix-engineers. The primary dimensions of variation in the full dataset of mixes were ‘amplitude’, ‘brightness’, ‘bass’ and ‘width’, as determined by feature-extraction and subsequent principal component analysis. The distribution of representative features approximated a normal distribution and this is then used to obtain general trends and tolerance bounds for these features. The results presented here are useful as parametric guidance for intelligent music production systems.

Healthcare Big Data Voice Pathology Assessment Framework

The fast-growing healthcare big data plays an important role in healthcare service providing. Healthcare big data comprise data from different structured, semi-structured, and unstructured sources. These data sources vary in terms of heterogeneity, volume, variety, velocity, and value that traditional frameworks, algorithms, tools, and techniques are not fully capable of handling. Therefore, a framework is required that facilitates collection, extraction, storage, classification, processing, and modeling of this vast heterogeneous volume of data. This paper proposes a healthcare big data framework using voice pathology assessment (VPA) as a case study. In the proposed VPA system, two robust features, MPEG-7 low-level audio and the interlaced derivative pattern, are used for processing the voice or speech signals. The machine learning algorithms in the form of a support vector machine, an extreme learning machine, and a Gaussian mixture model are used as the classifier. In the experiments, the proposed VPA system shows its efficiency in terms of accuracy and time requirement.

Multimodal Integration of Dynamic Audio–Visual Cues in the Communication of Agreement and Disagreement

Recent research has stressed the importance of using multimodal and dynamic features to investigate the role of nonverbal behavior in social perception. This paper examines the influence of low-level visual and auditory cues on the communication of agreement, disagreement, and the ability to convince others. In addition, we investigate whether the judgment of these attitudes depends on ratings of socio-emotional dimensions such as dominance, arousal, and valence. The material we used consisted of audio–video excerpts that represent statements of agreement and disagreement, as well as neutral utterances taken from political discussions. Each excerpt was rated on a number of dimensions: agreement, disagreement, dominance, valence, arousal, and convincing power in three rating conditions: audio-only, video-only, and audio–video. We extracted low-level dynamic visual features using optical flow. Auditory features consisted of pitch measurements, vocal intensity, and articulation rate. Results show that judges were able to distinguish statements of disagreement from agreement and neutral utterances on the basis of nonverbal cues alone, in particular, when both auditory and visual information were present. Visual features were more influential when presented along with auditory features. Perceivers mainly used changes in pitch and the maximum speed of vertical movements to infer agreement and disagreement, and targets appeared more convincing when they showed consistent and rapid movements on the vertical plane. The effect of nonverbal features on ratings of agreement and disagreement was completely mediated by ratings of dominance, valence, and arousal, indicating that the impact of low-level audio–visual features on the perception of agreement and disagreement depends on the perception of fundamental socio-emotional dimensions.

Fast play: a novel feature for digital consumer video devices

The advent of digital devices with innovative features has changed the world of home video entertainment. Nowadays we have access to devices that store several hours of video material, skip commercials, and even pause the play out of a live event. Despite these innovative features, modern digital devices access to recorded material in a way very similar to old analog VCRs. In this paper we propose Fast Play, a mechanism that uses simple low-level audio and video analysis to provide a novel feature in digital video devices. This novel feature is a hybrid between the classic Play service and the Fast-Forward service, and aims at playing out videos in a time-reduced manner, by maintaining the time evolving nature of the original video, and by providing videos with completely intelligible audio. An experimental assessment shows that Fast Play produces videos that are informative, enjoyable, largely reduced, and also shows that users really appreciate the Fast Play feature as it produces a pleasant surrogate of the original video.

Digital TV

Digital TV, from a TV manufacturer's standpoint, could simply imply replacement of the conventional analog video signal processing system in a color TV receiver by digital signal processing. This concept could also be extended to the low level audio and deflection systems portion of the TV set. Digital TV could also imply the addition of "digital" features such as Picture-in- Picture (PIP), Multi Picture-in-Picture (MPIP), Freeze Frame, Zoom, etc. and picture performance improvements such as Progressive Scan, motion-compensated Noise Reduction and Frame Combing, etc. These latter schemes need storage of a field and/or frame of video information. This paper gives an overview of the above concepts and addresses them from several viewpoints, including implementation, cost, manufacturing advantages, picture quality etc.