Automatic Audio Research Articles

Automatic Audio and Image Caption Generation with Deep Learning

A novel approach to image caption generation tailored specifically for visually impaired individuals. The proposed system employs advanced computer vision algorithms to analyze images and generate descriptive textual captions. Furthermore, it integrates seamless text-to-speech conversion functionality, allowing for the automatic transformation of these captions into spoken audio, thereby enabling access to visual content for individuals with visual impairments. The goal of this project is to generate descriptive captions for a given photograph or image. We achieve this by employing Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) models, both of which are advanced deep learning techniques. Using computer vision, the system identifies the content of the image and generates a relevant caption. This caption is then converted into audio using Natural Language Processing (NLP).

Automatic Audio Feature Extraction for Keyword Spotting

Automatic Audio Feature Extraction for Keyword Spotting

Effects of landscape and distance in automatic audio based bird species identification.

The present work focuses on how the landscape and distance between a bird and an audio recording unit affect automatic species identification. Moreover, it is shown that automatic species identification can be improved by taking into account the effects of landscape and distance. The proposed method uses measurements of impulse responses between the sound source and the recorder. These impulse responses, characterizing the effect of a landscape, can be measured in the real environment, after which they can be convolved with any number of recorded bird sounds to modify an existing set of bird sound recordings. The method is demonstrated using autonomous recording units on an open field and in two different types of forests, varying the distance between the sound source and the recorder. Species identification accuracy improves significantly when the landscape and distance effect is taken into account when building the classification model. The method is demonstrated using bird sounds, but the approach is applicable to other animal and non-animal vocalizations as well.

INTELLIGENT VIDEO SURVEILLANCE USING DEEP LEARNING

Video surveillance, also known as CCTV (closed-circuit television), is a rapidly growing industry that has been around for more than 30 years and has seen its fair share of technological advancements. In today's world, video surveillance has become an essential component for ensuring public safety. Security can be defined in a variety of ways depending on the context, such as theft detection, violence detection, explosion risk, and so on. The term "security" in crowded public places refers to almost any type of abnormal event.Intelligent video surveillance delivers cutting-edge smart security by recording unexpected activities in homes, offices, and public locations based on the preferences of the user. In the event of an abnormal incident, the video surveillance system will actively respond to detect actions in advance through real-time monitoring and promptly communicate data. The primary focus is on the use of deep learning techniques to provide tracking of a moving target, high-definition picture quality, and night vision technology triggered by motion sensors, which means the system isn't running when nothing is happening at your location, automatic audio and visual detection, video recording initiation, and detection of suspicious activities that can trigger and alert the systems in all climate conditions.Deep learning technology will be used in the data processing model design to visualise data for abnormal activities, and this design also proposes an intelligent surveillance system to quickly and effectively detect activities by sending a video image and an alert message to the web via real-time processing. Recent improvements in computer vision, particularly deep learning approaches, have opened up new possibilities for these systems, enhancing their capabilities and launching new research areas in this field.

Cloud-Based AI for Automatic Audio Production for Personalized Immersive XR Experiences

In this article, we focus on the machine-learning approach developed for automatic audio source recognition and mixing for the U.K. Government Department of Culture Media and Sport (DCMS) funded collaborative project called 5G Edge-XR. Leveraging graphics processing unit (GPU) acceleration, we deployed innovative algorithms in the cloud so that content can be automatically mixed on-the-fly for a personalized, immersive, and interactive experience for audiences. We describe the algorithms involved, the system architecture, how it has been implemented for immersive live boxing, and also how we are using it to enhance a live in-stadium experience.

Automatic Audio Chord Recognition With MIDI-Trained Deep Feature and BLSTM-CRF Sequence Decoding Model

With the advances of machine learning technologies, data-driven feature extraction and sequence modeling approaches are being widely explored for automatic chord recognition tasks. Currently, there is a bottleneck in the amount of enough annotated data for training robust acoustic models, as hand-annotating time-synchronized chord labels requires professional musical skills and considerable labor. To cope with this limitation, in this paper, we propose a convolutional neural network (CNN) based deep feature extractor, which is trained on a large set of time, synchronized musical instrument digital interface audio data pairs and can robustly estimate pitch class activations of real-world music audio recordings. The CNN feature extractor plus a bidirectional long short-term memory conditional random field decoding model forms the proposed hybrid system for automatic chord recognition. Experiments show that the proposed model is compatible for both regular major/minor triad chord classification and larger vocabulary chord recognition, and outperforms other state-of-the-art chord recognition systems.

Auto-Associative Recurrent Neural Networks and Long Term Dependencies in Novelty Detection for Audio Surveillance Applications

Machine Learning applied to Automatic Audio Surveillance has been attracting increasing attention in recent years. In spite of several investigations based on a large number of different approaches, little attention had been paid to the environmental temporal evolution of the input signal. In this work, we propose an exploration in this direction comparing the temporal correlations extracted at the feature level with the one learned by a representational structure. To this aim we analysed the prediction performances of a Recurrent Neural Network architecture varying the length of the processed input sequence and the size of the time window used in the feature extraction. Results corroborated the hypothesis that sequential models work better when dealing with data characterized by temporal order. However, so far the optimization of the temporal dimension remains an open issue.

Comparing the Effect of Audio and Visual Notifications on Workspace Awareness Using Head-Mounted Displays for Remote Collaboration in Augmented Reality

In many fields of activity, working in teams is necessary for completing tasks in a proper manner and often requires visual context-related information to be exchanged between team members. In such a collaborative environment, awareness of other people’s activity is an important feature of shared-workspace collaboration. We have developed an augmented reality framework for virtual colocation that supports visual communication between two people who are in different physical locations. We address these people as the remote user, who uses a laptop and the local user, who wears a head-mounted display with an RGB camera. The remote user can assist the local user in solving a spatial problem, by providing instructions in form of virtual objects in the view of the local user. For annotating the shared workspace, we use the state-of-the-art algorithm for localization and mapping without markers that provides “anchors” in the 3D space for placing virtual content. In this paper, we report on a user study that explores on how automatic audio and visual notifications about the remote user’s activities affect the local user’s workspace awareness. We used an existing game to research virtual colocation, addressing a spatial challenge on increasing levels of task complexity. The results of the user study show that participants clearly preferred visual notifications over audio or no notifications, no matter the level of the difficulty of the task.

Automatic Audio Mixing Algorithm for Real Time Free-Viewpoint TV

Automatic Audio Mixing Algorithm for Real Time Free-Viewpoint TV

Automatic Audio and Video Event Recognition in an Intelligent Resource Management System

Event recognition systems have high potential to support crisis management and emergency response. For large-scale scenarios, however, the sheer amount of possible audio and video channels requires adequate processing of the material by automatic means. In this article, the authors focus on automatic audio and video event recognition, by means of detecting abnormalities both in train noise as well as surveillance videos, and by conducting automatic speech recognition on fire fighter communication. All components are integrated in an overall intelligent resource management system. The authors elaborate on the challenges expected from real life data and the solutions that the authors applied. The overall system, based on Event-Driven Service-Oriented Architecture, has been implemented and partly integrated into the end users' infrastructures. The system has been continuously running for more than two years, collecting data for research purposes.

A comparative study on automatic audio–visual fusion for aggression detection using meta-information

Multimodal fusion is a complex topic. For surveillance applications audio–visual fusion is very promising given the complementary nature of the two streams. However, drawing the correct conclusion from multi-sensor data is not straightforward. In previous work we have analysed a database with audio–visual recordings of unwanted behavior in trains (Lefter et al., 2012) and focused on a limited subset of the recorded data. We have collected multi- and unimodal assessments by humans, who have given aggression scores on a 3 point scale. We showed that there are no trivial fusion algorithms to predict the multimodal labels from the unimodal labels since part of the information is lost when using the unimodal streams. We proposed an intermediate step to discover the structure in the fusion process. This step is based upon meta-features and we find a set of five which have an impact on the fusion process. In this paper we extend the findings in (Lefter et al., 2012) for the general case using the entire database. We prove that the meta-features have a positive effect on the fusion process in terms of labels. We then compare three fusion methods that encapsulate the meta-features. They are based on automatic prediction of the intermediate level variables and multimodal aggression from state of the art low level acoustic, linguistic and visual features. The first fusion method is based on applying multiple classifiers to predict intermediate level features from the low level features, and to predict the multimodal label from the intermediate variables. The other two approaches are based on probabilistic graphical models, one using (Dynamic) Bayesian Networks and the other one using Conditional Random Fields. We learn that each approach has its strengths and weaknesses in predicting specific aggression classes and using the meta-features yields significant improvements in all cases.

Modeling nuisance variabilities with factor analysis for GMM-based audio pattern classification

Abstract Audio pattern classification represents a particular statistical classification task and includes, for example, speaker recognition, language recognition, emotion recognition, speech recognition and, recently, video genre classification. The feature being used in all these tasks is generally based on a short-term cepstral representation. The cepstral vectors contain at the same time useful information and nuisance variability, which are difficult to separate in this domain. Recently, in the context of GMM-based recognizers, a novel approach using a Factor Analysis (FA) paradigm has been proposed for decomposing the target model into a useful information component and a session variability component. This approach is called Joint Factor Analysis (JFA), since it models jointly the nuisance variability and the useful information, using the FA statistical method. The JFA approach has even been combined with Support Vector Machines, known for their discriminative power. In this article, we successfully apply this paradigm to three automatic audio processing applications: speaker verification, language recognition and video genre classification. This is done by applying the same process and using the same free software toolkit. We will show that this approach allows for a relative error reduction of over 50% in all the aforementioned audio processing tasks.

Tempo and beat tracking for audio signals with music genre classification

Most people follow the music to hum or the rhythm to tap sometimes. We may get different meanings of a music style if it is explained or felt by different people. Therefore we cannot obtain a very explicit answer if there is no music notation. Tempo and beats are very important elements in the perceptual music. Therefore, tempo estimation and beat tracking are fundamental techniques in automatic audio processing, which are crucial to multimedia applications. We first develop an artificial neural network to classify the music excerpts into the evaluation preference. And then, with the preference classification, we can obtain accurate estimation for tempo and beats, by either Ellis's method or Dixon's method. We test our method with mixed data set which contains ten music genres from the 'ballroom dancer' database. Our experimental results show that the accuracy of our method is higher than only one individual Ellis's method or Dixon's method.

On the Correlation of Automatic Audio and Visual Segmentations of Music Videos

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> The study of the associations between audio and video content has numerous important applications in the fields of information retrieval and multimedia content authoring. In this work, we focus on music videos which exhibit a broad range of structural and semantic relationships between the music and the video content. To identify such relationships, a two-level automatic structuring of the music and the video is achieved separately. Note onsets are detected from the music signal, along with section changes. The latter is achieved by a novel algorithm which makes use of feature selection and statistical novelty detection approaches based on kernel methods. The video stream is independently segmented to detect changes in motion activity, as well as shot boundaries. Based on this two-level segmentation of both streams, four audio–visual correlation measures are computed. The usefulness of these correlation measures is illustrated by a query by video experiment on a 100 music video database, which also exhibits interesting genre dependencies. </para>

Statistical Analysis of features used in Automatic Audio Genre Classification

This paper presents statistical models for some of the most important features used to classify audio signals into musical genres. The genres used here are selected according to a given taxonomy. The features are computed for each genre using the signals from a dataset and the results are grouped into histograms. Each proposed statistical model consists of an estimated Probability Density Function (PDF), optimized to best fit a determined histogram and the optimization criterion is the minimization of the Mean Square Error (MSE). Finally, the paper discusses how these models can be applied to classify audio signals into genres.

Equipment and Strategies for Signal Quality Monitoring for Digital Television Transmission Networks

This paper identifies the problems of objective assessment of signal quality on digital TV networks. It aims also to present equipment for automatic audio and video quality assessment designed in the Quovadis project. Monitoring of signals on digital TV networks sets specific technical constraints that have to be taken into account.

Adaptive digital techniques for audio noise cancellation

Discusses two adaptive digital techniques for audio noise cancellation. The first technique, adaptive predictive deconvolution, uses an adaptive linear predictor to estimate and cancel (time) correlated noise components on an audio signal. The second technique, adaptive filtering, employs two audio signal inputs, the first having the desired audio signal along with noise and the second sensing principally the noise. The second, noise signal is adaptively filtered and subtracted from the first signal cancelling noise components common to the two inputs. A stand-alone digital signal processor has been developed to carry out these two noise cancellation techniques. This automatic digital audio processor, ADAP, carries out these processes in real time with up to a 256th order digital filter.

A DEVICE FOR AUTOMATIC AUDIO TAPE RECORDING1

Adaptation of a commercially available timer for use as a means of operating an audio tape recorder several times during the day is described. Data on a mother's rates of commanding her children were collected via both physically present observer and recorder methods in order to compare the usefulness of the recordings with direct observation. There was a high positive relationship between observer-recorder command rates, with the observer rates being consistently higher, when data were collected via both methods simultaneously as well as at different points in time.

Design of an Automatic Telephone Intercept Switch

A system has been designed to automatically perform the functions of intercepting misdirected telephone calls and providing correct dialing information. This paper concentrates on the design of the switching unit of the system. The switch is a space-division, wired-logic controlled device using dry-reed relays as crosspoints. It is organized so that separate networks perform the functions of concentrating incoming lines and selecting operator positions for identification of the the customer thinks he called. While the presently installed system is not completely automatic, provision is made for incorporation of automatic identification of the called number when the necessary telephone central office features become available. Although the system was built for a particular telephone installation, modular construction makes it readily adaptable to installations with a variety of requirements. The switching networks can handle a maximum of 200 incoming intercept trunks, 64 automatic audio response trunks, 22 called-number identifying operators, and 14 special intercept operators.