Perceptual Audio Quality Research Articles

Speech Inpainting Based on Multi-Layer Long Short-Term Memory Networks

Audio inpainting plays an important role in addressing incomplete, damaged, or missing audio signals, contributing to improved quality of service and overall user experience in multimedia communications over the Internet and mobile networks. This paper presents an innovative solution for speech inpainting using Long Short-Term Memory (LSTM) networks, i.e., a restoring task where the missing parts of speech signals are recovered from the previous information in the time domain. The lost or corrupted speech signals are also referred to as gaps. We regard the speech inpainting task as a time-series prediction problem in this research work. To address this problem, we designed multi-layer LSTM networks and trained them on different speech datasets. Our study aims to investigate the inpainting performance of the proposed models on different datasets and with varying LSTM layers and explore the effect of multi-layer LSTM networks on the prediction of speech samples in terms of perceived audio quality. The inpainted speech quality is evaluated through the Mean Opinion Score (MOS) and a frequency analysis of the spectrogram. Our proposed multi-layer LSTM models are able to restore up to 1 s of gaps with high perceptual audio quality using the features captured from the time domain only. Specifically, for gap lengths under 500 ms, the MOS can reach up to 3~4, and for gap lengths ranging between 500 ms and 1 s, the MOS can reach up to 2~3. In the time domain, the proposed models can proficiently restore the envelope and trend of lost speech signals. In the frequency domain, the proposed models can restore spectrogram blocks with higher similarity to the original signals at frequencies less than 2.0 kHz and comparatively lower similarity at frequencies in the range of 2.0 kHz~8.0 kHz.

Towards Improved Objective Perceptual Audio Quality Assessment - Part 1: A Novel Data-Driven Cognitive Model

Towards Improved Objective Perceptual Audio Quality Assessment - Part 1: A Novel Data-Driven Cognitive Model

Nonintrusive Perceptual Audio Quality Assessment for User-Generated Content Using Deep Learning

With the boom of social media communication, teleconferencing, and online classes, audiovisual communication over bandwidth strained networks has become an integral part of our lives. Consequently, the growing demand for the quality of experience necessitates developing algorithms to measure and enrich user experience. Prior studies have mainly focused on assessing speech quality and intelligibility with reference to audio quality assessment, while other categories in user-generated multimedia (UGM) are less explored. Moreover, frequency-domain properties of speech and UGM audio are significantly different from each other. Furthermore, there is a lack of a standard dataset for the quality assessment of UGM. Considering these limitations, in this article, we first develop the IIT-JMU-UGM audio dataset consisting of 1150 audio clips, with diverse context, content, and types of degradation commonly observed in real-world scenarios and annotated with the subjective quality scores. Finally, we propose a non-intrusive audio quality assessment metric using a stacked gated-recurrent-unit-based deep learning framework. The proposed model outperforms several baseline methods, including state-of-the-art non-intrusive and intrusive approaches. The resulting Pearson’s correlation coefficient of 0.834 indicates that the proposed method efficiently mirrors human auditory perception.

BioCPPNet: automatic bioacoustic source separation with deep neural networks

We introduce the Bioacoustic Cocktail Party Problem Network (BioCPPNet), a lightweight, modular, and robust U-Net-based machine learning architecture optimized for bioacoustic source separation across diverse biological taxa. Employing learnable or handcrafted encoders, BioCPPNet operates directly on the raw acoustic mixture waveform containing overlapping vocalizations and separates the input waveform into estimates corresponding to the sources in the mixture. Predictions are compared to the reference ground truth waveforms by searching over the space of (output, target) source order permutations, and we train using an objective function motivated by perceptual audio quality. We apply BioCPPNet to several species with unique vocal behavior, including macaques, bottlenose dolphins, and Egyptian fruit bats, and we evaluate reconstruction quality of separated waveforms using the scale-invariant signal-to-distortion ratio (SI-SDR) and downstream identity classification accuracy. We consider mixtures with two or three concurrent conspecific vocalizers, and we examine separation performance in open and closed speaker scenarios. To our knowledge, this paper redefines the state-of-the-art in end-to-end single-channel bioacoustic source separation in a permutation-invariant regime across a heterogeneous set of non-human species. This study serves as a major step toward the deployment of bioacoustic source separation systems for processing substantial volumes of previously unusable data containing overlapping bioacoustic signals.

Human Laughter Generation using Hybrid Generative Models

Laughter is one of the most important nonverbal sound that human generates. It is a means for expressing his emotions. The acoustic and contextual features of this specific sound are different from those of speech and many difficulties arise during their modeling process. During this work, we propose an audio laughter generation system based on unsupervised generative models: the autoencoder (AE) and its variants. This procedure is the association of three main sub-process, (1) the analysis which consist of extracting the log magnitude spectrogram from the laughter database, (2) the generative models training, (3) the synthesis stage which incorporate the involvement of an intermediate mechanism: the vocoder. To improve the synthesis quality, we suggest two hybrid models (LSTM-VAE, GRU-VAE and CNN-VAE) that combine the representation learning capacity of variational autoencoder (VAE) with the temporal modelling ability of a long short-term memory RNN (LSTM) and the CNN ability to learn invariant features. To figure out the performance of our proposed audio laughter generation process, objective evaluation (RMSE) and a perceptual audio quality test (listening test) were conducted. According to these evaluation metrics, we can show that the GRU-VAE outperforms the other VAE models.

Objective Measures of Perceptual Audio Quality Reviewed: An Evaluation of Their Application Domain Dependence

Over the past few decades, computational methods have been developed to estimate perceptual audio quality. These methods, also referred to as objective quality measures, are usually developed and intended for a specific application domain. Because of their convenience, they are often used outside their original intended domain, even if it is unclear whether they provide reliable quality estimates in this case. This work studies the correlation of well-known state-of-the-art objective measures with human perceptual scores in two different domains: audio coding and source separation. The following objective measures are considered: fwSNRseg, dLLR, PESQ, PEAQ, POLQA, PEMO-Q, ViSQOLAudio, (SI-)BSSEval, PEASS, LKR-PI, 2f-model, and HAAQI. Additionally, a novel measure (SI-SA2f) is presented, based on the 2f-model and a BSSEval-based signal decomposition. We use perceptual scores from 7 listening tests about audio coding and 7 listening tests about source separation as ground-truth data for the correlation analysis. The results show that one method (2f-model) performs significantly better than the others on both domains and indicate that the dataset for training the method and a robust underlying auditory model are crucial factors towards a universal, domain-independent objective measure.

Fast Spectrogram Inversion Using Multi-Head Convolutional Neural Networks

We propose the multi-head convolutional neural network (MCNN) architecture for waveform synthesis from spectrograms. Nonlinear interpolation in MCNN is employed with transposed convolution layers in parallel heads. MCNN achieves more than an order of magnitude higher compute intensity than commonly-used iterative algorithms like Griffin-Lim, yielding efficient utilization for modern multi-core processors, and very fast (more than 300x real-time) waveform synthesis. For training of MCNN, we use a large-scale speech recognition dataset and losses defined on waveforms that are related to perceptual audio quality. We demonstrate that MCNN constitutes a very promising approach for high-quality speech synthesis, without any iterative algorithms or autoregression in computations.

Perceived Audio Quality Analysis in Digital Audio Broadcasting Plus System Based on PEAQ

Perceived Audio Quality Analysis in Digital Audio Broadcasting Plus System Based on PEAQ

Objective Assessment of Perceptual Audio Quality Using ViSQOLAudio

Digital audio broadcasting services transmit substantial amounts of data that is encoded to minimize bandwidth whilst maximizing user quality of experience. Many large service providers continually alter codecs to improve the encoding process. Performing subjective tests to validate each codec alteration would be impractical, necessitating the use of objective perceptual audio quality models. This paper evaluates the quality scores from ViSQOLAudio, an objective perceptual audio quality model, against the quality scores of PEAQ, POLQA, and PEMO-Q on three datasets containing fullband audio encoded with a variety of codecs and bitrates. The results show that ViSQOLAudio was more accurate than all other models on two of the datasets and performed well on the third, demonstrating the utility of ViSQOLAudio for predicting the perceptual audio quality for encoded music.

Directional and Causal Information Flow in EEG for Assessing Perceived Audio Quality

In this paper, electroencephalography (EEG) measurements are used to infer change in cortical functional connectivity in response to change in audio stimulus. Experiments are conducted wherein the EEG activity of human subjects is recorded as they listen to audio sequences whose quality varies with time. A causal information theoretic framework is then proposed to measure the information flow between EEG sensors appropriately grouped into different regions of interest (ROI) over the cortex. A new causal bidirectional information (CBI) measure is defined as an improvement over standard directed information measures for the purposes of identifying connectivity between ROIs in a generalized cortical network setting. CBI can be intuitively interpreted as a causal bidirectional modification of directed information, and inherently calculates the divergence of the observed data from a multiple access channel with feedback. Further, we determine the analytical relationship between the different causal measures and compare how well they are able to distinguish between the perceived audio quality. The connectivity results inferred indicate a significant change in the rate of information flow between ROIs as the subjects listen to different audio qualities, with CBI being the best in discriminating between the perceived audio quality, compared to using standard directed information measures.

Audio-Visual Multimedia Quality Assessment: A Comprehensive Survey

Measuring perceived quality of audio-visual signals at the end-user has become an important parameter in many multimedia networks and applications. It plays a crucial role in shaping audio-visual processing, compression, transmission and systems, along with their implementation, optimization, and testing. Service providers are enacting different quality of service (QoS) solutions to issue the best quality of experience (QoE) to their customers. Thus, devising precise perception-based quality metrics will greatly help improving multimedia services over wired and wireless networks. In this paper, we provide a comprehensive survey of the works that have been carried out over recent decades in perceptual audio, video, and joint audio-visual quality assessments, describing existing methodologies in terms of requirement of a reference signal, feature extraction, feature mapping, and classification schemes. In this context, an overview of quality formation and perception, QoS, QoE as well as quality of perception is also presented. Finally, open issues and challenges in audio-visual quality assessment are highlighted and potential future research directions are discussed.

Audio bandwidth extension using ensemble of recurrent neural networks

In audio communication systems, the perceptual audio quality of the reproduced audio signals such as the naturalness of the sound is limited by the available audio bandwidth. In this paper, a wideband to super-wideband audio bandwidth extension method is proposed using an ensemble of recurrent neural networks. The feature space of wideband audio is firstly divided into different regions through clustering. For each region in the feature space, a specific recurrent neural network with a sparsely connected hidden layer, referred as the echo state network, is employed to dynamically model the mapping relationship between wideband audio features and high-frequency spectral envelope. In the following step, the outputs of multiple echo state networks are weighted and fused by means of network ensemble, in order to further estimate the high-frequency spectral envelope. Finally, combining the high-frequency fine spectrum extended by spectral translation, the proposed method can effectively extend the bandwidth of wideband audio to super wideband. Objective evaluation results show that the proposed method outperforms the hidden Markov model-based bandwidth extension method on the average in terms of both static and dynamic distortions. In subjective listening tests, the results indicate that the proposed method is able to improve the auditory quality of the wideband audio signals and outperforms the reference method.

Perceived Audio Quality of Sounds Degraded by Nonlinear Distortions and Single-Ended Assessment Using HASQI

For field recordings and user generated content recorded on phones, tablets, and other mobile devices nonlinear distortions caused by clipping and limiting at pre-amplification stages, and dynamic range control (DRC) are common causes of poor audio quality. A single-ended method to detect these distortions and predict perceived degradation in speech, music, and soundscapes has been developed. This was done by training an ensemble of decision trees. During training, both clean and distorted audio was available and so the perceived quality could be gauged using HASQI (Hearing Aid Sound Quality Index). The new single-ended method can correctly predict HASQI from distorted samples to an accuracy of ±0.19 (95% confidence interval) using a quality range between 0.0 and 1.0. The method also has potential for estimating HASQI when other types of degradations are present. Subsequent perceptual tests validated the method for music and soundscapes. For the average mean opinion score for perceived audio quality on a scale from 0 to 1, the single ended method could estimate it within ±0.33.

Subjective and Objective Assessment of Perceived Audio Quality of Current Digital Audio Broadcasting Systems and Web-Casting Applications

This paper investigates the impact of different audio codecs typically deployed in current digital audio broadcasting (DAB) systems and web-casting applications, which represent a main source of quality impairment in these systems and applications, on the quality perceived by the end user. Both subjective and objective assessments are used. Two different audio quality prediction models, namely Perceptual Evaluation of Audio Quality (PEAQ) and Perceptual Objective Listening Quality Assessment (POLQA) Music, are evaluated by comparing the predictions with subjectively obtained grades. The results show that the degradations introduced by the typical lossy audio codecs deployed in current DAB systems and web-casting applications operating at the lowest bit rate typically used in these distribution systems and applications seriously impact the subjective audio quality perceived by the end user. Furthermore, it is shown that a retrained POLQA Music provides the best overall correlations between predicted objective measurements and subjective scores allowing to predict the final perceived quality with good accuracy when scores are averaged over a small set of musical fragments (R = 0.95).

Data reduction of audio by exploiting musical repetition

This paper presents and evaluates a method of audio compression specifically designed to exploit the natural repetition that occurs within musical audio. Our system is entitled Audio Compression Exploiting Repetition (ACER). ACER is a perceptual technique, but one that does not consider exploiting masking, but rather attempts to apply the principles of Lempel-Ziv and run-length encoding, by substituting audio sequences for numeric or character strings. The ACER procedure applies a pseudo exhaustive search process and spectral difference grading. Since ACER exploits musical structure, the amount of data reduction achieved varies from piece-to-piece. The system is described before results on a corpus of material are presented. The analysis shows moderate amounts of data reduction take place whilst the system is operating within parameters designed to maintain high-levels of perceptual audio quality, whilst lower rates of perceptual quality yield greater data reduction. Objective quality evaluations are conducted that reveal degradation in fidelity that is relative to the compression parameters.

A smart background music mixing algorithm for portable digital imaging devices

In this paper, we propose a smart background music (BGM) mixing algorithm for portable digital imaging devices to enable users to enjoy video content with BGM. The proposed algorithm automatically adjusts the BGM output energy based on the activity and energy of foreground audio (FGA) contained in a video file. To this end, the proposed algorithm classifies each segment of FGA as speech, non-speech, or a mixed signal. After that, it estimates a scale factor for mixing FGA and BGM according to the signal classification result and the energy of FGA. In addition, a fade-in and fade-out process is incorporated in the proposed algorithm in order to improve the perceptual quality of output audio at the boundaries where signal classification is changed. In order to demonstrate the effectiveness of the proposed algorithm, we implement it on a portable digital imaging device in real time and compare the user's preference of the proposed algorithm with those of conventional algorithms that mixes FGA with BGM based on voice activity detection or a predefined fixed scale factor. It is shown from the experiments that the proposed algorithm is pretty much preferred by around 79%, compared to the conventional algorithms.

Perceptual-based quality assessment for audio–visual services: A survey

Accurate measurement of the perceived quality of audio–visual services at the end-user is becoming a crucial issue in digital applications due to the growing demand for compression and transmission of audio–visual services over communication networks. Content providers strive to offer the best quality of experience for customers linked to their different quality of service (QoS) solutions. Therefore, developing accurate, perceptual-based quality metrics is a key requirement in multimedia services. In this paper, we survey state-of-the-art signal-driven perceptual audio and video quality assessment methods independently, and investigate relevant issues in developing joint audio–visual quality metrics. Experiments with respect to subjective quality results have been conducted for analyzing and comparing the performance of the quality metrics. We consider emerging trends in audio–visual quality assessment, and propose feasible solutions for future work in perceptual-based audio–visual quality metrics.

Joint coding rate control for audio streaming in short range wireless networks

Streaming high quality audio/video (AV) from home media sources to TV sets and audio speakers over a wireless local area network (WLAN) is a challenging problem because of the fluctuating bandwidth caused by interference and fading. Retransmission and data buffering, as common techniques in data transfer over error-prone channels, are not adequate for delay sensitive applications such as high quality audio streaming. Our approach is to adjust the audio data rate dynamically in order to improve the perceptual quality of audio according to wireless channel quality. In this paper we specifically focus on robust audio streaming over short range wireless channels. The major strength of the proposed algorithm is in its ability to match the source and channel coding parameters to the given channel conditions, thus providing better quality and lower latency of audio in home networking.

대역 확장을 통한 MP3 오디오의 음질 향상

In this paper, we investigate methods to enhance the perceptual quality of MP3-coded audios. Based on the high frequency reconstruction method by Liu, in the proposed method, we determine adaptively the starting point of high frequency reconstruction. We also present an improved linear estimation method. For high frequency component generation, we compare two methods. One is a replication of low-frequency components and the other is an insertion of additive white Gaussian noise signals. Through subjective tests, we shall show that the proposed method can improve the perceptual quality of MP3-coded audio.

Prediction of perceived sound quality of hearing aids (algorithms) using perceptual models

This contribution presents an overview of basic approaches for predicting the perceived sound quality of hearing aids and hearing aid algorithms using auditory processing models for hearing impaired. Comparison‐based concepts will be considered in particular. The main characteristic of these concepts is the comparison of internal representations (the outputs of the auditory models) of a test and a reference signal. While this approach is straight‐forward and has proven to be successful for the prediction of sound quality of lossy speech and audio processing systems perceived by normal‐hearing listeners, the requirement of a reference representing the optimal quality can pose a problem in the case of hearing aids and hearing‐impaired listeners. The potential and limitations of comparison‐based approaches will be illustrated by example results from different studies obtained with an extended version of the perceptual audio quality model PEMO‐Q.