Subjective Listening Tests Research Articles

Non-intrusive method for audio quality assessment of lossy-compressed music recordings using convolutional neural networks

Most of the existing algorithms for the objective audio quality assessment are intrusive, as they require access both to an unimpaired reference recording and an evaluated signal. This feature excludes them from many practical applications. In this paper, we introduce a non-intrusive audio quality assessment method. The proposed method is intended to account for audio artefacts arising from the lossy compression of music signals. During its development, 250 high-quality uncompressed music recordings were collated. They were subsequently processed using the selection of five popular audio codecs, resulting in the repository of 13,000 audio excerpts representing various levels of audio quality. The proposed non-intrusive method was trained with the data obtained employing a well-established intrusive model (ViSQOL v3). Next, the performance of the trained model was evaluated utilizing the quality scores obtained in the subjective listening tests undertaken remotely over the Internet. The listening tests were carried out in compliance with the MUSHRA recommendation (ITU-R BS.1534-3). In this study, the following three convolutional neural networks were compared: (1) a model employing 1D convolutional filters, (2) an Inception-based model, and (3) a VGG-based model. The last-mentioned model outperformed the model employing 1D convolutional filters in terms of predicting the scores from the listening tests, reaching a correlation value of 0.893. The performance of the Inceptionbased model was similar to that of the VGG-based model. Moreover, the VGG-based model outperformed the method employing a stacked gated-recurrent-unit-based deep learning framework, recently introduced by Mumtaz et al. (2022).

Non-Intrusive Speech Quality Assessment Based on Deep Neural Networks for Speech Communication.

Traditionally, speech quality evaluation relies on subjective assessments or intrusive methods that require reference signals or additional equipment. However, over recent years, non-intrusive speech quality assessment has emerged as a promising alternative, capturing much attention from researchers and industry professionals. This article presents a deep learning-based method that exploits large-scale intrusive simulated data to improve the accuracy and generalization of non-intrusive methods. The major contributions of this article are as follows. First, it presents a data simulation method, which generates degraded speech signals and labels their speech quality with the perceptual objective listening quality assessment (POLQA). The generated data is proven to be useful for pretraining the deep learning models. Second, it proposes to apply an adversarial speaker classifier to reduce the impact of speaker-dependent information on speech quality evaluation. Third, an autoencoder-based deep learning scheme is proposed following the principle of representation learning and adversarial training (AT) methods, which is able to transfer the knowledge learned from a large amount of simulated speech data labeled by POLQA. With the help of discriminative representations extracted from the autoencoder, the prediction model can be trained well on a relatively small amount of speech data labeled through subjective listening tests. Fourth, an end-to-end speech quality evaluation neural network is developed, which takes magnitude and phase spectral features as its inputs. This phase-aware model is more accurate than the model using only the magnitude spectral features. A large number of experiments are carried out with three datasets: one simulated with labels obtained using POLQA and two recorded with labels obtained using subjective listening tests. The results show that the presented phase-aware method improves the performance of the baseline model and the proposed model with latent representations extracted from the adversarial autoencoder (AAE) outperforms the state-of-the-art objective quality assessment methods, reducing the root mean square error (RMSE) by 10.5% and 12.2% on the Beijing Institute of Technology (BIT) dataset and Tencent Corpus, respectively. The code and supplementary materials are available at https://github.com/liushenme/AAE-SQA.

Differences in speech intelligibility in noise under Ambisonics-based virtual acoustic environments with varying sound recording/rendering methods

The present study investigates the differences in speech intelligibility in noise under Ambisonics-based virtual acoustic environments realised by various recording/rendering methods. Subjective listening tests were conducted under four different implementations of Ambisonics-based virtual acoustic environments reproducing a seminar room using different modes of recording and rendering techniques. For recording, both first and higher order Ambisonics microphone arrays were utilised while for rendering headphones (binaural) and 32-ch loudspeaker array were used. The results were also compared against the data collected in the real room. The experimental results suggest the speech intelligibility under virtual acoustic environments does not differ significantly regardless of the modes of recording and rendering, however it was generally lower than that in their original room. The effect of spatial release from masking was also observed under the virtual acoustic environments when either/both the higher order Ambisonics microphone array or/and the 32-channel loudspeaker array were utilised for the implementation, however, it was limited to the case when the speech source was located close to the listener.

Examining the voice verification system resistance developed for banking to attacks employing the voice cloning

The developed system for verifying the speaker identity in banking branches is based on voice biometrics employing DeepSpeaker neural network model. The testing of its resistance to a potential attack using voice cloning also employed neural networks, such as SV2TTS, Tacotron, WaveRNN, and GE2E. Subjective listening tests indicated that people might easily get confused and point out cloned recordings as an original sample. Nearly 50% of respondents pointed to the wrong answer, confusing the synthesized recording with the original one. Subjects in most cases declared that the quality of cloned and original recordings is similar (Good or Fair according to ITU-R BS.1534 recommendation) and, in some cases, even better than the original (graded as Good for synthesized sample, Fair for original one). Meanwhile, nearly all verification attempts with cloned samples failed (98.8% of samples were rejected). It proves that voice biometrics based on deep neural networks can identify cloned samples better than human listeners. Methods and results of testing the resistance of the developed voice biometrics system to voice cloning attacks are discussed in thepaper. This research was funded from the budget of project No.POIR.01.01.01-0092/19 subsidized by the Polish National Centre for Research and Development (NCBR).

Mel-S3R: Combining Mel-spectrogram and self-supervised speech representation with VQ-VAE for any-to-any voice conversion

The self-supervised speech representation (S3R) has succeeded in many downstream tasks, such as speaker recognition and voice conversion thanks to its high-level information. Voice conversion (VC) is a task to convert the source speech into a target speaker’s voice. Though S3R features effectively encode content and speaker information, spectral features contain low-level acoustic information that is complementary to the S3R. As a result, solely relying on the S3R features for VC may not be optimal. In order to seek speech representation carrying both high-level learned information and low-level spectral details for VC, we proposed a three-level attention to combine Mel-spectrogram (Mel) and S3R, denoted as Mel-S3R. In particular, S3R features are high-level learned representations extracted by a pre-trained network with self-supervised learning. Whereas Mel is the spectral feature representing the acoustic information. Then the proposed Mel-S3R is used as the input of any-to-any VQ-VAE-based VC and the experiments are performed as a downstream task. Objective metrics and subjective listening tests have demonstrated that the proposed Mel-S3R speech representation facilitates the VC framework to achieve robust performance in terms of both speech quality and speaker similarity.

Method to control the amount of “musical” noise for speech quality assessments

This study presents a method of adding to clean speech signals a controlled degree of “musical” noise distortions that mimic typical artefacts of speech enhancement systems. The resulting distorted speech signals were evaluated with respect to listening effort and sound quality in subjective listening tests and via model predictions. Both subjective ratings and model prediction outcomes covered the entire rating scale from “excellent”/ “no effort” to “bad”/ “extreme effort”, respectively, in a consistent way. The proposed method proved to be useful for systematic assessments of “musical” noise distortions for the conditions tested in this study.

TTS-Guided Training for Accent Conversion Without Parallel Data

Accent Conversion (AC) seeks to change the accent of speech from one (source) to another (target) while preserving the speech content and speaker identity. However, many existing AC approaches rely on source-target parallel speech data during training or reference speech at run-time. We propose a novel accent conversion framework without the need for either parallel data or reference speech. Specifically, a text-to-speech (TTS) system is first pretrained with target-accented speech data. This TTS model and its hidden representations are expected to be associated only with the target accent. Then, a speech encoder is trained to convert the accent of the speech under the supervision of the pretrained TTS model. In doing so, the source-accented speech and its corresponding transcription are forwarded to the speech encoder and the pretrained TTS, respectively. The output of the speech encoder is optimized to be the same as the text embedding in the TTS system. At run-time, the speech encoder is combined with the pretrained speech decoder to convert the source-accented speech toward the target. In the experiments, we converted English with two source accents (Chinese/Indian) to the target accent (American/British/Canadian). Both objective metrics and subjective listening tests successfully validate that the proposed approach generates speech samples that are close to the target accent with high speech quality.

EmotionBox: A music-element-driven emotional music generation system based on music psychology.

With the development of deep neural networks, automatic music composition has made great progress. Although emotional music can evoke listeners' different auditory perceptions, only few research studies have focused on generating emotional music. This paper presents EmotionBox -a music-element-driven emotional music generator based on music psychology that is capable of composing music given a specific emotion, while this model does not require a music dataset labeled with emotions as previous methods. In this work, pitch histogram and note density are extracted as features that represent mode and tempo, respectively, to control music emotions. The specific emotions are mapped from these features through Russell's psychology model. The subjective listening tests show that the Emotionbox has a competitive performance in generating different emotional music and significantly better performance in generating music with low arousal emotions, especially peaceful emotion, compared with the emotion-label-based method.

A Comparative Study of Speech Coding Techniques for Electro Larynx Speech Production

Speech coding is a method of earning a tight speech signals representation for efficient storage and efficient transmission over band limited wired or wireless channels. This is usually achieved with acceptable representation and least number of bits without depletion in the perceptual quality. A number of speech coding methods already developed and various speech coding algorithms for speech analysis and synthesis are used. This paper deals with the comparison of selected coding methods for speech signal produced by Electro Larynx (EL) device. The latter is a device used by cancer patients with their vocal laryngeal cords being removed. The used methods are Residual-Excited Linear Prediction (RELP), Code Excited Linear Prediction (CELP), Algebraic Code Excited Linear Predictive (ACELP), Phase Vocoders based on Wavelet Transform (PVWT), Channel Vocoders based on Wavelet Transform (CVWT), and Phase vocoder based on Dual-Tree Rational-Dilation Complex Wavelet Transform (PVDT-RADWT). The aim here is to select the best coding approach based on the quality of the reproduced speech. The signal used in the test is speech signal recorded either directly by normal persons or else produced by EL device. The performance of each method is evaluated using both objective and subjective listening tests. The results indicate that PVWT and ACELP coders perform better than other methods having about 40 dB SNR and 3 PESQ score for EL speech and 75 dB with 3.5 PESQ score for normal speech, respectively.

Ultrasonic Doppler Based Silent Speech Interface Using Perceptual Distance

Moderate performance in terms of intelligibility and naturalness can be obtained using previously established silent speech interface (SSI) methods. Nevertheless, a common problem associated with SSI has involved deficiencies in estimating the spectrum details, which results in synthesized speech signals that are rough, harsh, and unclear. In this study, harmonic enhancement (HE), was used during postprocessing to alleviate this problem by emphasizing the spectral fine structure of speech signals. To improve the subjective quality of synthesized speech, the difference between synthesized and actual speech was established by calculating the distance in the perceptual domains instead of using the conventional mean square error (MSE). Two deep neural networks (DNNs) were employed to separately estimate the speech spectra and the filter coefficients of HE, connected in a cascading manner. The DNNs were trained to incrementally and iteratively minimize both the MSE and the perceptual distance (PD). A feasibility test showed that the perceptual evaluation of speech quality (PESQ) and the short-time objective intelligibility measure (STOI) were improved by 17.8 and 2.9%, respectively, compared with previous methods. Subjective listening tests revealed that the proposed method yielded perceptually preferred results compared with that of the conventional MSE-based method.

A Controllable Multi-Lingual Multi-Speaker Multi-Style Text-to-Speech Synthesis With Multivariate Information Minimization

In this letter, we propose a multivariate information minimization method that disentangles three or more latent representations. We show that control factors can be disentangled by minimizing interactive dependency, which can be expressed as a sum of mutual information upper bound terms. Since the upper bound estimate converges from the early training stage, there is little performance degradation due to auxiliary loss. The proposed technique is applied to train a text-to-speech synthesizer with multi-lingual, multi-speaker, and multi-style corpora. Subjective listening tests validate that the proposed method can improve the synthesizer in terms of quality as well as controllability.

Using machine learning to evaluate the fidelity of heavy equipment acoustic simulations

Audio quality is an important consideration when creating acoustic simulations. However, there has been a longstanding tradeoff in evaluating the quality or fidelity of sound: subjective listening tests are time consuming and expensive, but objective measures often fail to capture the nuances of human perception. The research presented here seeks to address this problem by investigating the use of machine learning to evaluate the fidelity of heavy equipment acoustic simulations. The developed models are designed to classify sounds into one of two categories, based on whether the audio is “natural” sounding or “artificial” sounding. Two distinct datasets are presented. The first, made up of a library of compressed recordings of heavy equipment, is used primarily for developmental purposes. The second, made up of a library of simulated audio clips, is used to test performance for the intended purpose of evaluating simulated audio fidelity. Several common algorithms are compared and various audio features considered in developing the machine learning models. The final model consists of a logistic regression algorithm and uses the input features loudness, sharpness, roughness, fluctuation strength, and Mel-frequency cepstral coefficients. The developed models accurately predict human perceptions of audio fidelity, achieving approximately 98% accuracy for both datasets. The accuracies achieved provide evidence that machine learning models could potentially supplant listening tests, although limitations including the scope of the dataset and the small number of listening test participants necessitate further validation.

Evaluation of digital watermarking on subjective speech quality

New methods of securing the distribution of audio content have been widely deployed in the last twenty years. Their impact on perceptive quality has, however, only been seldomly the subject of recent extensive research. We review digital speech watermarking state of the art and provide subjective testing of watermarked speech samples. Latest speech watermarking techniques are listed, with their specifics and potential for further development. Their current and possible applications are evaluated. Open-source software designed to embed watermarking patterns in audio files is used to produce a set of samples that satisfies the requirements of modern speech-quality subjective assessments. The patchwork algorithm that is coded in the application is mainly considered in this analysis. Different watermark robustness levels are used, which allow determining the threshold of detection to human listeners. The subjective listening tests are conducted following ITU-T P.800 Recommendation, which precisely defines the conditions and requirements for subjective testing. Further analysis tries to determine the effects of noise and various disturbances on watermarked speech’s perceived quality. A threshold of intelligibility is estimated to allow further openings on speech compression techniques with watermarking. The impact of language or social background is evaluated through an additional experiment involving two groups of listeners. Results show significant robustness of the watermarking implementation, retaining both a reasonable net subjective audio quality and security attributes, despite mild levels of distortion and noise. Extended experiments with Chinese listeners open the door to formulate a hypothesis on perception variations with geographical and social backgrounds.

Subjective evaluation of the combining effect between the virtual bass and head related transfer functions

Earphones are commonly equipped with miniature loudspeaker units, which cannot transmit enough power of low-frequency sound. Meanwhile, there is often only one loudspeaker unit employed on each side of the earphone, whereby the multi-channel spatial audio processing cannot be applied. Therefore, the combined usage of the virtual bass (VB) and head-related transfer functions (HRTFs) is necessary for an immersive listening experience with earphones. However, the combining effect of the VB and HRTFs has not been comprehensively reported. The VB is developed based on the missing fundamental effect, providing that the presence of harmonics can be perceived as their fundamental frequency, even if the fundamental frequency is not presented. HRTFs describe the transmission process of a sound propagating from the sound source to human ears. Monaural audio processed by a pair of HRTFs can be perceived by the listener as a sound source located in the direction associated with the HRTFs. This paper carries out subjective listening tests and their results reveal that the harmonics required by the VB should be generated in the same direction as the high-frequency sound. The bass quality is rarely distorted by the presence of HRTFs, but the localization accuracy is occasionally degraded by the VB.

Subjective studies on impact sound in times of a pandemic -- a comparison between a laboratory study and an online listening test

The National Research Council Canada is currently investigating the perceived annoyance due to impact sound in multi-unit residential buildings (MURBs). The first part of a subjective laboratory study on a number of different floor/ceiling assemblies was completed with 26 participants just before the start of the Covid-19 pandemic in 2020. To evaluate the feasibility of carrying out a similar study without in-person attendance, the same stimuli from the laboratory study were used to create an online listening test. The online listening test was created in JavaScript and HTML5 to run on any internet browser. This paper will present the results of the online listening test and compare them to the laboratory study, focusing on the obvious drawbacks of an uncontrolled remote study such as the uncertainty due to the participants' headphones and listening environment. With an expectation that in-person studies will remain difficult to realize in the near future, this contribution provides evidence whether remote subjective listening tests are a viable alternative to controlled laboratory studies for impact sound.

Go Listen: An End-to-End Online Listening Test Platform

Subjective listening tests are routinely conducted by academic researchers and industry professionals to assess the quality of various speech and audio processing algorithms and transmission services. Listening tests often take place in controlled environments for the sake of consistency, but in many cases, listening tests could be undertaken remotely using a suitable web interface. Despite the work of several projects in the past, there is no publicly available, fully hosted listening test platform which allows for easy test creation, deployment and data collection. Here, we present a fully functional end-to-end listening test platform which allows a user to create and share MUSHRA, ACR and A/B tests within minutes. Collected data can then be downloaded in various forms. For users who would prefer to host the system on their own servers, we provide Docker and AWS images for easy installation.

Multi-Metric Optimization Using Generative Adversarial Networks for Near-End Speech Intelligibility Enhancement

The intelligibility of speech severely degrades in the presence of environmental noise and reverberation. In this paper, we propose a novel deep learning based system for modifying the speech signal to increase its intelligibility under the equal-power constraint, i.e., signal power before and after modification must be the same. To achieve this, we use generative adversarial networks (GANs) to obtain time-frequency dependent amplification factors, which are then applied to the input raw speech to reallocate the speech energy. Instead of optimizing only a single, simple metric, we train a deep neural network (DNN) model to simultaneously optimize multiple advanced speech metrics, including both intelligibility- and quality-related ones, which results in notable improvements in performance and robustness. Our system can not only work in non-real-time mode for offline audio playback but also support practical real-time speech applications. Experimental results using both objective measurements and subjective listening tests indicate that the proposed system significantly outperforms state-of-the-art baseline systems under various noisy and reverberant listening conditions.

Using machine learning to evaluate the fidelity of acoustic simulations

There has been a longstanding tradeoff in evaluating the quality or fidelity of sound recordings: subjective listening tests are time consuming and expensive, but objective measures often fail to capture the nuances of human perception. The research presented here seeks to address this problem by investigating the use of machine learning to evaluate the fidelity of acoustic simulations. To begin, we created a dataset of recordings representing varying levels of audio fidelity. Participants listened to each of the recordings and subjectively classified the perceived fidelity. Various audio features were extracted from the recordings, including several psychoacoustic sound quality metrics and other features commonly used in speech recognition/assessment and music genre classification. These features were input to various machine learning algorithms to test which best modeled the human classifications. A logistic regression model was initially determined to be the most advantageous, dependent on using binary classification. Introducing a reference sound, and calculating each feature relative to the reference, significantly improved accuracy.

Noise and acoustic modeling with waveform generator in text-to-speech and neutral speech conversion

This article focuses on developing a system for high-quality synthesized and converted speech by addressing three fundamental principles. Although the noise-like component in the state-of-the-art parametric vocoders (for example, STRAIGHT) is often not accurate enough, a novel analytical approach for modeling unvoiced excitations using a temporal envelope is proposed. Discrete All Pole, Frequency Domain Linear Prediction, Low Pass Filter, and True envelopes are firstly studied and applied to the noise excitation signal in our continuous vocoder. Second, we build a deep learning model based text–to–speech (TTS) which converts written text into human-like speech with a feed-forward and several sequence-to-sequence models (long short-term memory, gated recurrent unit, and hybrid model). Third, a new voice conversion system is proposed using a continuous fundamental frequency to provide accurate time-aligned voiced segments. The results have been evaluated in terms of objective measures and subjective listening tests. Experimental results showed that the proposed models achieved the highest speaker similarity and better quality compared with the other conventional methods.

AMBIQUAL: Towards a Quality Metric for Headphone Rendered Compressed Ambisonic Spatial Audio

Spatial audio is essential for creating a sense of immersion in virtual environments. Efficient encoding methods are required to deliver spatial audio over networks without compromising Quality of Service (QoS). Streaming service providers such as YouTube typically transcode content into various bit rates and need a perceptually relevant audio quality metric to monitor users’ perceived quality and spatial localization accuracy. The aim of the paper is two-fold. First, it is to investigate the effect of Opus codec compression on the quality of spatial audio as perceived by listeners using subjective listening tests. Secondly, it is to introduce AMBIQUAL, a full reference objective metric for spatial audio quality, which derives both listening quality and localization accuracy metrics directly from the B-format Ambisonic audio. We compare AMBIQUAL quality predictions with subjective quality assessments across a variety of audio samples which have been compressed using the Opus 1.2 codec at various bit rates. Listening quality and localization accuracy of first and third-order Ambisonics were evaluated. Several fixed and dynamic audio sources (single and multiple) were used to evaluate localization accuracy. Results show good correlation regarding listening quality and localization accuracy between objective quality scores using AMBIQUAL and subjective scores obtained during listening tests.