Perceptual Distortion Measure Research Articles

Single image super-resolution via Image Quality Assessment-Guided Deep Learning Network.

In recent years, deep learning (DL) networks have been widely used in super-resolution (SR) and exhibit improved performance. In this paper, an image quality assessment (IQA)-guided single image super-resolution (SISR) method is proposed in DL architecture, in order to achieve a nice tradeoff between perceptual quality and distortion measure of the SR result. Unlike existing DL-based SR algorithms, an IQA net is introduced to extract perception features from SR results, calculate corresponding loss fused with original absolute pixel loss, and guide the adjustment of SR net parameters. To solve the problem of heterogeneous datasets used by IQA and SR networks, an interactive training model is established via cascaded network. We also propose a pairwise ranking hinge loss method to overcome the shortcomings of insufficient samples during training process. The performance comparison between our proposed method with recent SISR methods shows that the former achieves a better tradeoff between perceptual quality and distortion measure than the latter. Extensive benchmark experiments and analyses also prove that our method provides a promising and opening architecture for SISR, which is not confined to a specific network model.

Perceptual orthogonal matching pursuit for speech sparse modelling

The perceptual orthogonal matching pursuit (POMP), a sparse approximation algorithm built upon the known orthogonal matching pursuit (OMP), is introduced. It is designed for speech processing and can be of great use in speech coding applications. It can handle all types of real dictionaries, including predefined and adaptive dictionaries. Being a suboptimal method, POMP performs a series of local updates where it minimises a perceptual distortion measure involving a perceptual weighting filter. This filter is tailored for speech signals and is used in AMR 3GPP coders. Experiments show that POMP outperforms the standard OMP for predefined and adaptive dictionaries.

A multiresolution non-negative tensor factorization approach for single channel sound source separation

We propose a single channel audio source separation method to alleviate the smearing effects caused by fixed time-frequency (TF) resolution Short-Time Fourier Transform (STFT). We introduce a multiresolution representation based on Non-negative Tensor Factorization (NTF) where each layer of the tensor represents the mixture signal at a different time-frequency resolution. In order to fuse the information at different layers, the source separation is modeled as a joint optimization problem where the optimal solution is derived based on the Kullback–Leibler (KL) divergence. The resynthesis is made through an additional adaptive weighted fusion procedure which combines the sources separated at different scales by maximizing energy concentration. Numerical results over a large sound database indicate that the proposed joint optimization scheme enhances the quality of the separated sources both in terms of the conventional and the perceptual distortion measures.

Speech energy redistribution for intelligibility improvement in noise based on a perceptual distortion measure

A speech pre-processing algorithm is presented that improves the speech intelligibility in noise for the near-end listener. The algorithm improves intelligibility by optimally redistributing the speech energy over time and frequency according to a perceptual distortion measure, which is based on a spectro-temporal auditory model. Since this auditory model takes into account short-time information, transients will receive more amplification than stationary vowels, which has been shown to be beneficial for intelligibility of speech in noise. The proposed method is compared to unprocessed speech and two reference methods using an intelligibility listening test. Results show that the proposed method leads to significant intelligibility gains while still preserving quality. Although one of the methods used as a reference obtained higher intelligibility gains, this happened at the cost of decreased quality. Matlab code is provided.

Perceptual Distortion Measure for Polygon-Based Shape Coding

In this paper, we present a perceptual distortion measure (PDM) for polygon-based shape coding. Wemodel the PDM as the salience of relevance triangle, and express the PDM by using three properties derived from the salience of visual part. Performance analysis and experimental results show that our proposal can improve the quality of the shape reconstruction when the object contour has sharp protrusions. key words: video coding, image coding, image quality

A New Technique for Perceptual Distortion Measure on A Spectro Temporal Auditory Model

Speech processing is used widely in every day’s applications that most people take for granted, such as network wire lines, cellular telephony, telephony system and telephone answering machines. Due to its popularity and increasing of demand, engineers are trying various approaches of improving the process. One of the methods for improving is trying on different methods of filtering techniques. Thus, this instigates an introduction of a filtering technique known as Kalman filtering. In the early days, Kalman filtering was very popular in the research field of navigation because of its magnificent accurate estimation characteristic. Since then, electronic engineers manipulate its advantages to useful purpose in speech processing. Consequently, today it had become a popular filtering technique for estimating and resolving redundant errors containing in speech. A speech pre-processing algorithm is presented to improve the speech intelligibility in noise for the near-end listener. The algorithm improves the intelligibility by optimally redistributing the speech energy over time and frequency for a perceptual distortion measure, which is based on a spectro-temporal auditory model. In contrast o spectral-only models, short-time information is taken into account. As a consequence, the algorithm is more sensitive to transient regions, which will therefore receive more amplification compared to stationary vowels. It is known from literature that changing the vowel-transient energy ratio is beneficial for improving speech intelligibility in noise. Objective intelligibility prediction results show that the proposed method has higher speech intelligibility in noise compared to other reference methods, without modifying the global speech energy.

Perceptual video coding based on MB classification and rate–distortion optimization

The efficiency of a video coding scheme should be defined as the video quality achieved for a given bit-rate. The quality refers here to the subjective quality perceived by the final user. This is the only valid criteria, and it involves the user vision properties. Therefore, in order to optimize this perceived video quality, a new perceptual coding scheme is proposed taking into account the Human Visual System (HVS). Perceptual distortion measures are included in the encoding loop to compute an adaptive local quantization step. The idea is that optimizing the choice of macroblock quantization parameters based on a more accurate perceptual distortion measure would result in an improved subjective quality. The resulting macroblock-level rate allocation problem is first modeled as a constrained optimization problem solved with a Lagrangian multiplier-based algorithm. The local adaptation is performed by a classification of each macroblock according to its distortion–quantization properties. A learning strategy applied on a set of sequences provides a set of representative distortion–quantization curves for all macroblock types. Then a rate–quantization model is derived from the ρ-domain linear approximation to reach the target bit-rate. A two-pass only algorithm is required to compute the necessary models and encode each macroblock. The complexity of the proposed solution is thus reduced compared to exhaustive search algorithms. The SSIM (Structural SIMilarity) is used as in-loop distortion measure, and the performances are evaluated with another perceptual measure, more correlated to the user perception, called the WQA (Wavelet Quality Assessment). The coding schemes being based on a biological possible modeling of the Human Visual System, the video includes less visual artifacts and a more uniform subjective quality, compared to traditional rate–distortion coding.

Adaptive Signal Modeling Based on Sparse Approximations for Scalable Parametric Audio Coding

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> This paper deals with the application of adaptive signal models for parametric audio coding. A fully parametric audio coder, which decomposes the audio signal into sinusoids, transients and noise, is here proposed. Adaptive signal models for sinusoidal, transient, and noise modeling are therefore included in the parametric scheme in order to achieve high-quality and low bit-rate audio coding. In this paper, a new sinusoidal modeling method based on a perceptual distortion measure is proposed. For transient modeling, a fast and effective method based on matching pursuit with a mixed dictionary is chosen. The residue of the previous models is analyzed as a noise-like signal. The proposed parametric audio coder allows high quality audio coding for one-channel audio signals at 16 kbits/s (average bit rate). A bit-rate scalable version of the parametric audio coder is also proposed in this work. Bit-rate scalability is intended for audio streaming applications, which are highly demanded nowadays. The performance of the proposed parametric audio coders (nonscalable and scalable coders) is assessed in comparison to widely used audio coders operating at similar bit rates. </para>

Robust Parametric Audio Coding Using Multiple Description Coding

We propose a new multiple description spherical quantization with repetitively coded amplitudes (MDSQRA) scheme suited for quantization of sinusoidal parameters. The quantization scheme is constituted by a set of spherical quantizers inspired by the multiple description spherical trellis-coded quantization (MDSTCQ) scheme. In this scheme, we apply repetitive coding on the amplitudes, while multiple description coding are applied on the phases and frequencies. Thereby, MDSQRA becomes directly implementable, as opposed to MDSTCQ, since the phase and frequency quantizers depend on the amplitudes which have dissimilar descriptions in MDSTCQ. Furthermore, we implement MDSQRA into a perceptual matching pursuit based sinusoidal audio coder. Finally, we evaluate MDSQRA through perceptual distortion measurements and MUSHRA listening tests. The tests show that MDSQRA outperforms MDSTCQ with respect to a expected perceptual distortion measure. The same results are obtained through the MUSHRA tests performed on sound clips coded using MDSQRA and MDSTCQ.

Bark scale-based perceptual matching pursuit for improving sinusoidal audio modeling

In this paper we propose an improved sinusoidal modeling method based on perceptual matching pursuits computed in the bark scale for parametric audio coding applications. Complex exponentials compose the overcomplete dictionary for matching pursuits. The main contribution is the minimization of a perceptual distortion measure defined in the bark scale to select the optimum atom at each iteration of the pursuits. Furthermore, a psychoacoustic stopping criterion for the pursuits is presented. The proposed sinusoidal modeling method is suitable to be integrated into a parametric audio coder based on the three-part model of sines, transients and noise (STN model), as can be appreciated in experimental results. Our method provides significant advantages regarding previous works mainly because it operates in the bark scale rather than in frequency domain.

Multiple Description Trellis-Coded Quantization of Sinusoidal Parameters

A new scheme for sinusoidal audio coding named multiple description spherical trellis-coded quantization is proposed and analytic expressions for the point densities and expected distortion of the quantizers are derived based on a high-resolution assumption. The proposed quantizers are of variable dimensions meaning that any number of sinusoids can be quantized jointly for each audio segment whereby a lower distortion is achieved compared to previously published scalar spherical quantizers. The quantizers are designed to minimize a perceptual distortion measure subject to an entropy constraint for a given packet-loss probability. In experiments, the performance of the quantizers is assessed and compared to the corresponding single description spherical quantizer and associated bounds under various conditions and is found to increase robustness towards packet-loss.

Design of Linear Equalizers Optimized for the Structural Similarity Index

We propose an algorithm for designing linear equalizers that maximize the structural similarity (SSIM) index between the reference and restored signals. The SSIM index has enjoyed considerable application in the evaluation of image processing algorithms. Algorithms, however, have not been designed yet to explicitly optimize for this measure. The design of such an algorithm is nontrivial due to the nonconvex nature of the distortion measure. In this paper, we reformulate the nonconvex problem as a quasi-convex optimization problem, which admits a tractable solution. We compute the optimal solution in near closed form, with complexity of the resulting algorithm comparable to complexity of the linear minimum mean squared error (MMSE) solution, independent of the number of filter taps. To demonstrate the usefulness of the proposed algorithm, it is applied to restore images that have been blurred and corrupted with additive white gaussian noise. As a special case, we consider blur-free image denoising. In each case, its performance is compared to a locally adaptive linear MSE-optimal filter. We show that the images denoised and restored using the SSIM-optimal filter have higher SSIM index, and superior perceptual quality than those restored using the MSE-optimal adaptive linear filter. Through these results, we demonstrate that a) designing image processing algorithms, and, in particular, denoising and restoration-type algorithms, can yield significant gains over existing (in particular, linear MMSE-based) algorithms by optimizing them for perceptual distortion measures, and b) these gains may be obtained without significant increase in the computational complexity of the algorithm.

On perceptual distortion measures and parametric modeling

Over the past two decades, there has been much interest in incorporating human sound perception in signal processing algorithms, often in the form of perceptual distortion measure or an approximation thereof. An example of this is MDCT‐based audio coding where a good quality can be achieved at very low bit‐rates by taking masking effects into account. More recently, the same principles have been applied to parametric modeling and coding of audio signals. We discuss the inherent tradeoffs in choosing a perceptual distortion measure and a parametric model and the pros and cons of various ways of implementing such perceptual distortion measures is discussed. An important question that we seek to answer is whether perception should be taken into account in the estimation of model parameters or this should be done in a separate step.

High-Resolution Spherical Quantization of Sinusoidal Parameters

Sinusoidal coding is an often employed technique in low bit-rate audio coding. Therefore, methods for efficient quantization of sinusoidal parameters are of great importance. In this paper, we use high-resolution assumptions to derive analytical expressions for the optimal entropy-constrained unrestricted spherical quantizers for the amplitude, phase, and frequency parameters of the sinusoidal model. This is done both for the case of a single sinusoid, and for the more practically relevant case of multiple sinusoids distributed across multiple segments. To account for psychoacoustical effects of the auditory system, a perceptual distortion measure is used. The optimal quantizers minimize a high-resolution approximation of the expected perceptual distortion, while the corresponding quantization indices satisfy an entropy constraint. The quantizers turn out to be flexible and of low complexity, in the sense that they can be determined easily for varying bit rate requirements, without any sort of retraining or iterative procedures. In an objective comparison it is shown that for the squared error distortion measure, the rate-distortion performance of the proposed method is very close to that of the theoretically optimal entropy-constrained vector quantization. Furthermore, for the perceptual distortion measure, the proposed scheme is shown to objectively outperform an existing sinusoidal quantization scheme, where frequency quantization is done independently. Finally, a subjective listening test, in which the proposed scheme is compared to an existing state-of-the-art sinusoidal quantization scheme with fixed quantizers for all input signals, indicates that the proposed scheme leads to an average bit rate reduction of 20%, at the same subjective quality level as the existing scheme

The Sensitivity Matrix: Using Advanced Auditory Models in Speech and Audio Processing

Perceptually optimal processing of speech and audio signals demands a rigorous approach using a distortion measure that resembles human perception. This requires distortion measures based on sophisticated, complex auditory models. Under the assumption of small distortions these models can be simplified by means of a sensitivity matrix. In this paper, we show the power of this approach. We present a method to derive the sensitivity matrix for distortion measures based on spectro-temporal auditory models. This method is applied to an example auditory model and the region of validity of the approximation and the application of linear algebra to analyze the characteristics of the given model are discussed. Furthermore, we show how to build a coder minimizing a sensitivity matrix distortion measure given the typically long support of a perceptual distortion measure

Resolution-Constrained Quantization With JND-Based Perceptual-Distortion Measures

When the squared error of observable signal parameters is below the just noticeable difference (JND), it is not registered by human perception. We modify commonly used distortion criteria to account for this phenomenon and study the implications for quantizer design and performance. Taking the JND into account in the design of the quantizer generally leads to improved performance in terms of mean distortion and the number of outliers. Moreover, the resulting quantizer exhibits better robustness against source mismatch

Efficient parametric coding of transients

In this paper, methods for improved parametric coding of transients are presented. We propose a signal model for coding of transients consisting of a sum of sinusoids each being amplitude-modulated by a different gamma envelope. These envelopes are characterized by an onset time, an attack and a decay parameter. An efficient method for estimating these parameters is presented. Further, methods are proposed that combine this transient model with a constant-amplitude sinusoidal model in order to achieve efficient coding of both stationary and transient signal parts. By rate-distortion optimization using a perceptual distortion measure, we combine variable rate bit allocation and segmentation in an optimal way. Formal, as well as informal, listening tests show that significant improvements can be achieved with the proposed model as compared to a state-of-the-art sinusoidal coder by the combination of optimal segmentation and amplitude modulated sinusoidal audio coding

Amplitude modulated sinusoidal signal decomposition for audio coding

In this letter, we present a decomposition for sinusoidal coding of audio, based on an amplitude modulation of sinusoids via a linear combination of arbitrary basis vectors. The proposed method, which incorporates a perceptual distortion measure, is based on a relaxation of a nonlinear least-squares minimization. Rate-distortion curves and listening tests show that, compared to a constant-amplitude sinusoidal coder, the proposed decomposition offers perceptually significant improvements in critical transient signals

On perceptual distortion minimization and nonlinear least-squares frequency estimation

In this paper, we present a framework for perceptual error minimization and sinusoidal frequency estimation based on a new perceptual distortion measure, and we state its optimal solution. Using this framework, we relate a number of well-known practical methods for perceptual sinusoidal parameter estimation such as the prefiltering method, the weighted matching pursuit, and the perceptual matching pursuit. In particular, we derive and compare the sinusoidal estimation criteria used in these methods. We show that for the sinusoidal estimation problem, the prefiltering method and the weighted matching pursuit are equivalent to the perceptual matching pursuit under certain conditions.

Rate-distortion optimized quantization in multistage audio coding

In this work, we develop a new method for quantization in multistage audio coding. Given a (perceptual) distortion measure and a bit-rate constraint, we analytically derive the optimal rate distribution between subcoders (stages) and the corresponding optimal quantizers using high-rate theory. The analytical solutions for optimal quantizers allow a coder to easily adapt to changes in bit-rate requirements. As an illustration of the new method, we consider quantization in a two-stage sinusoidal/waveform coder that is a widely used combination in audio coding. We show that at low total rates most of the rate should be assigned to the sinusoidal (model-based, subspace) subcoder, while at high total rates most of the rate should be assigned to the waveform (full-space) subcoder. We compare the new method to a reference quantization method that does not use rate-distortion optimization. A significantly higher performance of the new method is shown by means of a listening test.