Codebook Entries Research Articles

Reparameterizing and dynamically quantizing image features for image generation

For autoregressive image generation, vector-quantized VAEs (VQ-VAEs) quantize image features with discrete codebook entries and reconstruct images from quantized features. However, they treat each codebook entry separately, which causes losses of image details. In this paper, we propose to reparameterize image features with weight vectors to treat all codebook entries as an entity, and present a novel dynamically vector quantized VAE (DVQ-VAE) to quantize reparameterized image features. Specifically, each image feature corresponds to a weight vector and we sum weighted codebook entries to obtain values of image features. In this way, image features can incorporate information from different codebook entries. Additionally, a novel continuous weight regularization loss is proposed to improve the reconstruction of image details. Our method achieves competitive results with prior state-of-the-art works for image generation and extensive experiments are conducted to take a deep insight into our DVQ-VAE.

Investigations on the Optimal Estimation of Speech Envelopes for the Two-Stage Speech Enhancement.

Using the source-filter model of speech production, clean speech signals can be decomposed into an excitation component and an envelope component that is related to the phoneme being uttered. Therefore, restoring the envelope of degraded speech during speech enhancement can improve the intelligibility and quality of output. As the number of phonemes in spoken speech is limited, they can be adequately represented by a correspondingly limited number of envelopes. This can be exploited to improve the estimation of speech envelopes from a degraded signal in a data-driven manner. The improved envelopes are then used in a second stage to refine the final speech estimate. Envelopes are typically derived from the linear prediction coefficients (LPCs) or from the cepstral coefficients (CCs). The improved envelope is obtained either by mapping the degraded envelope onto pre-trained codebooks (classification approach) or by directly estimating it from the degraded envelope (regression approach). In this work, we first investigate the optimal features for envelope representation and codebook generation by a series of oracle tests. We demonstrate that CCs provide better envelope representation compared to using the LPCs. Further, we demonstrate that a unified speech codebook is advantageous compared to the typical codebook that manually splits speech and silence as separate entries. Next, we investigate low-complexity neural network architectures to map degraded envelopes to the optimal codebook entry in practical systems. We confirm that simple recurrent neural networks yield good performance with a low complexity and number of parameters. We also demonstrate that with a careful choice of the feature and architecture, a regression approach can further improve the performance at a lower computational cost. However, as also seen from the oracle tests, the benefit of the two-stage framework is now chiefly limited by the statistical noise floor estimate, leading to only a limited improvement in extremely adverse conditions. This highlights the need for further research on joint estimation of speech and noise for optimum enhancement.

Experimental study of a compressive line sensing imaging system in a turbulent environment.

Turbulence poses challenges in many atmospheric and underwater surveillance applications. The compressive line sensing (CLS) active imaging scheme has been demonstrated in simulations and test tank experiments to be effective in scattering media such as turbid coastal water, fog, and mist. The CLS sensing model adopts the distributed compressive sensing theoretical framework that exploits both intrasignal sparsity and the highly correlated nature of adjacent areas in a natural scene. During sensing operation, the laser illuminates the spatial light modulator digital micromirror device to generate a series of one-dimensional binary sensing patterns from a codebook to encode the current target line segment. A single element detector photomultiplier tube acquires target reflections as the encoder output. The target can then be recovered using the encoder output and a predicted on-target codebook that reflects the environmental interference of original codebook entries. In this work, we investigated the effectiveness of the CLS imaging system in a turbulent environment. The development of a compact CLS prototype will be discussed, as will a series of experiments using various turbulence intensities at the Naval Research Lab's Simulated Turbulence and Turbidity Environment. The experimental results showed that the time-averaged measurements improved both the signal-to-noise radio and the resolution of the reconstructed image in the extreme turbulence environment. The contributing factors for this intriguing and promising result will be discussed.

Speech enhancement based on AR model parameters estimation

In this paper, we propose a speech and noise auto-regressive (AR) model parameters estimation method under noisy conditions used for speech enhancement, which exploits a priori information about speech and noise spectral shapes (parameterized as AR coefficients) described by trained codebooks. The expectation maximization (EM) algorithm is first employed to obtain AR gains of speech and noise, which correspond to each pair of codebook entries of speech and noise spectral shapes. Then the K-nearest neighbor (KNN) rule is used to select some candidates from the optimized AR parameters (AR coefficients and AR gains) of speech and noise for constructing the weighted Wiener filter (WWF). Furthermore, by using sigmoid function, we propose a posteriori speech-presence probability (SPP) estimation method. Combining the a posteriori SPP with the WWF, the residual noise of enhanced speech is effectively reduced. The test results demonstrate the performance superiority of the proposed speech enhancement scheme compared to the reference methods.

Outage minimization for parallel fading channels with limited feedback

We address an optimal power allocation problem for minimizing the outage probability for M parallel block-Nakagami-fading channels under a long-term average sum transmit power constraint with finite rate feedback of channel state information (CSI). A simulation-based optimization technique called simultaneous perturbation stochastic approximation algorithm (SPSA) is employed first to numerically derive a locally optimal power codebook. Due to the high computational complexity and long convergence time of SPSA, we make an ordering assumption on the power codebook entries and derive effective hyperplane based approximations to the channel quantization regions and present a number of low-complexity suboptimal quantized power codebook design algorithms. Unlike existing work on outage minimization for multiple-input multiple-output (MIMO) channels with limited feedback, we do not assume that identical transmission power is used for all channels within each channel quantization region. We also do not resort to a Gaussian approximation for the instantaneous mutual information in general as used in many existing work. Based on our power ordering assumption and hyperplane based approximations, we show that allocating identical power to all channels within a given channel quantization region in the limited feedback scenario is asymptotically optimal only at high average power (or average signal-to-noise ratio (SNR)) for the Rayleigh fading case, whereas for the general Nakagami case, the transmit power allocation for an individual channel within each quantized region is asymptotically proportional to the corresponding Nakagami fading parameter (severity of fading). We also present a novel diversity order result for the outage probability for the Nakagami fading case. Finally, we derive a suitable Gaussian approximation based low-complexity power allocation scheme for a large number of parallel channels, which has important applications in wideband slow-fading orthogonal frequency-division multiplexing (OFDM) systems. Extensive numerical results illustrate that only a few bits of feedback close the gap substantially in outage performance between the limited feedback case and the full instantaneous CSI at the transmitter case.

A pitch synchronous approach to design voice conversion system using source-filter correlation

We propose a pitch synchronous approach to design the voice conversion system taking into account the correlation between the excitation signal and vocal tract system characteristics of speech production mechanism. The glottal closure instants (GCIs) also known as epochs are used as anchor points for analysis and synthesis of the speech signal. The Gaussian mixture model (GMM) is considered to be the state-of-art method for vocal tract modification in a voice conversion framework. However, the GMM based models generate overly-smooth utterances and need to be tuned according to the amount of available training data. In this paper, we propose the support vector machine multi-regressor (M-SVR) based model that requires less tuning parameters to capture a mapping function between the vocal tract characteristics of the source and the target speaker. The prosodic features are modified using epoch based method and compared with the baseline pitch synchronous overlap and add (PSOLA) based method for pitch and time scale modification. The linear prediction residual (LP residual) signal corresponding to each frame of the converted vocal tract transfer function is selected from the target residual codebook using a modified cost function. The cost function is calculated based on mapped vocal tract transfer function and its dynamics along with minimum residual phase, pitch period and energy differences with the codebook entries. The LP residual signal corresponding to the target speaker is generated by concatenating the selected frame and its previous frame so as to retain the maximum information around the GCIs. The proposed system is also tested using GMM based model for vocal tract modification. The average mean opinion score (MOS) and ABX test results are 3.95 and 85 for GMM based system and 3.98 and 86 for the M-SVR based system respectively. The subjective and objective evaluation results suggest that the proposed M-SVR based model for vocal tract modification combined with modified residual selection and epoch based model for prosody modification can provide a good quality synthesized target output. The results also suggest that the proposed integrated system performs slightly better than the GMM based baseline system designed using either epoch based or PSOLA based model for prosody modification.

Robust Precoding With Limited Feedback Design Based on Precoding MSE for MU-MISO Systems

For the separation of the signals in the vector broadcast channel (BC), some information about the channel state is necessary at the transmitter. In many cases, this channel state information (CSI) must be fed back from the receivers to the transmitter. We jointly design the channel estimators and the quantizers at the receivers together with the precoder at the transmitter based on a precoder-centric criterion, i.e., the minimization of a mean square error (MSE) metric appropriate for the precoder design. This is in contrast to our previous works, where the quantizer design was based on a CSI MSE metric, i.e., based on the minimization of the MSE between the true channel and the channel recovered by the transmitter using a feedback channel. Interestingly, the estimators resulting from this joint formulation are independent of the used codebook. The codebook entries are the employed precoders. Therefore, each receiver feeds back the index of a set of precoders and the intersection of the sets gives the appropriate precoder. Since the quantizers of the different receivers have to work separately, the metric for the computation of the partition cells cannot be expressed as a simple squared error depending on the quantizer output. The proposed system based on a joint optimization clearly outperforms previous designs with separate optimization of feedback and precoding.

Model-based speech enhancement using a bone-conducted signal

Codebook-based single-microphone noise suppressors, which exploit prior knowledge about speech and noise statistics, provide better performance in nonstationary noise. However, as the enhancement involves a joint optimization over speech and noise codebooks, this results in high computational complexity. A codebook-based method is proposed that uses a reference signal observed by a bone-conduction microphone, and a mapping between air- and bone-conduction codebook entries generated during an offline training phase. A smaller subset of air-conducted speech codebook entries that accurately models the clean speech signal is selected using this reference signal. Experiments support the expected improvement in performance at low computational complexity.

Progressive Refinement of Beamforming Vectors for High-Resolution Limited Feedback

Limited feedback enables the practical use of channel state information in multiuser multiple-input multiple-output (MIMO) wireless communication systems. Using the limited feedback concept, channel state information at the receiver is quantized by choosing a representative element from a codebook known to both the receiver and transmitter. Unfortunately, achieving the high resolution required with multiuser MIMO communication is challenging due to the large number of codebook entries required. This paper proposes to use a progressively scaled local codebook to enable high resolution quantization and reconstruction for multiuser MIMO with zero-forcing precoding. Several local codebook designs are proposed including one based on a ring and one based on mutually unbiased bases; both facilitate efficient implementation. Structure in the local codebooks is used to reduce search complexity in the progressive refinement algorithm. Simulation results illustrate sum rate performance as a function of the number of refinements.

Codebook driven short-term predictor parameter estimation for speech enhancement

In this paper, we present a new technique for the estimation of short-term linear predictive parameters of speech and noise from noisy data and their subsequent use in waveform enhancement schemes. The method exploits a priori information about speech and noise spectral shapes stored in trained codebooks, parameterized as linear predictive coefficients. The method also uses information about noise statistics estimated from the noisy observation. Maximum-likelihood estimates of the speech and noise short-term predictor parameters are obtained by searching for the combination of codebook entries that optimizes the likelihood. The estimation involves the computation of the excitation variances of the speech and noise auto-regressive models on a frame-by-frame basis, using the a priori information and the noisy observation. The high computational complexity resulting from a full search of the joint speech and noise codebooks is avoided through an iterative optimization procedure. We introduce a classified noise codebook scheme that uses different noise codebooks for different noise types. Experimental results show that the use of a priori information and the calculation of the instantaneous speech and noise excitation variances on a frame-by-frame basis result in good performance in both stationary and nonstationary noise conditions.

Signature Optimization for CDMA With Limited Feedback

We study the performance of joint signature-receiver optimization for direct-sequence code-division multiple access (DS-CDMA) with limited feedback. The receiver for a particular user selects the signature from a signature codebook, and relays the corresponding B index bits to the transmitter over a noiseless channel. We study the performance of a random vector quantization (RVQ) scheme in which the codebook entries are independent and isotropically distributed. Assuming the interfering signatures are independent, and have independent and identically distributed (i.i.d.) elements, we evaluate the received signal-to-interference plus noise ratio (SINR) in the large system limit as the number of users, processing gain, and feedback bits B all tend to infinity with fixed ratios. This SINR is evaluated for both the matched filter and linear minimum mean-squared error (MMSE) receivers. Furthermore, we show that this large system SINR is the maximum that can be achieved over any sequence of codebooks. Numerical results show that with the MMSE receiver, one feedback bit per signature coefficient achieves close to single-user performance. We also consider a less complex and suboptimal reduced-rank signature optimization scheme in which the user's signature is constrained to lie in a lower dimensional subspace. The optimal subspace coefficients are scalar-quantized and relayed to the transmitter. The large system performance of the quantized reduced-rank scheme can be approximated, and numerical results show that it performs in the vicinity of the RVQ bound. Finally, we extend our analysis to the scenario in which a subset of users optimize their signatures in the presence of random interference.

Codebook based face point trajectory synthesis algorithm using speech input

This paper presents a novel algorithm which generates three-dimensional face point trajectories for a given speech file with or without its text. The proposed algorithm first employs an off-line training phase. In this phase, recorded face point trajectories along with their speech data and phonetic labels are used to generate phonetic codebooks. These codebooks consist of both acoustic and visual features. Acoustics are represented by line spectral frequencies (LSF), and face points are represented with their principal components (PC). During the synthesis stage, speech input is rated in terms of its similarity to the codebook entries. Based on the similarity, each codebook entry is assigned a weighting coefficient. If the phonetic information about the test speech is available, this is utilized in restricting the codebook search to only several codebook entries which are visually closest to the current phoneme (a visual phoneme similarity matrix is generated for this purpose). Then these weights are used to synthesize the principal components of the face point trajectory. The performance of the algorithm is tested on held-out data, and the synthesized face point trajectories showed a correlation of 0.73 with true face point trajectories.

A neural synaptic compression codec for efficient image transmission

For efficient cellular communication channel usage, we propose a neural computation model for image coding. In a constant-time unsupervised learning, our neural model approximates optimal pattern clustering from training example images through a memory adaptation process, and builds a compression codebook in its synaptic weight matrix. This neural codebook can be distributed to both ends of a transmission channel for fast codec operations on general images. The transmission is merely the indices of the codebook entries best matching the patterns in the image to be transmitted. These indices can further be compressed through a classical entropy coding method to yield even more transmission reduction. Other advantages of our model are the low training time complexity, high utilization of neurons, robust pattern clustering capability, and simple computation. A VLSI implementation is also highly suitable for the intrinsic parallel nature of neural networks. Our compression results are competitive compared to JPEG and wavelet methods. We also reveal the general codebook's cross-compression results, filtering effects by special training methods, and learning enhancement techniques for obtaining a compact codebook to yield both high compression and picture quality.

Online adaptive vector quantization with variable size codebook entries

We present a new image compression algorithm that employs some of the most successful approaches to adaptive lossless compression to perform adaptive online (single pass) vector quantization with variable size codebook entries. We have tested this algorithm with a host of standard test images (e.g., gray scale magazine images, medical images, space and scientific images, fingerprint images, and handwriting images) and with no prior knowledge of the data or training, for a given fidelity, the compression achieved typically equals or exceeds that of the JPEG standard. The only information that this algorithm needs in advance is the fidelity criterion. In addition, our algorithm is more flexible than standard VQ and provides tighter guarantees on fidelity.

Robust vector quantization for noisy channels using soft decision and sequential decoding

AbstractThis paper considers the optimization of a vector quantizer whose indices are transmitted over a channel with known noise statistics (pdf). A decoder is introduced which uses a linear combination of VQ codebook entries with weights depending on the received signal. Rather than using a received index to select an entry in the codebook (which implies a hard decision on the index), the decoder uses the received signal and the channel statistics to weight the codebook entries in order to produce a good reconstructed signal. Borrowing a term used in digital communications, we will say that the optimal decoder uses a “soft” decision rule. The corresponding approach will be called Soft Decision Vector Quantization (SDVQ). It is shown that SDVQ achieves a significant performance improvement over both the source optimized VQ and the channel optimized VQ (the latter is based on known channel transition probabilities). For correlated sources, it is shown that SDVQ performance can be further improved by using sequential decoding. Results are presented for an AR(1) source and for speech line spectral pair (LSP) parameter quantization at 24 bits/frame.

Improved techniques for single-pass adaptive vector quantization

Constantinescu and Storer presented a new single-pass adaptive vector quantization algorithm that learns a codebook of variable size and shape entries; they presented experiments on a set of test images showing that with no training or prior knowledge of the data, for a given fidelity, the compression achieved typically equals or exceeds that of the JPEG standard. This paper presents improvements in speed (by employing K-D trees), simplicity of codebook entries, and visual quality with no loss in either the amount of compression or the SNR as compared to the original full-search version.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Method of an apparatus for generating auxiliary information for expediting sparse codebook search

In many applications involving the coding and processing of speech signals the relevant applicable codebook is one which may be termed a sparse codebook. That is, the majority of elements in the codebook are zero valued. The searching of such a sparse codebook is accelerated in accord with the present invention by generating auxiliary information defining the sparse nature of the codebok and using this information to assist and speed up searches of the codebook. In a particular method of searching the calculation of the distance between a target vector and a stored codebook vector is enhanced by use of a distortion metric derived from energy terms and correlation terms of the codebook entries. Calculation of these energy and correlation terms is speeded up by exploiting the sparseness of the codebook entries. The non-zero elements (NZE) of the space codebook are each identified and are defined by their offset from a reference point.

Speech coding using sparse vector codebook and cyclic shift techniques

First Page

Improving performance of code excited LPC-coders by joint optimization

This article describes a CELP speech coding algorithm where the coder parameters are jointly optimized. For that, the relation between pitch period, pitch predictor coefficient, codebook entry and scaling factor is derived. Different approaches to solve this optimization problem with low computational load are discussed. It is shown that the length of the excitation codebook can be reduced significantly compared to the sequential optimization of the coder parameters. The influence of the parameter quantization is examined and an estimation of the computational load is given.

Unsupervised speaker adaptation based on hierarchical spectral clustering

The author proposes an automatic speaker adaptation algorithm for speech recognition, in which a small amount of training material of unspecified text can be used. The algorithm is easily applied to vector-quantization- (VQ) speech recognition systems consisting of a VQ codebook and a word dictionary in which each word is represented as a sequence of codebook entries. In the adaptation algorithm, the VQ codebook is modified for each new speaker, whereas the word dictionary is universally used for all speakers. The important feature of this algorithm is that a set of spectra in training frames and the codebook entries are clustered hierarchically. Based on the vectors representing deviation between centroids of the training frame clusters and the corresponding codebook clusters, adaptation is performed hierarchically from small to large numbers of clusters. The spectral resolution of the adaptation process is improved accordingly. Results of recognition experiments using utterances of 100 Japanese city names show that adaptation reduces the mean word recognition error rate from 4.9 to 2.9%. Since the error rate for speaker-dependent recognition is 2.2%, the adaptation method is highly effective. >