Overcomplete Representations Research Articles

Learning Speaker-Specific Characteristics With a Deep Neural Architecture

Speech signals convey various yet mixed information ranging from linguistic to speaker-specific information. However, most of acoustic representations characterize all different kinds of information as whole, which could hinder either a speech or a speaker recognition (SR) system from producing a better performance. In this paper, we propose a novel deep neural architecture (DNA) especially for learning speaker-specific characteristics from mel-frequency cepstral coefficients, an acoustic representation commonly used in both speech recognition and SR, which results in a speaker-specific overcomplete representation. In order to learn intrinsic speaker-specific characteristics, we come up with an objective function consisting of contrastive losses in terms of speaker similarity/dissimilarity and data reconstruction losses used as regularization to normalize the interference of non-speaker-related information. Moreover, we employ a hybrid learning strategy for learning parameters of the deep neural networks: i.e., local yet greedy layerwise unsupervised pretraining for initialization and global supervised learning for the ultimate discriminative goal. With four Linguistic Data Consortium (LDC) benchmarks and two non-English corpora, we demonstrate that our overcomplete representation is robust in characterizing various speakers, no matter whether their utterances have been used in training our DNA, and highly insensitive to text and languages spoken. Extensive comparative studies suggest that our approach yields favorite results in speaker verification and segmentation. Finally, we discuss several issues concerning our proposed approach.

On the Stable Recovery of the Sparsest Overcomplete Representations in Presence of Noise

Let x be a signal to be sparsely decomposed over a redundant dictionary A, i.e., a sparse coefficient vector s has to be found such that x = As. It is known that this problem is inherently unstable against noise, and to overcome this instability, Donoho, Elad and Temlyakov ["Stable recovery of sparse overcomplete representations in the presence of noise," IEEE Trans. Inf. Theory, vol. 52, no. 1, pp. 6-18, Jan. 2006] have proposed to use an "approximate" decomposition, that is, a decomposition satisfying ∥x - As∥ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> ≤ δ rather than satisfying the exact equality x = As. Then, they have shown that if there is a decomposition with ∥s∥ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sub> <;; (1 + M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-1</sup> )/2, where M denotes the coherence of the dictionary, this decomposition would be stable against noise. On the other hand, it is known that a sparse decomposition with ∥s∥ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sub> <;; (1/2)spark(A) is unique. In other words, although a decomposition with ∥s∥ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sub> <;; (1/2)spark(A) is unique, its stability against noise has been proved only for highly more restrictive decompositions satisfying ∥s∥ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sub> <;; (1 + M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-1</sup> )/2, because usually (1 + M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-1</sup> )/2 ≪ (1/2)spark(A). This limitation maybe had not been very important before, because ∥s∥ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sub> <;; (1 + M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-1</sup> )/2 is also the bound which guaranties that the sparse decomposition can be found via minimizing the l <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> norm, a classic approach for sparse decomposition. However, with the availability of new algorithms for sparse decomposition, namely SL0 and robust-SLO, it would be important to know whether or not unique sparse decompositions with (1 + M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-1</sup> )/2 ≤ ∥s∥ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sub> <;; (1/2)spark(A) are stable. In this correspondence, we show that such decompositions are indeed stable. In other words, we extend the stability bound from ∥s∥ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sub> <;; (1 + M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-1</sup> )/2 to the whole uniqueness range ∥s∥0 <;; (1/2)spark(A). In summary, we show that all unique sparse decompositions are stably recoverable. Moreover, we see that sparser decompositions are "more stable".

Face Recognition with Quantum Associative Networks Using Overcomplete Gabor Wavelet

Gabor wavelet is considered the best mathematical descriptor for receptive fields in the striate cortex. Besides, as a basis function, it is suitable to sparsely represent natural scenes due to its property in maximizing information. It is argued that Gabor-like receptive fields are emerged by sparseness-enforcing or infomax method. In this paper, we incorporate Gabor overcomplete representation into quantum holography for image recognition tasks, with suggestions in improvements through iterative method for reconstruction.

Seislet transform and seislet frame

We introduce a digital waveletlike transform, which is tailored specifically for representing seismic data. The transform provides a multiscale orthogonal basis with basis functions aligned along seismic events in the input data. It is defined with the help of the wavelet-lifting scheme combined with local plane-wave destruction. In the 1D case, the seislet transform is designed to follow locally sinusoidal components. In the 2D case, it is designed to follow local plane-wave components with smoothly variable slopes. If more than one component is present, the transform turns into an overcomplete representation or a tight frame. In these terms, the classic digital wavelet transform is simply a seislet transform for a zero frequency (in one dimension) or zero slope (in two dimensions). The main objective of the new transform is an effective seismic-data compression for designing efficient data-analysis algorithms. Traditional signal-processing tasks such as noise attenuation and trace interpolation become simply defined in the seislet domain. When applied in the offset direction on common-midpoint or common-image-point gathers, the seislet transform finds an additional application in optimal stacking of seismic records.

Application of Gabor Wavelet in Quantum Holography for Image Recognition

Gabor wavelet is considered the best mathematical descriptor for receptive fields in the striate cortex. As a basis function, it is suitable to sparsely represent natural scenes due to its property in maximizing information. It is argued that Gabor-like receptive fields emerged by the sparseness-enforcing or infomax method, with sparseness-enforcing being more biologically plausible. This paper incorporates Gabor over-complete representation into Quantum Holography for image recognition tasks. Correlations are performed using sampled result from all frequencies as well as the optimum frequency. Correlation is also performed using only those points of least activity, which shows improvements in recognition. Analysis on the use of conjugation in reconstruction is provided. The authors also suggest improvements through iterative methods for reconstruction.

All bits are welcome: a coherent model for synergistic generalization and discrimination

Event Abstract Back to Event All bits are welcome: a coherent model for synergistic generalization and discrimination András Lőrincz1* and Gábor Szirtes1 1 Eötvös Loránd University, Software Technology and Methodology, Hungary In modeling neural information processing, the ability to generalize and to discriminate are two desirable features, yet they seem to be in conflict. Based on recent results in signal processing we propose a two-stage representational model that learns to decompose the observations into a typical part serving generalization and a sparse form using an overcomplete dictionary. The latter form is useful in grasping peculiarities in a concise way, but even small perturbations can result in large variations. When these two representational forms are tuned in concert then both become more powerful: generalization gets more robust against outliers, while sparse coding can develop increasingly sparse overcomplete representations less affected by noise. On natural image patches our model simultaneously develops Difference-of-Gaussian like basis as preprocessing for the representation of the typicalities and Gabor filter like basis for representing sparse features of the inputs. In turn, the proposed model may provide a simple and coherent account on the hierarchical arrangement of the spatial receptive fields at the first stages of early visual processing (that is in the retina, LGN and primary visual cortex). The iterative algorithm can be implemented in a generative neural network form and it can be seen as a generalization and unification of other attractive proposals, like infomax, slow-feature analysis and sparse coding. We argue that the principles behind the proposed algorithm make it a central component in diverse hierarchical computations performed by neural systems. Keywords: Information Processing*, Learning, Memory, modelling, sensory processing Conference: Bernstein Conference on Computational Neuroscience, Berlin, Germany, 27 Sep - 1 Oct, 2010. Presentation Type: Presentation Topic: Bernstein Conference on Computational Neuroscience Citation: Lőrincz A and Szirtes G (2010). All bits are welcome: a coherent model for synergistic generalization and discrimination. Front. Comput. Neurosci. Conference Abstract: Bernstein Conference on Computational Neuroscience. doi: 10.3389/conf.fncom.2010.51.00012 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 01 Sep 2010; Published Online: 22 Sep 2010. * Correspondence: Dr. András Lőrincz, Eötvös Loránd University, Software Technology and Methodology, Budapest, Hungary, andras.lorincz@t-online.hu Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers András Lőrincz Gábor Szirtes Google András Lőrincz Gábor Szirtes Google Scholar András Lőrincz Gábor Szirtes PubMed András Lőrincz Gábor Szirtes Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

Removal of BCG Artifacts Using a Non-Kirchhoffian Overcomplete Representation

We present a nonlinear unmixing approach for extracting the ballistocardiogram (BCG) from EEG recorded in an MR scanner during simultaneous acquisition of functional MRI (fMRI). First, an overcomplete basis is identified in the EEG based on a custom multipath EEG electrode cap. Next, the overcomplete basis is used to infer non-Kirchhoffian latent variables that are not consistent with a conservative electric field. Neural activity is strictly Kirchhoffian while the BCG artifact is not, and the representation can hence be used to remove the artifacts from the data in a way that does not attenuate the neural signals needed for optimal single-trial classification performance. We compare our method to more standard methods for BCG removal, namely independent component analysis and optimal basis sets, by looking at single-trial classification performance for an auditory oddball experiment. We show that our overcomplete representation method for removing BCG artifacts results in better single-trial classification performance compared to the conventional approaches, indicating that the derived neural activity in this representation retains the complex information in the trial-to-trial variability.

Sparse Coding Neural Gas: Learning of overcomplete data representations

We consider the problem of learning an unknown (overcomplete) basis from data that are generated from unknown and sparse linear combinations. Introducing the Sparse Coding Neural Gas algorithm, we show how to employ a combination of the original Neural Gas algorithm and Oja's rule in order to learn a simple sparse code that represents each training sample by only one scaled basis vector. We generalize this algorithm by using Orthogonal Matching Pursuit in order to learn a sparse code where each training sample is represented by a linear combination of up to k basis elements. We evaluate the influence of additive noise and the coherence of the original basis on the performance with respect to the reconstruction of the original basis and compare the new method to other state of the art methods. For this analysis, we use artificial data where the original basis is known. Furthermore, we employ our method to learn an overcomplete representation for natural images and obtain an appealing set of basis functions that resemble the receptive fields of neurons in the primary visual cortex. An important result is that the algorithm converges even with a high degree of overcompleteness. A reference implementation of the methods is provided. 1 1 http://www.inb.uni-luebeck.de/tools-demos/scng

Enhancing Sparsity by Reweighted ℓ 1 Minimization

It is now well understood that (1) it is possible to reconstruct sparse signals exactly from what appear to be highly incomplete sets of linear measurements and (2) that this can be done by constrained l1 minimization. In this paper, we study a novel method for sparse signal recovery that in many situations outperforms l1 minimization in the sense that substantially fewer measurements are needed for exact recovery. The algorithm consists of solving a sequence of weighted l1-minimization problems where the weights used for the next iteration are computed from the value of the current solution. We present a series of experiments demonstrating the remarkable performance and broad applicability of this algorithm in the areas of sparse signal recovery, statistical estimation, error correction and image processing. Interestingly, superior gains are also achieved when our method is applied to recover signals with assumed near-sparsity in overcomplete representations—not by reweighting the l1 norm of the coefficient sequence as is common, but by reweighting the l1 norm of the transformed object. An immediate consequence is the possibility of highly efficient data acquisition protocols by improving on a technique known as Compressive Sensing.

Optimal denoising in redundant representations.

Image denoising methods are often designed to minimize mean-squared error (MSE) within the subbands of a multiscale decomposition. However, most high-quality denoising results have been obtained with overcomplete representations, for which minimization of MSE in the subband domain does not guarantee optimal MSE performance in the image domain. We prove that, despite this suboptimality, the expected image-domain MSE resulting from applying estimators to subbands that are made redundant through spatial replication of basis functions (e.g., cycle spinning) is always less than or equal to that resulting from applying the same estimators to the original nonredundant representation. In addition, we show that it is possible to further exploit overcompleteness by jointly optimizing the subband estimators for image-domain MSE. We develop an extended version of Stein's unbiased risk estimate (SURE) that allows us to perform this optimization adaptively, for each observed noisy image. We demonstrate this methodology using a new class of estimator formed from linear combinations of localized "bump" functions that are applied either pointwise or on local neighborhoods of subband coefficients. We show through simulations that the performance of these estimators applied to overcomplete subbands and optimized for image-domain MSE is substantially better than that obtained when they are optimized within each subband. This performance is, in turn, substantially better than that obtained when they are optimized for use on a nonredundant representation.

Overcomplete topographic independent component analysis

Topographic and overcomplete representations of natural images/videos are important problems in computational neuroscience. We propose a new method using both topographic and overcomplete representations of natural images, showing emergence of properties similar to those of complex cells in primary visual cortex (V1). This method can be considered as an extension of model in Hyvärinen et al. [Topographic independent component analysis, Neural Comput. 13 (7) (2001) 1527–1558], which uses complete topographic representation. We utilize a sparse and approximately uncorrelated decompositions and define a topographic structure on coefficients (the dot products between basis vectors and whitened observed data vectors). The overcomplete topographic basis vectors can be learned via estimation of independent component analysis (ICA) model based on the prior assumption upon basis vectors. Computer simulations are provided to show the relationship between our model and the basic properties of complex cells in V1 cortex. The learned bases are shown to have better coding efficiency than ordinary topographic ICA (TICA) bases.

A Note on Lewicki-Sejnowski Gradient for Learning Overcomplete Representations

Overcomplete representations have greater robustness in noise environment and also have greater flexibility in matching structure in the data. Lewicki and Sejnowski (2000) proposed an efficient extended natural gradient for learning the overcomplete basis and developed an overcomplete representation approach. However, they derived their gradient by many approximations, and their proof is very complicated. To give a stronger theoretical basis, we provide a brief and more rigorous mathematical proof for this gradient in this note. In addition, we propose a more robust constrained Lewicki-Sejnowski gradient.

Visual Recognition and Inference Using Dynamic Overcomplete Sparse Learning

We present a hierarchical architecture and learning algorithm for visual recognition and other visual inference tasks such as imagination, reconstruction of occluded images, and expectation-driven segmentation. Using properties of biological vision for guidance, we posit a stochastic generative world model and from it develop a simplified world model (SWM) based on a tractable variational approximation that is designed to enforce sparse coding. Recent developments in computational methods for learning overcomplete representations (Lewicki & Sejnowski, 2000; Teh, Welling, Osindero, & Hinton, 2003) suggest that overcompleteness can be useful for visual tasks, and we use an overcomplete dictionary learning algorithm (Kreutz-Delgado, et al., 2003) as a preprocessing stage to produce accurate, sparse codings of images. Inference is performed by constructing a dynamic multilayer network with feedforward, feedback, and lateral connections, which is trained to approximate the SWM. Learning is done with a variant of the back-propagation-through-time algorithm, which encourages convergence to desired states within a fixed number of iterations. Vision tasks require large networks, and to make learning efficient, we take advantage of the sparsity of each layer to update only a small subset of elements in a large weight matrix at each iteration. Experiments on a set of rotated objects demonstrate various types of visual inference and show that increasing the degree of overcompleteness improves recognition performance in difficult scenes with occluded objects in clutter.

Robust Coding Over Noisy Overcomplete Channels

We address the problem of robust coding in which the signal information should be preserved in spite of intrinsic noise in the representation. We present a theoretical analysis for 1- and 2-D cases and characterize the optimal linear encoder and decoder in the mean-squared error sense. Our analysis allows for an arbitrary number of coding units, thus including both under- and over-complete representations, and provides insights into optimal coding strategies. In particular, we show how the form of the code adapts to the number of coding units and to different data and noise conditions in order to achieve robustness. We also present numerical solutions of robust coding for high-dimensional image data, demonstrating that these codes are substantially more robust than other linear image coding methods such as PCA, ICA, and wavelets.

ARE SPARSE-CODING SIMPLE CELL RECEPTIVE FIELD MODELS PHYSIOLOGICALLY PLAUSIBLE?

Olshausen and Field (1996) developed a simple cell receptive field model for natural scene processing in V1, based on unsupervised learning and non-orthogonal basis function optimization of an overcomplete representation of visual space. The model was originally tested with an ensemble of whitened natural scenes, simulating pre-cortical filtering in the retinal ganglia and lateral geniculate nucleus, and the basis functions qualitatively resembled the orientation-specific responses of V1 simple cells in the spatial domain. In this study, the quantitative tuning responses of the basis functions in the spectral domain are estimated using a Gaussian model, to determine their goodness-of-fit to the known bandwidths of simple cells in primate V1. Five simulation experiments which examined key features of the model are reported: changing the size of the basis functions; using a complete versus over-complete representation; changing the sparseness factor; using a variable learning rate; and mapping the basis functions with a whitening spatial function. The key finding of this study is that across all image themes, basis function sizes, number of basis functions, sparseness factors and learning rates, the spatial-frequency tuning did not closely resemble that of primate area 17 -- the model results more closely resembled the unclassified cat neurones of area 19 with a single exception, and not area 17 as predicted.

Sparse and structured decompositions of signals with the molecular matching pursuit

This paper describes the Molecular Matching Pursuit (MMP), an extension of the popular Matching Pursuit (MP) algorithm for the decomposition of signals. The MMP is a practical solution which introduces the notion of structures within the framework of sparse overcomplete representations; these structures are based on the local dependency of significant time-frequency or time-scale atoms. We show that this algorithm is well adapted to the representation of real signals such as percussive audio signals. This is at the cost of a slight sub-optimality in terms of the rate of convergence for the approximation error, but the benefits are numerous, most notably a significant reduction in the computational cost, which facilitates the processing of long signals. Results show that this algorithm is very promising for high-quality adaptive coding of audio signals

Frames and Overcomplete Representations in Signal Processing, Communications, and Information Theory

Many problems in signal processing, communications, and information theory deal with linear signal expansions. The corresponding basis functions are typically orthogonal (nonredundant) signal sets. It is well known that the use of redundancy in engineering systems improves robustness and numerical stability. Motivated by this observation, redundant linear signal expansions (also known as “frames”) have found widespread use in many different engineering disciplines. Recent examples include sampling theory, A/D conversion, oversampled filter banks, pattern classification, multiple description source coding, wavelet-based and framebased denoising, and space-time coding for wireless communications. This special issue of EURASIP JASP brings together researchers from areas as diverse as harmonic analysis, image processing, and wireless communications by combining invited papers with regular contributions related to these topics. The papers in this issue are broadly classified into four main areas:

Sparse overcomplete Gabor wavelet representation based on local competitions

Gabor representations present a number of interesting properties despite the fact that the basis functions are nonorthogonal and provide an overcomplete representation or a nonexact reconstruction. Overcompleteness involves an expansion of the number of coefficients in the transform domain and induces a redundancy that can be further reduced through computational costly iterative algorithms like Matching Pursuit. Here, a biologically plausible algorithm based on competitions between neighboring coefficients is employed for adaptively representing any source image by a selected subset of Gabor functions. This scheme involves a sharper edge localization and a significant reduction of the information redundancy, while, at the same time, the reconstruction quality is preserved. The method is characterized by its biological plausibility and promising results, but it still requires a more in depth theoretical analysis for completing its validation.

Sparse Representations Are Most Likely to Be the Sparsest Possible

Given a signal and a full-rank matrix with, we define the signal's overcomplete representations as all satisfying. Among all the possible solutions, we have special interest in the sparsest one—the one minimizing. Previous work has established that a representation is unique if it is sparse enough, requiring. The measure stands for the minimal number of columns from that are linearly dependent. This bound is tight—examples can be constructed to show that with or more nonzero entries, uniqueness is violated. In this paper we study the behavior of overcomplete representations beyond the above bound. While tight from a worst-case standpoint, a probabilistic point of view leads to uniqueness of representations satisfying. Furthermore, we show that even beyond this point, uniqueness can still be claimed with high confidence. This new result is important for the study of the average performance of pursuit algorithms—when trying to show an equivalence between the pursuit result and the ideal solution, one must also guarantee that the ideal result is indeed the sparsest.

Stable recovery of sparse overcomplete representations in the presence of noise

Overcomplete representations are attracting interest in signal processing theory, particularly due to their potential to generate sparse representations of signals. However, in general, the problem of finding sparse representations must be unstable in the presence of noise. This paper establishes the possibility of stable recovery under a combination of sufficient sparsity and favorable structure of the overcomplete system. Considering an ideal underlying signal that has a sufficiently sparse representation, it is assumed that only a noisy version of it can be observed. Assuming further that the overcomplete system is incoherent, it is shown that the optimally sparse approximation to the noisy data differs from the optimally sparse decomposition of the ideal noiseless signal by at most a constant multiple of the noise level. As this optimal-sparsity method requires heavy (combinatorial) computational effort, approximation algorithms are considered. It is shown that similar stability is also available using the basis and the matching pursuit algorithms. Furthermore, it is shown that these methods result in sparse approximation of the noisy data that contains only terms also appearing in the unique sparsest representation of the ideal noiseless sparse signal.