High-noise Regime Research Articles

Denoising Drug Discovery Data for Improved Absorption, Distribution, Metabolism, Excretion, and Toxicity Property Prediction.

Predicting absorption, distribution, metabolism, excretion, and toxicity (ADMET) properties of small molecules is a key task in drug discovery. A major challenge in building better ADMET models is the experimental error inherent in the data. Furthermore, ADMET predictors are typically regression tasks due to the continuous nature of the data, which makes it difficult to apply existing denoising methods from other domains as they largely focus on classification tasks. Here, we develop denoising schemes based on deep learning to address this. We find that the training error (TE) can be used to identify the noise in regression tasks while ensemble-based and forgotten event-based metrics fail to detect the noise. The most significant performance increase occurs when the original model is finetuned with the denoised data using TE as the noise detection metric. Our method has the ability to improve models with medium noise and does not degrade the performance of models with noise outside this range (low noise and high noise regimes). To our knowledge, our denoising scheme is the first to improve model performance for ADMET data and has implications for improving models for experimental assay data in general.

Purifying Photon Indistinguishability through Quantum Interference.

Indistinguishability between photons is a key requirement for scalable photonic quantum technologies. We experimentally demonstrate that partly distinguishable single photons can be purified to reach near-unity indistinguishability by the process of quantum interference with ancillary photons followed by heralded detection of a subset of them. We report on the indistinguishability of the purified photons by interfering two purified photons and show improvements in the photon indistinguishability of 2.774(3)% in the low-noise regime, and as high as 10.2(5)% in the high-noise regime.

Bayesian autoencoders for data-driven discovery of coordinates, governing equations and fundamental constants

Recent progress in autoencoder-based sparse identification of nonlinear dynamics (SINDy) underℓ1constraints allows joint discoveries of governing equations and latent coordinate systems from spatio-temporal data, including simulated video frames. However, it is challenging forℓ1-based sparse inference to perform correct identification for real data due to the noisy measurements and often limited sample sizes. To address the data-driven discovery of physics in the low-data and high-noise regimes, we propose Bayesian SINDy autoencoders, which incorporate a hierarchical Bayesian Spike-and-slab Gaussian Lasso prior. Bayesian SINDy autoencoder enables the joint discovery of governing equations and coordinate systems with uncertainty estimate. To resolve the challenging computational tractability of the Bayesian hierarchical setting, we adapt an adaptive empirical Bayesian method with Stochastic Gradient Langevin Dynamics (SGLD) which gives a computationally tractable way of Bayesian posterior sampling within our framework. Bayesian SINDy autoencoder achieves better physics discovery with lower data and fewer training epochs, along with valid uncertainty quantification suggested by the experimental studies. The Bayesian SINDy autoencoder can be applied to real video data, withaccurate physics discovery which correctly identifies the governing equation and provides a close estimate for standard physics constants like gravityg, for example, in videos of a pendulum.

Linear inverse problems with nonnegativity constraints: Singularity of optimisers

We look at continuum solutions in optimisation problems associated to linear inverse problems $ y = Ax $ with non-negativity constraint $ x \geq 0 $. We focus on the case where the noise model leads to maximum likelihood estimation through general divergences, which cover a wide range of common noise statistics such as Gaussian and Poisson. Considering $ x $ as a Radon measure over the domain on which the reconstruction is taking place, we show a general singularity result. In the high noise regime corresponding to $ y \notin\{A x \mid x \geq 0\} $ and under a key assumption on the divergence as well as on the operator $ A $, any optimiser has a singular part with respect to the Lebesgue measure. We hence provide an explanation as to why any possible algorithm successfully solving the optimisation problem will lead to undesirably spiky-looking images when the image resolution gets finer, a phenomenon well documented in the literature. We illustrate these results with several numerical examples inspired by medical imaging.

Deep-blur: Blind identification and deblurring with convolutional neural networks.

We propose a neural network architecture and a training procedure to estimate blurring operators and deblur images from a single degraded image. Our key assumption is that the forward operators can be parameterized by a low-dimensional vector. The models we consider include a description of the point spread function with Zernike polynomials in the pupil plane or product-convolution expansions, which incorporate space-varying operators. Numerical experiments show that the proposed method can accurately and robustly recover the blur parameters even for large noise levels. For a convolution model, the average signal-to-noise ratio of the recovered point spread function ranges from 13dB in the noiseless regime to 8dB in the high-noise regime. In comparison, the tested alternatives yield negative values. This operator estimate can then be used as an input for an unrolled neural network to deblur the image. Quantitative experiments on synthetic data demonstrate that this method outperforms other commonly used methods both perceptually and in terms of SSIM. The algorithm can process a 512 512 image under a second on a consumer graphics card and does not require any human interaction once the operator parameterization has been set up.1.

Meta Derivative Identity for the Conditional Expectation

Consider a pair of random vectors (X,Y) and the conditional expectation operator E[X|Y = y]. This work studies analytical properties of the conditional expectation by characterizing various derivative identities. The paper consists of two parts. In the first part of the paper, a general derivative identity for the conditional expectation is derived. Specifically, for the Markov chain U ↔ X ↔ Y, a compact expression for the Jacobian matrix of E[ψ(Y,U)|Y = y] for a smooth function ψ is derived. In the second part of the paper, the main identity is specialized to the exponential family and two main applications are shown. First, it is demonstrated that, via various choices of the random vector U and function ψ, one can recover and generalize several known identities (e.g., Tweedie’s formula) and derive some new ones. For example, a new relationship between conditional expectations and conditional cumulants is established. Second, it is demonstrated how the derivative identities can be used to establish new lower bounds on the estimation error. More specifically, using one of the derivative identities in conjunction with a Poincaré inequality, a new lower bound on the minimum mean squared error, which holds for all prior distributions on the input signal, is derived. The new lower bound is shown to be tight in the high-noise regime for the additive Gaussian noise setting.

Derivative-based SINDy (DSINDy): Addressing the challenge of discovering governing equations from noisy data

Recent advances in the field of data-driven dynamics allow for the discovery of ODE systems using state measurements. One approach, known as Sparse Identification of Nonlinear Dynamics (SINDy), assumes the dynamics are sparse within a predetermined basis in the states and finds the expansion coefficients through linear regression with sparsity constraints. This approach requires an accurate estimation of the state time derivatives, which is not necessarily possible in the high-noise regime without additional constraints. We present an approach called Derivative-based SINDy (DSINDy) that combines two novel methods to improve ODE recovery at high-noise levels. First, we denoise the state variables by applying a projection operator that leverages the assumed basis for the system dynamics. Second, we use a second order cone program (SOCP) to find the derivative and governing equations simultaneously. We derive theoretical results for the projection-based denoising step, which allow us to estimate the values of hyperparameters used in the SOCP formulation. This underlying theory helps limit the number of required user-specified parameters. We present results demonstrating that our approach leads to improved system recovery for the Van der Pol oscillator, the Duffing oscillator, the Rössler attractor, and the Lorenz 96 model.

The DNA Storage Channel: Capacity and Error Probability Bounds

The DNA storage channel is considered, in which the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$M$ </tex-math></inline-formula> Deoxyribonucleic acid (DNA) molecules comprising each codeword are stored without order, sampled <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$N$ </tex-math></inline-formula> times with replacement, and then sequenced over a discrete memoryless channel. For a constant coverage depth <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$M/N$ </tex-math></inline-formula> and molecule length scaling <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\Theta (\log M)$ </tex-math></inline-formula> , lower (achievability) and upper (converse) bounds on the capacity of the channel, as well as a lower (achievability) bound on the reliability function of the channel are provided. Both the lower and upper bounds on the capacity generalize a bound which was previously known to hold only for the binary symmetric sequencing channel, and only under certain restrictions on the molecule length scaling and the crossover probability parameters. When specified to binary symmetric sequencing channel, these restrictions are completely removed for the lower bound and are significantly relaxed for the upper bound in the high-noise regime. The lower bound on the reliability function is achieved under a universal decoder, and reveals that the dominant error event is that of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">outage</i> – the event in which the capacity of the channel induced by the DNA molecule sampling operation does not support the target rate.

Sparse Multi-Reference Alignment: Phase Retrieval, Uniform Uncertainty Principles and the Beltway Problem

Motivated by cutting-edge applications like cryo-electron microscopy (cryo-EM), the Multi-Reference Alignment (MRA) model entails the learning of an unknown signal from repeated measurements of its images under the latent action of a group of isometries and additive noise of magnitude \(\sigma \). Despite significant interest, a clear picture for understanding rates of estimation in this model has emerged only recently, particularly in the high-noise regime \(\sigma \gg 1\) that is highly relevant in applications. Recent investigations have revealed a remarkable asymptotic sample complexity of order \(\sigma ^6\) for certain signals whose Fourier transforms have full support, in stark contrast to the traditional \(\sigma ^2\) that arise in regular models. Often prohibitively large in practice, these results have prompted the investigation of variations around the MRA model where better sample complexity may be achieved. In this paper, we show that sparse signals exhibit an intermediate \(\sigma ^4\) sample complexity even in the classical MRA model. Further, we characterize the dependence of the estimation rate on the support size s as \(O_p(1)\) and \(O_p(s^{3.5})\) in the dilute and moderate regimes of sparsity respectively. Our techniques have implications for the problem of crystallographic phase retrieval, indicating a certain local uniqueness for the recovery of sparse signals from their power spectrum. Our results explore and exploit connections of the MRA estimation problem with two classical topics in applied mathematics: the beltway problem from combinatorial optimization, and uniform uncertainty principles from harmonic analysis. Our techniques include a certain enhanced form of the probabilistic method, which might be of general interest in its own right.

Learning mixtures of permutations: Groups of pairwise comparisons and combinatorial method of moments

In applications such as rank aggregation, mixture models for permutations are frequently used when the population exhibits heterogeneity. In this work, we study the widely used Mallows mixture model. In the high-dimensional setting, we propose a polynomial-time algorithm that learns a Mallows mixture of permutations on n elements with the optimal sample complexity that is proportional to logn, improving upon previous results that scale polynomially with n. In the high-noise regime, we characterize the optimal dependency of the sample complexity on the noise parameter. Both objectives are accomplished by first studying demixing permutations under a noiseless query model using groups of pairwise comparisons, which can be viewed as moments of the mixing distribution, and then extending these results to the noisy Mallows model by simulating the noiseless oracle.

Dihedral Multi-Reference Alignment

We study the dihedral multi-reference alignment problem of estimating the orbit of a signal from multiple noisy observations of the signal, acted on by random elements of the dihedral group. We show that if the group elements are drawn from a generic distribution, the orbit of a generic signal is uniquely determined from the second moment of the observations. This implies that the optimal estimation rate in the high noise regime is proportional to the square of the variance of the noise. This is the first result of this type for multi-reference alignment over a non-abelian group with a non-uniform distribution of group elements. Based on tools from invariant theory and algebraic geometry, we also delineate conditions for unique orbit recovery for multi-reference alignment models over finite groups (namely, when the dihedral group is replaced by a general finite group) when the group elements are drawn from a generic distribution. Finally, we design and study numerically three computational frameworks for estimating the signal based on group synchronization, expectation-maximization, and the method of moments.

An approximate expectation-maximization for two-dimensional multi-target detection.

We consider the two-dimensional multi-target detection (MTD) problem of estimating a target image from a noisy measurement that contains multiple copies of the image, each randomly rotated and translated. The MTD model serves as a mathematical abstraction of the structure reconstruction problem in single-particle cryo-electron microscopy, the chief motivation of this study. We focus on high noise regimes, where accurate detection of image occurrences within a measurement is impossible. To estimate the image, we develop an expectation-maximization framework that aims to maximize an approximation of the likelihood function. We demonstrate image recovery in highly noisy environments, and show that our framework outperforms the previously studied autocorrelation analysis in a wide range of parameters.

Two-Dimensional Multi-Target Detection: An Autocorrelation Analysis Approach

We consider thetwo-dimensional multi-target detection problem of recovering a target image from a noisy measurement that contains multiple copies of the image, each randomly rotated and translated. Motivated by the structure reconstruction problem in single-particle cryo-electron microscopy, we focus on the high noise regime, where the noise hampers accurate detection of the image occurrences. We develop an autocorrelation analysis framework to estimate the image directly from a measurement with an arbitrary spacing distribution of image occurrences, bypassing the estimation of individual locations and rotations. We conduct extensive numerical experiments, and demonstrate image recovery in highly noisy environments. <xref ref-type="fn" rid="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">¹</xref>

A Toolkit for Data-Driven Discovery of Governing Equations in High-Noise Regimes

We consider the data-driven discovery of governing equations from time-series data in the limit of high noise. The algorithms developed describe an extensive toolkit of methods for circumventing the deleterious effects of noise in the context of the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">sparse identification of nonlinear dynamics</i> (SINDy) framework. We offer two primary contributions, both focused on noisy data acquired from a system <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\dot { \boldsymbol x} = { \boldsymbol f} ({ \boldsymbol x})$ </tex-math></inline-formula> . First, we propose, for use in high-noise settings, an extensive toolkit of critically enabling extensions for the SINDy regression method, to progressively cull functionals from an over-complete library and yield a set of sparse equations that regress to the derivate <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\dot { \boldsymbol {x}}$ </tex-math></inline-formula> . This toolkit includes: (regression step) weight timepoints based on estimated noise, use ensembles to estimate coefficients, and regress using FFTs; (culling step) leverage linear dependence of functionals, and restore and protect culled functionals based on Figures of Merit (FoMs). In a novel Assessment step, we define FoMs that compare model predictions to the original time-series (i.e., <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${ \boldsymbol x}(t)$ </tex-math></inline-formula> rather than <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\dot { \boldsymbol {x}}(t)$ </tex-math></inline-formula> ). These innovations can extract sparse governing equations and coefficients from high-noise time-series data (e.g., 300% added noise). For example, it discovers the correct sparse libraries in the Lorenz system, with median coefficient estimate errors equal to 1%−3% (for 50% noise), 6%−8% (for 100% noise), and 23%−25% (for 300% noise). The enabling modules in the toolkit are combined into a single method, but the individual modules can be tactically applied in other equation discovery methods (SINDy or not) to improve results on high-noise data. Second, we propose a technique, applicable to any model discovery method based on <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\dot { \boldsymbol x} = { \boldsymbol f} ({ \boldsymbol x})$ </tex-math></inline-formula> , to assess the accuracy of a discovered model in the context of non-unique solutions due to noisy data. Currently, this non-uniqueness can obscure a discovered model’s accuracy and thus a discovery method’s effectiveness. We describe a technique that uses linear dependencies among functionals to transform a discovered model into an equivalent form that is closest to the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">true</i> model, enabling more accurate assessment of a discovered model’s correctness.

Robust Localization With Bounded Noise: Creating a Superset of the Possible Target Positions via Linear-Fractional Representations

Locating a target is key in many applications, namely in high-stakes real-world scenarios, like detecting humans or obstacles in vehicular networks. In scenarios where precise statistics of the measurement noise are unavailable, applications require localization methods that assume minimal knowledge on the noise distribution. We present a scalable algorithm delimiting a tight superset of all possible target locations, assuming range measurements to known landmarks, contaminated with bounded noise and unknown distributions. This superset is of primary interest in robust statistics since it is a tight majorizer of the set of Maximum-Likelihood (ML) estimates parametrized by noise densities respecting two main assumptions: (1) the noise distribution is supported on a ellipsoidal uncertainty region and (2) the measurements are non-negative with probability one. We create the superset through convex relaxations that use Linear Fractional Representations (LFRs), a well-known technique in robust control. For low noise regimes the supersets created by our method double the accuracy of a standard semidefinite relaxation. For moderate to high noise regimes our method still improves the benchmark but the benefit tends to be less significant, as both supersets tend to have the same size (area).

Comparative Study of Sampling-Based Simulation Costs of Noisy Quantum Circuits

Noise in quantum operations often negates the advantage of quantum computation. However, most classical simulations of quantum computers calculate the ideal probability amplitudes by either storing full state vectors or using sophisticated tensor-network contractions. Here we investigate sampling-based classical simulation methods for noisy quantum circuits. Specifically, we characterize the simulation costs of two major schemes, stabilizer-state sampling of magic states and Heisenberg propagation, for quantum circuits subject to stochastic Pauli noise, such as depolarizing and dephasing noise. To this end, we introduce several techniques for the stabilizer-state sampling to reduce the simulation costs under such noise. It is revealed that in the low-noise regime, stabilizer-state sampling results in a smaller sampling cost, while Heisenberg propagation is better in the high-noise regime. Furthermore, for a high depolarizing noise rate of approximately 10%, these methods provide better scaling than that given by the low-rank stabilizer decomposition. We believe that this knowledge of classical simulation costs is useful to achieve possible quantum advantage on near-term noisy quantum devices as well as efficient classical simulation methods.

Hyperentanglement-enhanced quantum illumination

In quantum illumination, the signal mode of light, entangled with an idler mode, is dispatched towards a suspected object bathed in thermal noise and the returning mode, along with the stored idler mode, is measured to determine the presence or absence of the object. In this process, entanglement is destroyed but its benefits in the form of classical correlations and enlarged Hilbert space survive. Here, we propose the use of probe state hyperentangled in two degrees of freedom - polarization and frequency, to achieve a significant 12dB performance improvement in error probability exponent over the best known quantum illumination procedure in the low noise regime. We present a simple receiver model using four optical parametric amplifiers (OPA) that exploits hyperentanglement in the probe state to match the performance of the feed-forward sum-frequency generator (FF-SFG) receiver in the high noise regime. By replacing each OPA in the proposed model with a FF-SFG receiver, further 3dB improvement in the performance of a lone FF-SFG receiver can be seen.

Device-independent quantum key distribution with random key basis

Device-independent quantum key distribution (DIQKD) is the art of using untrusted devices to distribute secret keys in an insecure network. It thus represents the ultimate form of cryptography, offering not only information-theoretic security against channel attacks, but also against attacks exploiting implementation loopholes. In recent years, much progress has been made towards realising the first DIQKD experiments, but current proposals are just out of reach of today’s loophole-free Bell experiments. Here, we significantly narrow the gap between the theory and practice of DIQKD with a simple variant of the original protocol based on the celebrated Clauser-Horne-Shimony-Holt (CHSH) Bell inequality. By using two randomly chosen key generating bases instead of one, we show that our protocol significantly improves over the original DIQKD protocol, enabling positive keys in the high noise regime for the first time. We also compute the finite-key security of the protocol for general attacks, showing that approximately 108–1010 measurement rounds are needed to achieve positive rates using state-of-the-art experimental parameters. Our proposed DIQKD protocol thus represents a highly promising path towards the first realisation of DIQKD in practice.

Synaptic plasticity as Bayesian inference.

Learning, especially rapid learning, is critical for survival. However, learning is hard; a large number of synaptic weights must be set based on noisy, often ambiguous, sensory information. In such a high-noise regime, keeping track of probability distributions over weights is the optimal strategy. Here we hypothesize that synapses take that strategy; in essence, when they estimate weights, they include error bars. They then use that uncertainty to adjust their learning rates, with more uncertain weights having higher learning rates. We also make a second, independent, hypothesis: synapses communicate their uncertainty by linking it to variability in postsynaptic potential size, with more uncertainty leading to more variability. These two hypotheses cast synaptic plasticity as a problem of Bayesian inference, and thus provide a normative view of learning. They generalize known learning rules, offer an explanation for the large variability in the size of postsynaptic potentials and make falsifiable experimental predictions.

The generalized orthogonal Procrustes problem in the high noise regime

AbstractWe consider the problem of estimating a cloud of points from numerous noisy observations of that cloud after unknown rotations and possibly reflections. This is an instance of the general problem of estimation under group action, originally inspired by applications in three-dimensional imaging and computer vision. We focus on a regime where the noise level is larger than the magnitude of the signal, so much so that the rotations cannot be estimated reliably. We propose a simple and efficient procedure based on invariant polynomials (effectively: the Gram matrices) to recover the signal, and we assess it against fundamental limits of the problem that we derive. We show our approach adapts to the noise level and is statistically optimal (up to constants) for both the low and high noise regimes. In studying the variance of our estimator, we encounter the question of the sensivity of a type of thin Cholesky factorization, for which we provide an improved bound which may be of independent interest.