Perturbing Noise Research Articles

How deep convolutional neural networks lose spatial information with training

A central question of machine learning is how deep nets manage to learn tasks in high dimensions. An appealing hypothesis is that they achieve this feat by building a representation of the data where information irrelevant to the task is lost. For image datasets, this view is supported by the observation that after (and not before) training, the neural representation becomes less and less sensitive to diffeomorphisms acting on images as the signal propagates through the network. This loss of sensitivity correlates with performance and surprisingly correlates with a gain of sensitivity to white noise acquired during training. Which are the mechanisms learned by convolutional neural networks (CNNs) responsible for the these phenomena? In particular, why is the sensitivity to noise heightened with training? Our approach consists of two steps. (1) Analyzing the layer-wise representations of trained CNNs, we disentangle the role of spatial pooling in contrast to channel pooling in decreasing their sensitivity to image diffeomorphisms while increasing their sensitivity to noise. (2) We introduce model scale-detection tasks, which qualitatively reproduce the phenomena reported in our empirical analysis. In these models we can assess quantitatively how spatial pooling affects these sensitivities. We find that the increased sensitivity to noise observed in deep ReLU networks is a mechanistic consequence of the perturbing noise piling up during spatial pooling, after being rectified by ReLU units. Using odd activation functions like tanh drastically reduces the CNNs’ sensitivity to noise.

NoiseMol: A noise-robusted data augmentation via perturbing noise for molecular property prediction

Simplified Molecular-Input Line-Entry System (SMILES) is one of a widely used molecular representation methods for molecular property prediction. We conjecture that all the characters in the SMILES string of a molecule are essential for making up the molecules, but most of them make little contribution to determining a particular property of the molecule. Therefore, we verified the conjecture in the pre-experiment. Motivated by the result, we propose to inject proper noisy information into the SMILES to augment the training data by increasing the diversity of the labeled molecules. To this end, we explore injecting perturbing noise into the original labeled SMILES strings to construct augmented data for alleviating the limitation of the labeled compound data and enhancing the model to extract more useful molecular representation for molecular property prediction. Specifically, we directly adopt mask, swap, deletion, and fusion operations on SMILES strings to randomly mask, swap, and delete atoms in SMILES strings. Then, the augmented data is used by two strategies: each epoch alternately feeds the original and perturbing noisy molecules, or each batch alternately feeds the original and perturbing noisy molecules. We conduct experiments on both Transformer and BiGRU models to validate the effectiveness by adopting widely used datasets from MoleculeNet and ZINC. Experimental results demonstrate that the proposed method outperforms strong baselines on all the datasets. NoiseMol obtains the best performance on BBBP and FDA when compared with state-of-the-art methods. Besides, NoiseMol achieves the best accuracy on LogP. Therefore, injecting perturbing noise into the labeled SMILES strings is an effective and efficient method, which improves the prediction performance, generalization, and robustness of the deep learning models.

Learning-aware feature denoising discriminator

Although generative adversarial networks (GANs) show great prospects for the task of image synthesis, the quality of synthesized images by existing GANs is sometimes inferior to real images because their discriminators cannot effectively learn robust identification features from input images. In addition, the training process of discriminator is prone to be unstable. To this end, inspired by the denoising auto-encoders, we propose a learning-aware feature denoising discriminator. It is designed to pay attention to robust features of input images, so as to improve its robustness in identifying features and recognition ability in training process. First, we use a decoder to generate perturbing noise and add it to real image to get corrupted image. Then, we get the encodings of the corrupted image and real image through an encoder. Finally, we minimize both types of encoding to constitute a denoising penalty and add it to the loss of the discriminator. We also show that our method is compatible with most existing GANs for three image synthesis tasks. Extensive experimental results show that compared with baseline models, our proposed method not only improves the quality of synthesized images, but also stabilizes the training process of discriminator.

Topological order detection and qubit encoding in Su–Schrieffer–Heeger type quantum dot arrays

In this study, building on the 1D topological Su–Schrieffer–Heeger (SSH) model, we propose a model of quantum dot arrays with odd and even parity and variable on-site local potentials to examine topological edge states and a possible quantum information encoding, using these states. We first investigate the SSH model with alternating tunneling amplitudes t1 and t2. We study the model in a ring-like structure and then proceed to minimal open-end chains with even (N=4) and odd (N=5) number of dots. Furthermore, we depart from the basic SSH model by introducing local potentials μi, which offer additional control at the cost of breaking the chiral symmetry of the Hamiltonian and study the implications. Then, we propose an idealized “static” charge qubit design, based on encoding the topological invariant ν as qubit states, that exploits the topological nature of the edge states and their collective character. We introduce perturbing noise δtij(t) into the system and demonstrate the robustness of the states for some range of the ratio ξ=t1/t2. Moreover, we show a possible way to detect the presence of topological order in the system using equilibrium dynamics for both even and odd chains. We utilize the quantum informatic measure of bipartite mutual information I{b:e}(2)(t) as a measure of bulk-edge quantum correlations and a quantitative indicator for the manifestation of bulk-edge correspondence; here, we also propose a dynamical qubit encoding with ν for specific quantum chain parity. Finally, we offer a few remarks on potential future explorations.

Noise induces continuous and noncontinuous transitions in neuronal interspike intervals range

Noise appears in the brain due to various sources, such as ionic channel fluctuations and synaptic events. They affect the activities of the brain and influence neuron action potentials. Stochastic differential equations have been used to model firing patterns of neurons subject to noise. In this work, we consider perturbing noise in the adaptive exponential integrate-and-fire (AEIF) neuron. The AEIF is a two- dimensional model that describes different neuronal firing patterns by varying its parameters. Noise is added in the equation related to the membrane potential. We show that a noise current can induce continuous and noncontinuous transitions in neuronal interspike intervals. Moreover, we show that the noncontinuous transition occurs mainly for parameters close to the border between tonic spiking and burst activities of the neuron without noise

Compensatory mechanisms affect sensorimotor integration during ongoing vocal motor acts in marmoset monkeys.

Any transmission of vocal signals faces the challenge of acoustic interferences such as heavy rain, wind, animal or urban sounds. Consequently, several mechanisms and strategies have evolved to optimize signal-to-noise ratio. Examples to increase detectability are the Lombard effect, an involuntary rise in call amplitude in response to masking ambient noise, which is often associated with other vocal changes such as call frequency and duration, as well as the animals' capability of limiting calling to periods where noise perturbation is absent. Previous studies revealed vocal flexibility and various audio-vocal integration mechanisms in marmoset monkeys. Using acoustic perturbation triggered by vocal behaviour, we investigated whether marmosets are capable of exhibiting changes in call structure when perturbing noise starts after call onset or whether such effects only occur if noise perturbation starts prior to call onset. We show that marmosets are capable of rapidly modulating call amplitude and frequency in response to such noise perturbation. Vocalizations swiftly increased call frequency after noise onset indicating a rapid effect of perturbing noise on vocal motor production. Call amplitudes were also affected. Interestingly, however, the marmosets did not exhibit the Lombard effect as previously reported but decreased call intensity in response to noise. Our findings indicate that marmosets possess a general avoidance strategy to call in the presence of ambient noise and suggest that these animals are capable of counteracting a previously thought involuntary audio-vocal mechanism, the Lombard effect. These findings will pave the way to investigate the underlying audio-vocal integration mechanisms explaining these behaviours.

On the asymptotic exit control problem for stochastically perturbed prescription opioid epidemic models

In this paper, we consider the problem of asymptotic exit control for a prescription opioid epidemic that describes the interaction between the regular prescription or addictive use of opioid drugs, and the process of rehabilitation and that of relapsing into the opioid drug use. In particular, our interest is in the situation, when the optimal control effort appearing linearly in the opioid epidemic model is interpreted as the rate at which the susceptible individuals are effectively removed from the population due to an opioid-related intervention policy, while a small perturbing noise enters through the dynamics of the susceptible group in the population compartmental model. To this end, we introduce a mathematical apparatus that minimizes the asymptotic exit-rate with which the solution for such stochastically perturbed prescription opioid epidemics exits from a given bounded open domain. Moreover, under certain conditions, we provide an admissible optimal control for the corresponding optimal control problem that optimally effected removal of the susceptible or recovered individuals from the population dynamics.

Altering Stochastic Noise in Gene Expression for HIV Therapy

HIV's ability to enter a transcriptionally dormant state and establish a reservoir of latently infected cells is the major barrier to eradicating HIV from infected patients. Significant efforts are aimed at reactivating latent HIV and purging the reservoir through targeting of molecular mechanisms implicated in the establishment of latency. These efforts to perturb the latent reservoir have faced substantial challenges. Our previous studies implicated stochastic noise (i.e. fluctuations) in an HIV transcriptional positive-feedback circuit as crucial for the establishment of HIV latency and predicted that perturbing noise would alter HIV latency (Weinberger et al. Cell 2005; Dar et al. PNAS 2012). Here, we demonstrate that manipulation of stochastic noise in HIV gene expression radically perturbs HIV latency. Screening a library of small-molecule drug compounds identified over 50 compounds that modulate noise in the HIV promoter without changing the promoter's mean expression level. Strikingly, the noise-modulating compounds synergize with conventional transcriptional activators and surpass current best-in-class reactivation cocktails, while maintaining greater cell viability. Thus, noise-modulating compounds may present an approach to perturb the stability of the latent state. More generally, stochastic noise may represent a new unexplored axis for drug discovery that allows enhanced control over cell-fate specification decisions, metastasis, and pathogen persistence phenotypes.

Stochastic Averaging for a Hamiltonian System with Skew Random Perturbations

We consider a stochastic process that arises due to small random perturbations of a particle moving along the orbits of a double-well Hamiltonian. The perturbing noise is assumed to have skewness along the infinity-shaped homoclinic orbit of the Hamiltonian. As the noise strength goes to zero, we characterize the limiting dynamics in terms of a graph-valued Markov process. The effort centers around the construction of certain perturbed test functions needed to prove convergence. Following Sowers, we accomplish this through the development of suitable corrector functions in a boundary layer near the homoclinic orbit.

Does noise shift or delete spikes?

Stochastic leaky integrate-and-fire neurons are widely used to study properties of neural networks (e.g. [1]) as well as the spontaneous activity [2] and signal transmission [3] of single neurons. They have also been employed in studies of the 'common-noise' problem, i.e. the question of how correlated input to two cells causes output spike train correlations of these neurons [4]. In the cortex, input correlations seem to be weak [5] (in terms of the above sketch, c<<1), however, in the sensory periphery with a strong time-dependent stimulus the situation can be completely different (e.g.[6]). The two neurons receive the same strong stimulus (often modeled as a random signal) and each neuron is subject to a small amount of intrinsic noise (1-c<<1). Without the intrinsic noise two identical neurons would fire in complete synchrony. How does the weak noise change the spikes? To address this question, we first study two versions of inhomogeneous Poisson processes. In one version the intrinsic noise can shift spike times, in the other the noise leads to deletions and additions of spikes. We construct these processes in such a way that the correlations between the modulated spike trains and the perturbing noise is the same in both versions. In this setup, it is possible to analytically calculate the cross-correlation between two spike trains with independent intrinsic noise. By comparison with extensive simulations of stochastic integrate-and-fire neurons we then inspect, which of the two modulation methods is more appropriate to capture the effect of a weak noise on the spike train of a dynamic neuron model. Figure 1 Two neurons receive a common signal and distinct intrinsic noise. c is the correlation parameter that tunes the strength of the input correlation.

The misconception of mean-reversion

The notion of random motion in a potential well is elemental in the physical sciences and beyond. Quantitatively, this notion is described by reverting diffusions—asymptotically stationary diffusion processes which are simultaneously (i) driven toward a reversion level by a deterministic force, and (ii) perturbed off the reversion level by a random white noise. The archetypal example of reverting diffusions is the Ornstein–Uhlenbeck process, which is mean-reverting. In this paper we analyze reverting diffusions and establish that: (i) if the magnitude of the perturbing noise is constant then the diffusion's stationary density is unimodal and the diffusion is mode-reverting; (ii) if the magnitude of the perturbing noise is non-constant then, in general, neither is the diffusion's stationary density unimodal, nor is the diffusion mode-reverting. In the latter case we further establish a result asserting when unimodality and mode-reversion do hold. In particular, we demonstrate that the notion of mean-reversion, which is fundamental in economics and finance, is a misconception—as mean-reversion is an exception rather than the norm.

Regularization and Stabilization of Randomly Switching Dynamic Systems

This work concerns regularization (or explosion suppression) and stabilization of randomly switching dynamic systems by selecting suitable input functions. Starting with a randomly switching system modulated by a continuous-time Markov chain, our aim is to find a feedback control (input function) so that the system becomes stable. But before addressing the stabilization issue, another problem needs to be resolved first. Because of the fast growth in the continuous state variable, the solution of the system has a finite explosion time with probability one. To ensure the existence of global solutions, a feedback control (a noise perturbation) is added to make the system regular. Then another feedback control (a second perturbing noise) is added to ensure the stability of the resulting systems. Owing to the nonlinearity, closed-form solutions are often virtually impossible to obtain. Thus, a discrete-time approximation algorithm is constructed. Under simple conditions, a suitably interpolated sequence of the discrete-time algorithm is shown to converge to the limit switching diffusion. Finally, regularization and stabilization of the approximating sequence are dealt with.

Robustness, Weak Stability, and Stability in Distribution of Adaptive Filtering Algorithms under Model Mismatch

This work is concerned with robustness, convergence, and stability of adaptive filtering (AF) type algorithms in the presence of model mismatch. The algorithms under consideration are recursive and have inherent multiscale structure. They can be considered as dynamic systems, in which the “state” changes much more slowly than the perturbing noise. Beyond the existing results on adaptive algorithms, model mismatch significantly affects convergence properties of AF algorithms, raising issues of algorithm robustness. Weak convergence and weak stability (i.e., recurrence) under model mismatch are derived. Based on the limiting stochastic differential equations of suitably scaled iterates, stability in distribution is established. Then algorithms with decreasing step sizes and their convergence properties are examined. When input signals are large, identification bias due to model mismatch will become large and unacceptable. Methods for reducing such bias are introduced when the identified models are used in r...

NLMS algorithm with decreasing step size for adaptive IIR filters

In this paper, we modify the GLMS algorithm to become the proposed LMS-SAS algorithm, which can more effectively converge to the global minimum of the mean-square output error (MSE) objective function. We also derive the infinite impulse response-normalized least-squares algorithm (IIR-NLMS), whose behavior is similar to the LMS-SAS algorithm. The GLMS algorithm achieves its global search capability by appending a random perturbing noise to the LMS algorithm. Similarly, we suggest that such perturbing noise is to be multiplied by its MSE objective function in the proposed LMS-SAS algorithm. For the NLMS algorithm, we use the gradient estimation error, which exists naturally in the adaptive process, to act as perturbing noise. We have shown, theoretically and experimentally, that the LMS-SAS and NLMS algorithm do converge to the global minimum.

Convergence of a global stochastic optimization algorithm with partial step size restarting

This work develops a class of stochastic global optimization algorithms that are Kiefer-Wolfowitz (KW) type procedures with an added perturbing noise and partial step size restarting. The motivation stems from the use of KW-type procedures and Monte Carlo versions of simulated annealing algorithms in a wide range of applications. Using weak convergence approaches, our effort is directed to proving the convergence of the underlying algorithms under general noise processes.

Convergence of a global stochastic optimization algorithm with partial step size restarting

This work develops a class of stochastic global optimization algorithms that are Kiefer-Wolfowitz (KW) type procedures with an added perturbing noise and partial step size restarting. The motivation stems from the use of KW-type procedures and Monte Carlo versions of simulated annealing algorithms in a wide range of applications. Using weak convergence approaches, our effort is directed to proving the convergence of the underlying algorithms under general noise processes.

Determination of joint functional axes from noisy marker data using the finite helical axis

When video-based motion analysis systems are used to measure segmental kinematics, the finite helical displacement computed between two adjacent body segments in two successive positions i, i + 1 is often used to approximate the instantaneous joint movement. The measured trajectories of the external markers glued on the skin are very perturbed compared to the real displacement of the bony structure, and the inaccuracy in the measurement leads to stochastic errors in the position and direction of the finite helical axis of motion (FHA). As the errors associated with the FHA estimates are inversely proportional to the rotaion magnitude (Woltring, H.J., Huiskes, R., de Lange, A., 1983. Measurement error influence on helical axis accuracy in the description of 3D finite joint movement in biomechanics. In: Woo, S.L., Mates, R.E. (Eds.), Biomechanics symposium AMD 56 (FED 1), New York ASME, pp. 19–22), it is illusive to expect to assess the helical displacement between two neighbouring positions, and so to describe the joint evolution using FHA theory in such a context. A quantification of the errors on the FHA parameters computed between two successive positions i, i + 1 is proposed in this paper, using a numerical simulation of the knee joint evolution during gait. This case has been chosen because previous studies (Cappozzo, A., Catani, F., Leardini, A., 1993. Skin movement artifacts in human movement photogrammetry. Proceedings of the 14th Congress of the International Society of Biomechanics. Paris, France, pp. 238–239) have experimentally described skin and soft tissues perturbations. The results obtained from this simulation lead to two conclusions: first, they confirm the relative FHA cannot be used to represent accurately the joint kinematics during a given movement; and second, they allow a prediction of the minimum joint displacement required in order to have a reliable determination of the helical axis. The aim of this paper is not to present a new calculation method for the FHA, but to propose an alternative use of the FHA. It is generally assumed to describe a joint displacement using a sequence of rotations about three successive axes. In this case, the difficulty for clinical applications is to correctly locate these axes, in order that they coincide with the functional axes of the considered joint. If the FHA theory is used to determine the location and orientation of these functional axes from corresponding pure movement recording, then the results can be very accurate provided that the measured displacement between the two finite positions i and j be sufficient with respect to the perturbing noise. One consequence of this remark is that the rotation axis of the considered, joint may remain stable in the range of motion between these positions. An example of this altenative use of the FHA is displayed in this paper which concerns the determination of the flexion/extension axis of the elbow joint. The elbow joint has been chosen for two reasons: first, it deals with a stable joint rotation axis and second, experimental data were available on a fleshless upper-limb on which the flexion/extension axis of the elbow was marked by an anatomy specialist.

Optimal Residence Time Control of Hamiltonian Systems Perturbed by White Noise

Optimal control of perturbed Hamiltonian systems in $\Re^2$ is studied. Systems are considered with a control term scaling with the size of a small perturbing noise. The dynamics are shown to converge in a certain sense to a diffusion on a graph. Using the approach developed in [M. I. Freidlin and A. D. Wentzell, Mem. Amer. Math. Soc., 109 (1994), pp. 1--82] and [M. I. Freidlin and A. D. Wentzell, Ann. Probab., 21 (1993), pp. 2215--2245] for random perturbations of Hamiltonian systems, a convergence theorem is discussed. An optimal control theorem is then developed to maximize the expected exit time from a domain. This control is asymptotically robust for small noise. Several examples are provided.

The finite-element wave propagator approach and the tsunami inversion problem

Inversion of tsunami data is of great importance since it provides independent and reliable information on tsunami source processes. Good results on estimation of heterogeneous slip distribution along the causative seismic fault have been recently achieved, both by inversion of tsunami tide-gauge records and run-up data. This work addresses some questions of the general ill-posed problem of inversion of tsunami tide-gauge records, from a new point of view, namely the Wave Propagator (WP) approach. The WP is a time dependent linear operator that describes the physical system completely. It is to be stressed that, while the direct problem is well posed, for its solution is obtained by simply applying the WP to the initial perturbation, on the contrary, the inversion of one or more tide-gauge records to determine the initial perturbation results into an ill-posed problem. For illustrative purposes, we will conduct our discussion by taking into account a simple case, that is, the linear propagation of an initial bulge along a rectangular flat channel. First we will point out some important properties of the Green's functions concerning symmetry and the symmetry rupture owing to the boundary conditions. Subsequently some numerical experiments will be illustrated in order to discuss the main features of the inversion procedure. In particular we will study the dependence of the goodness of the inverted fields on some parameters, namely the sampling interval of the tide-gauge records, the number and the spatial distribution of tide-gauge records and the perturbing noise on the records.

A stochastic approach to two-level ion dynamics driven by light beams

Abstract The probability distribution of a two-level ion confined in a harmonic trap and perturbed by the absorption and emission of photons is calculated. The ion is assumed to be excited by a light beam and relaxes back either by induced emissions or irreversible spontaneous emissions. The photon-ion interaction is modeled as a perturbing noise acting on the ion by modifying its energy and momentum. The current calculations reproduce the quantum results of the laser cooling of ions in static traps in the heavy particle limit.