Stochastic Averaging Research Articles

Average decayed dynamics of one-step transformation process with randomly varying return transition rate

The problem of stochastic averaging of the decayed dynamics of the output state population of a one-step transformation process with constant deterministic forward transition rate and randomly varying return transition rate is solved in approximation where random variation is modeled as a dichotomous stochastic process. The form of the obtained solution represented as a product between bimodal sigmoid rise of average population and its unimodal exponential decay is shown to largely be dependent on the stochastic frequency and amplitude parameters. For example, at high stochastic frequency, the behavior of population is reduced to that of a decayed one-step deterministic system. However, for resonance stochastic amplitude at low stochastic frequency, such behavior coincides with that of three-exponential rise-decay kinetics typical rather of a three-step deterministic slowly decaying process. Thus, there is an equivalence between using a more complex deterministic kinetic model and a less complex stochastic kinetic model for describing the decayed dynamics of different irreversible systems.

Deep Neural Network Confidence Calibration from Stochastic Weight Averaging

Overconfidence in deep neural networks (DNN) reduces the model’s generalization performance and increases its risk. The deep ensemble method improves model robustness and generalization of the model by combining prediction results from multiple DNNs. However, training multiple DNNs for model averaging is a time-consuming and resource-intensive process. Moreover, combining multiple base learners (also called inducers) is hard to master, and any wrong choice may result in lower prediction accuracy than from a single inducer. We propose an approximation method for deep ensembles that can obtain ensembles of multiple DNNs without any additional costs. Specifically, multiple local optimal parameters generated during the training phase are sampled and saved by using an intelligent strategy. We use cycle learning rates starting at 75% of the training process and save the weights associated with the minimum learning rate in every iteration. Saved sets of the multiple model parameters are used as weights for a new model to perform forward propagation during the testing phase. Experiments on benchmarks of two different modalities, static images and dynamic videos, show that our method not only reduces the calibration error of the model but also improves the accuracy of the model.

Model order reduction and stochastic averaging for the analysis and design of micro-electro-mechanical systems

Electro-mechanical systems are key elements in engineering. They are designed to convert electrical signals and power into mechanical motion and vice-versa. As the number of networked systems grows, the corresponding mathematical models become more and more complex, and novel sophisticated techniques for their analysis and design are required. We present a novel methodology for the analysis and design of electro-mechanical systems subject to random external inputs. The method is based on the joint application of a model order reduction technique, by which the original electro-mechanical variables are projected onto a lower dimensional space, and of a stochastic averaging technique, which allows the determination of the stationary probability distribution of the system mechanical energy. The probability distribution can be exploited to assess the system performance and for system optimization and design. As examples of application, we apply the method to power factor correction for the optimization of a vibration energy harvester, and to analyse a system composed by two coupled electro-mechanical resonators for sensing applications.

Analysis of the stationary probability density of a generalized and bistable van der Pol system excited by colored noise

The stochastic P-bifurcation behavior of bi-stability in a generalized van der Pol oscillator with the fractional damping under colored noise and thermal excitation is investigated. Firstly, using the principle of minimal mean square error and lin-earization method, the non-linear stiffness terms can be equivalent to a linear stiffness which is a function of the system amplitude, and the original system is simplified to an equivalent integer order van der Pol system. Secondly, the system amplitude stationary probability density function is obtained by the stochastic averaging, and then based on the singularity theory, the critical parametric con-ditions for the system amplitude stochastic P-bifurcation are found. Finally, the types of the stationary probability density function of the system amplitude are qualitatively analyzed in each area divided by the transition set curves. The con-sistency between the analytical results and the numerical results acquired from Monte-Carlo simulation also testifies the theoretical analysis in this paper and the method used in this paper can directly guide the design of the fractional order controller to adjust the response of the system.

Supervision by Denoising.

Learning-based image reconstruction models, such as those based on the U-Net, require a large set of labeled images if good generalization is to be guaranteed. In some imaging domains, however, labeled data with pixel- or voxel-level label accuracy are scarce due to the cost of acquiring them. This problem is exacerbated further in domains like medical imaging, where there is no single ground truth label, resulting in large amounts of repeat variability in the labels. Therefore, training reconstruction networks to generalize better by learning from both labeled and unlabeled examples (called semi-supervised learning) is problem of practical and theoretical interest. However, traditional semi-supervised learning methods for image reconstruction often necessitate handcrafting a differentiable regularizer specific to some given imaging problem, which can be extremely time-consuming. In this work, we propose "supervision by denoising" (SUD), a framework to supervise reconstruction models using their own denoised output as labels. SUD unifies stochastic averaging and spatial denoising techniques under a spatio-temporal denoising framework and alternates denoising and model weight update steps in an optimization framework for semi-supervision. As example applications, we apply SUD to two problems from biomedical imaging-anatomical brain reconstruction (3D) and cortical parcellation (2D)-to demonstrate a significant improvement in reconstruction over supervised-only and ensembling baselines. Our code available at https://github.com/seannz/sud.

Transient Response Analysis of Nonlinear Oscillators With Fractional Derivative Elements Under Gaussian White Noise Using Complex Fractional Moments

Abstract Complex fractional moment (CFM), which is defined as the Mellin transform of a probability density function (PDF), has been successfully employed to find the response PDF of a wide variety of integer-order nonlinear oscillators. In this paper, a CFM-based analysis is performed to determine the transient response PDF of nonlinear oscillators with fractional derivative elements under Gaussian white noise. First, an equivalent linear system is introduced for the purpose of deriving the Fokker–Planck (FP) equation for response amplitude. The equivalent natural frequency and equivalent damping coefficient of the system need to be determined, taking into account both the nonlinear and fractional derivative elements of the original oscillator. Moreover, to convert the FP equation into the governing equation of CFMs, these equivalent coefficients must be given in polynomial form of amplitude. This paper proposes formulas for appropriately determining the equivalent coefficients, based on an equivalent linearization technique. Then, applying stochastic averaging, the FP equation is derived from the equivalent linear system. Next, the Mellin transform converts the FP equation into coupled linear ordinary differential equations for amplitude CFMs, which are solved with a constraint corresponding to the normalization condition for a PDF. Finally, the inverse Mellin transform of the CFMs yields the amplitude PDF. The joint PDF of displacement and velocity is also obtained from the amplitude PDF. Three linear and nonlinear fractional oscillators are considered in numerical examples. For all cases, the analytical results are in good agreement with the pertinent Monte Carlo simulation results.

Stochastic Averaging Principle for Two-Time-Scale SDEs with Distribution-Dependent Coefficients Driven by Fractional Brownian Motion

Stochastic Averaging Principle for Two-Time-Scale SDEs with Distribution-Dependent Coefficients Driven by Fractional Brownian Motion

An Improved Stochastic Averaging on the Piezoelectric Vibrational Harvester Model with Stiffness and Inertia Nonlinearities

In this paper, an improved stochastic averaging method is introduced to solve a monostable piezoelectric vibration energy harvester (VEH) model subject to basal Gaussian white and colored noises. First, the electric voltage is expressed as perturbations to the linear stiffness term and the damping term of the vibrational equation. Second, the original model is simplified into an Itô differential equation by the improved stochastic averaging method. Third, stationary solution of the Fokker–Plank–Kolmogorov function driven from the Itô equation is obtained, so that the stationary probability density functions (PDFs) of the displacement, velocity, and electrical voltage are obtained. With the help of the Monte Carlo, the effectiveness of these theoretical predictions is verified. Finally, the impacts of the nonlinear stiffness coefficient and nonlinear inertia coefficient on the responses are compared. It is found that the stiffness nonlinearity has stronger effect on the outputn electric voltage than the inertia nonlinearity.

A Dual Averaging Algorithm for Online Modeling of Infinite Memory Nonlinear Systems

An on-line modelling algorithm is derived from a generic stochastic Dual Averaging (DA) method. It employs a negative entropy as a distance-generating function and the Volterra series expansion as a dictionary. Assuming that the measurement data are not <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.i.d.</i> but generated by a nonlinear dynamical system with an infinite, exponentially fading memory, the error bounds are established for both the generic DA method and for the proposed modelling algorithm. The experiments performed on a set of benchmark systems confirm the applicability of the algorithm in real-world scenarios and demonstrate its low computational complexity.

A numerically efficient output-only system-identification framework for stochastically forced self-sustained oscillators

Self-sustained oscillations are ubiquitous in nature and engineering. In this paper, we propose a novel output-only system-identification framework for identifying the system parameters of a self-sustained oscillator affected by Gaussian white noise. A Langevin model that characterizes the self-sustained oscillator is postulated, and the corresponding Fokker–Planck equation is derived from stochastic averaging. From the drift and diffusion terms of the Fokker–Planck equation, unknown parameters of the system are identified. We develop a numerically efficient algorithm for enhancing the accuracy of parameter identification. In particular, a modified Levenberg–Marquardt optimization algorithm tailored to output-only system identification is introduced. The proposed framework is demonstrated on both numerical and experimental oscillators with varying system parameters that develop into self-sustained oscillations. The results show that the computational cost required for performing the system identification is dramatically reduced by using the proposed framework. Also, system parameters that were difficult to be extracted with the existing method could be efficiently computed with the system identification method developed in this study. Pertaining to the robustness and computational efficiency of the presented framework, this study can contribute to an accurate and fast diagnosis of dynamical systems under stochastic forcing.

Data-Driven Stochastic Averaging

Abstract Stochastic averaging, as an effective technique for dimension reduction, is of great significance in stochastic dynamics and control. However, its practical applications in industrial and engineering fields are severely hindered by its dependence on governing equations and the complexity of mathematical operations. Herein, a data-driven method, named data-driven stochastic averaging, is developed to automatically discover the low-dimensional stochastic differential equations using only the random state data captured from the original high-dimensional dynamical systems. This method includes two successive steps, that is, extracting all slowly varying processes hidden in fast-varying state data and identifying drift and diffusion coefficients by their mathematical definitions. It automates dimension reduction and is especially suitable for cases with unavailable governing equations and excitation data. Its application, efficacy, and comparison with theory-based stochastic averaging are illustrated through several examples, numerical or experimental, with pure Gaussian white noise excitation or combined excitations.

A Dynamic Behavior Analysis of a Rolling Mill’s Main Drive System with Fractional Derivative and Stochastic Disturbance

Taking the random factors into account, a fractional main drive system of a rolling mill with Gaussian white noise is developed. First, the potential deterministic bifurcation is investigated by a linearized stability analysis. The results indicate that the fractional order changes the system from a stable point to a limit cycle with symmetric phase trajectories. Then, the stochastic response is obtained with the aid of the equivalent transformation of the fractional derivative and stochastic averaging methods. It is found that the joint stationary probability density function appears to have symmetric distribution. Finally, the influence of the fractional order and noise intensity on system dynamics behavior is discussed. The study is beneficial to understand the intrinsic mechanisms of vibration abatement.

Illumination Adaptive Multi-Scale Water Surface Object Detection with Intrinsic Decomposition Augmentation

Visual object detection is an essential task for the intelligent navigation of an Unmanned Surface Vehicle (USV), which can sense the obstacles while navigating. However, the harsh illumination conditions and large scale variation of the objects significantly harm the performance of object detection methods. To address the above problems, we propose a robust water surface object detection method named multi-scale feature fusion network with intrinsic decomposition generative adversarial network data augmentation (MFFDet-IDGAN). We introduce intrinsic decomposition as data augmentation for the object detection to achieve illumination adapting. And an intrinsic decomposition generative adversarial network (IDGAN) is proposed to achieve unsupervised intrinsic decomposition. Moreover, the multi-scale feature fusion network (MFFDet) adopts an improved bidirectional feature pyramid network (BiFPN) and spatial pyramid pooling (SPP) blocks to fuse features of different resolution for better multi-scale detection. And an improved weighted stochastic weight averaging (SWA) is proposed and applied in the training process to improve the generalization performance. We conduct extensive experiments on the Water Surface Object Detection Dataset (WSODD), and the results show that the proposed method can achieve 44% improvement over the baseline. And we further test our method on a real USV in the sailing process, the results show that our method can exceeding the baseline by 4.5%.

On the Nonlinear Stochastic Dynamics of an Atomic Force Microscope Cantilever

AbstractAtomic force microscopy (AFM) serves characterization and actuation in nanoscale applications. We study the stochastic dynamics of an AFM cantilever under tip-sample interactions represented by the Lennard–Jones and Morse potential energy functions. In both cases, we also study the contrasting dynamic effects of additive (external) and multiplicative (internal) noise. Moreover, for multiplicative noise, we study the two sub-cases arising from the Itô and Stratonovich interpretations of stochastic integrals. In each case, we also investigate the stochastic stability of the system by tracing the time evolution of the maximal Lyapunov exponent. Additionally, we obtain stationary probability densities for the unforced dynamics using stochastic averaging.

Electrical Response Estimation of Vibratory Energy Harvesters via Hilbert Transform Based Stochastic Averaging

Abstract Converting mechanical vibrations into electrical power with vibratory energy harvesters can ensure the portability, efficiency, and sustainability of electronic devices and batteries. Vibratory energy harvesters are typically modeled as nonlinear oscillators subject to random excitation, and their design requires a complete characterization of their probabilistic responses. However, simulation techniques such as Monte Carlo are computationally prohibitive when the accurate estimation of the response probability distribution is needed. Alternatively, approximate methods such as stochastic averaging can estimate the probabilistic response of such systems at a reduced computational cost. In this paper, the Hilbert transform based stochastic averaging is used to model the output voltage amplitude as a Markovian stochastic process with dynamics governed by a stochastic differential equation with nonlinear drift and diffusion terms. Moreover, the voltage amplitude dependent damping and stiffness terms are determined via an appropriate equivalent linearization, and the stationary probability distribution of the output voltage amplitude is obtained analytically by solving the corresponding Fokker–Plank equation. Two examples are used to demonstrate the accuracy of the obtained analytical probability distributions via comparisons with Monte Carlo simulation data.

A Histopathological Image Classification Method Based on Model Fusion in the Weight Space

Automatic classification of histopathological images plays an important role in computer-aided diagnosis systems. The automatic classification model of histopathological images based on deep neural networks has received widespread attention. However, the performance of deep models is affected by many factors, such as training hyperparameters, model structure, dataset quality, and training cost. In order to reduce the impact of the above factors on model training and reduce the training and inference costs of the model, we propose a novel method based on model fusion in the weight space, which is inspired by stochastic weight averaging and model soup. We use the cyclical learning rate (CLR) strategy to fine-tune the ingredient models and propose a ranking strategy based on accuracy and diversity for candidate model selection. Compared to the single model, the weight fusion of ingredient models can obtain a model whose performance is closer to the expected value of the error basin, which may improve the generalization ability of the model. Compared to the ensemble model with n base models, the testing cost of the proposed model is theoretically 1/n of that of the ensemble model. Experimental results on two histopathological image datasets show the effectiveness of the proposed model in comparison to baseline ones, including ResNet, VGG, DenseNet, and their ensemble versions.

Stationary random response evaluation of nonlinear oscillators with Preisach hysteresis discretized by RNN model

This manuscript is devoted to evaluating random stationary responses of nonlinear oscillators including Preisach hysteresis, which is modeled by a powerful tool, viz. Recurrent Neural Networks (RNN). First, a generalized harmonic transformation is introduced by adopting a geometric viewpoint of RNN-described hysteresis and intentionally separating the hysteretic force into conservative and dissipative components, to circumvent direct mathematical calculations of amounts of difference equations. The hysteretic force could be equivalently mimicked by a force that only depends on the present system states. Then, the system dimension is reduced by stochastic averaging, which is derived from the corresponding low-dimensional Fokker–Planck–Kolmogorov (FPK) equation associated with the probability density of slow-varying process. Additionally, solving the FPK equation yields the stationary probability density as well as various-order response statistics. Finally, two numerical examples, i.e., Duffing oscillator and van der Pol oscillator, are investigated to verify the efficacy of the proposed procedure.

Stochastic averaging for a completely integrable Hamiltonian system with fractional Brownian motion

This paper proposes an effective approximation result for the behavior of a small transversal perturbation to a completely integrable stochastic Hamiltonian system on a symplectic manifold. We derive an averaged stochastic differential equations (SDEs) in the action space for the action component of the perturbed system, where the averaged drift coefficient is characterized by the averages of that of the action component with respect to the invariant measure of the unperturbed system on the corresponding invariant manifolds. Then, the averaging principle is shown to be valid such that the action component of the perturbed system converges to the solution of averaged SDEs in the mean square sense.

Performance enhancement of snap-through vibration energy harvester with displacement amplifier

Vibration energy harvesters (VEH) based on bistable and snap-through mechanisms are being widely employed to harvest energy from different vibration sources. However, under weak ambient excitation, the performance of these systems is marginal. Multi-stability and enhancement techniques, such as displacement amplifier, are being used with bistable VEH to improve its performance. Similar to this, multi-stable snap-through VEH has also been developed; however, the usage of a displacement amplifier in the context of snap-through VEH has not yet been investigated. In this novel study, a displacement amplifier/ displacement amplification mechanism (DAM) is integrated into a snap-through VEH, and the enhanced performance under weak excitation is investigated. A generalized transformation is applied to develop a reduced-order model of the proposed system, and its dynamics and performance under harmonic and random excitation are investigated analytically and numerically. The frequency–amplitude relationship is derived under harmonic excitation, and a detailed investigation of the influence of system parameters on the frequency and force responses is performed. It is observed that by adjusting the mass and stiffness ratio between the VEH and DAM, it is possible to harvest more energy under harmonic excitation of low intensity. In the case of Gaussian white noise excitation, effective potential, and stochastic averaging methods are used to obtain marginal, joint, and stationary probability density functions and mean square power. The influence of noise intensity and other system parameters on the aforementioned measures is thoroughly investigated. It is observed that for specific parameter values, the proposed VEH can harvest 3.3 times more power under harmonic excitation and four times more power under random excitation compared to the snap-through VEH without DAM.

Stochastic time-varying extremum seeking and its applications

We investigate a stochastic time-varying extremum seeking method based on new stochastic averaging theorems for time-varying systems and its application to distributed source seeking with switching topology. Firstly, we present sufficient conditions for the approximation of the time-varying averaging system to the time-varying original system with stochastic perturbations and obtain the solution property of the original system under the stability of invariant sets of the average system. Then, by using our developed stochastic averaging theory, we prove that the proposed stochastic time-varying extremum seeking algorithm converges to the time-varying optimal solution. Furthermore, based on the method of stochastic time-varying extremum seeking, we propose a distributed stochastic source seeking algorithm which navigates multiple vehicles to seek the source of aggregated multiple signal fields under a switching topology. We can prove the exponential convergence of such a distributed algorithm by our stochastic averaging theorems and a Lyapunov-based approach. Finally, we present detailed simulation results to illustrate the effectiveness of our method.