Small Step Size Research Articles

True online temporal-difference learning

The temporal-difference methods TD(λ) and Sarsa(λ) form a core part of modern reinforcement learning. Their appeal comes from their good performance, low computational cost, and their simple interpretation, given by their forward view. Recently, new versions of these methods were introduced, called true online TD(λ) and true online Sarsa(λ), respectively (van Seijen & Sutton, 2014). Algorithmically, these true online methods only make two small changes to the update rules of the regular methods, and the extra computational cost is negligible in most cases. However, they follow the ideas underlying the forward view much more closely. In particular, they maintain an exact equivalence with the forward view at all times, whereas the traditional versions only approximate it for small step-sizes. We hypothesize that these true online methods not only have better theoretical properties, but also dominate the regular methods empirically. In this article, we put this hypothesis to the test by performing an extensive empirical comparison. Specifically, we compare the performance of true online TD(λ)/Sarsa(λ) with regular TD(λ)/Sarsa(λ) on random MRPs, a real-world myoelectric prosthetic arm, and a domain from the Arcade Learning Environment. We use linear function approximation with tabular, binary, and non-binary features. Our results suggest that the true online methods indeed dominate the regular methods. Across all domains/representations the learning speed of the true online methods are often better, but never worse than that of the regular methods. An additional advantage is that no choice between traces has to be made for the true online methods. Besides the empirical results, we provide an in-dept analysis of the theory behind true online temporal-difference learning. In addition, we show that new true online temporal-difference methods can be derived by making changes to the online forward view and then rewriting the update equations.

An energy-stable generalized-α method for the Swift–Hohenberg equation

We propose a second-order accurate energy-stable time-integration method that controls the evolution of numerical instabilities introducing numerical dissipation in the highest-resolved frequencies. Our algorithm further extends the generalized- α method and provides control over dissipation via the spectral radius. We derive the first and second laws of thermodynamics for the Swift–Hohenberg equation and provide a detailed proof of the unconditional energy stability of our algorithm. Finally, we present numerical results to verify the energy stability and its second-order accuracy in time. • We present a second-order energy-stable time-integration method for the Swift–Hohenberg equation that suppresses numerical instabilities. • Based on the generalized- α method, numerical dissipation is controlled via the spectral radius. • A detailed energy-stability proof and an estimate for the stabilization parameter are provided. • The proposed stabilization vanishes for sufficiently small time step sizes, recovering the original weak form of the equation. • Numerical results for a pattern-formation example are used to test the convergence and energy-stability. • We compare the primal and mixed formulation in terms of wall-clock time and approximation error.

A Numerical Algorithm for the Coupled PDEs Control Problem

For the coupled PDE control problem, at time \(t_i\) with the ith point, the standard algorithm will first obtain the two space variables \((z_i,v_i)\) and then obtain the control variables \((\varsigma _i^{opt},\mu _i^{opt})\) from the given initial points \((\varsigma _i^0,\mu _i^0)\). How many points i are determined by the facts of the case? We usually believe that the largest i defined by n is big because the small step size \(\tau =\frac{T-t_0}{n}\) will generate a good approximation, where T denotes the terminal time. Thus, the solution process is very tedious, and much CPU time is required. In this paper, we present a new method to overcome this drawback. This presented method, which fully utilizes the first-order conditions, simultaneously considers the two space variables \((z_i,v_i)\) and the control variables \((\varsigma _i^{opt},\mu _i^{opt})\) with \(t_i\) at i. The computational complexity of the new algorithm is \(O(N^3)\), whereas that of the normal algorithm is \(O(N^3+N^3K)\). The performance of the proposed algorithm is tested using an example.

On the stability of projection methods for the incompressible Navier–Stokes equations based on high-order discontinuous Galerkin discretizations

The present paper deals with the numerical solution of the incompressible Navier–Stokes equations using high-order discontinuous Galerkin (DG) methods for discretization in space. For DG methods applied to the dual splitting projection method, instabilities have recently been reported that occur for small time step sizes. Since the critical time step size depends on the viscosity and the spatial resolution, these instabilities limit the robustness of the Navier–Stokes solver in case of complex engineering applications characterized by coarse spatial resolutions and small viscosities. By means of numerical investigation we give evidence that these instabilities are related to the discontinuous Galerkin formulation of the velocity divergence term and the pressure gradient term that couple velocity and pressure. Integration by parts of these terms with a suitable definition of boundary conditions is required in order to obtain a stable and robust method. Since the intermediate velocity field does not fulfill the boundary conditions prescribed for the velocity, a consistent boundary condition is derived from the convective step of the dual splitting scheme to ensure high-order accuracy with respect to the temporal discretization. This new formulation is stable in the limit of small time steps for both equal-order and mixed-order polynomial approximations. Although the dual splitting scheme itself includes inf–sup stabilizing contributions, we demonstrate that spurious pressure oscillations appear for equal-order polynomials and small time steps highlighting the necessity to consider inf–sup stability explicitly.

A novel combination scheme of proportionate filter

The conventional convex combination schemes with two proportionate filters have high computational burdens due to the two proportionate filters running at the same time. Meanwhile, they also have poor convergence behavior at the intersection of the large step size filter and the small step size filter. To overcome these drawbacks, this paper proposes a family of combined-step-size (CSS) proportionate filters. The proposed CSS proportionate filters use a variable mixing factor to adaptively combine two different step sizes of one proportionate filter, where the large step size affords a fast convergence rate and the small one offers a small steady-state error. The variable mixing factor is defined as the output of a modified sigmoidal activation function and it updates indirectly by using the stochastic gradient descent method to minimize the L1-norm of the system output error. The proposed CSS proportionate filters have lower computational complexities than the corresponding convex combination proportionate filters because they only require one filter run at every moment. Simulations in three echo channels with different degrees of sparsity have demonstrated the superior convergence performance of this family of CSS proportionate filters.

DETERMINING PLANE-SWEEP SAMPLING POINTS IN IMAGE SPACE USING THE CROSS-RATIO FOR IMAGE-BASED DEPTH ESTIMATION

Abstract. With the emergence of small consumer Unmanned Aerial Vehicles (UAVs), the importance and interest of image-based depth estimation and model generation from aerial images has greatly increased in the photogrammetric society. In our work, we focus on algorithms that allow an online image-based dense depth estimation from video sequences, which enables the direct and live structural analysis of the depicted scene. Therefore, we use a multi-view plane-sweep algorithm with a semi-global matching (SGM) optimization which is parallelized for general purpose computation on a GPU (GPGPU), reaching sufficient performance to keep up with the key-frames of input sequences. One important aspect to reach good performance is the way to sample the scene space, creating plane hypotheses. A small step size between consecutive planes, which is needed to reconstruct details in the near vicinity of the camera may lead to ambiguities in distant regions, due to the perspective projection of the camera. Furthermore, an equidistant sampling with a small step size produces a large number of plane hypotheses, leading to high computational effort. To overcome these problems, we present a novel methodology to directly determine the sampling points of plane-sweep algorithms in image space. The use of the perspective invariant cross-ratio allows us to derive the location of the sampling planes directly from the image data. With this, we efficiently sample the scene space, achieving higher sampling density in areas which are close to the camera and a lower density in distant regions. We evaluate our approach on a synthetic benchmark dataset for quantitative evaluation and on a real-image dataset consisting of aerial imagery. The experiments reveal that an inverse sampling achieves equal and better results than a linear sampling, with less sampling points and thus less runtime. Our algorithm allows an online computation of depth maps for subsequences of five frames, provided that the relative poses between all frames are given.

Influence of Scanning Parameters on X-Ray Diffraction Peaks of Copper

The aim of this work is to study the influence of x-ray diffractometer scanning parameters on the integrated intensity and full-width at half maximum (FWHM) of copper powder x-ray diffraction peaks by using statistical analysis methods. X-ray diffraction (XRD) analysis of the copper powder was accomplished using step scan mode with step sizes of 0.03o and 0.05o 2q, and preset time changes from 0.1-3.5 s. Integrated intensity of an x-ray peak was calculated by the numerical method. FWHM was measured as the width of Pearson VII model of the x-ray peak at the half-maximum intensity. The statistical analysis methods including linear regression and statistical hypothesis test were used to analyze the correlation between the preset time and the error on integrated intensity calculation, and the FWHM of a peak on the XRD pattern. The results from statistical analysis show that increasing the preset time from 0.1 s to 3.5 s does not affect the FWHM of an x-ray peak, but it reduces the relative error in integrated intensity calculation. Moreover, using the preset time greater than 1 s will minimize the relative error in integrated intensity calculation of an x-ray peak. While step size did not affect both the relative error in integrated intensity calculation or FWHM, the smaller step size would provide more data points for better accurate model of an x-ray peak.

Randomized Hamiltonian Monte Carlo

Tuning the durations of the Hamiltonian flow in Hamiltonian Monte Carlo (also called Hybrid Monte Carlo) (HMC) involves a tradeoff between computational cost and sampling quality, which is typically challenging to resolve in a satisfactory way. In this article we present and analyze a randomized HMC method (RHMC), in which these durations are i.i.d. exponential random variables whose mean is a free parameter. We focus on the small time step size limit, where the algorithm is rejection-free and the computational cost is proportional to the mean duration. In this limit, we prove that RHMC is geometrically ergodic under the same conditions that imply geometric ergodicity of the solution to underdamped Langevin equations. Moreover, in the context of a multi-dimensional Gaussian distribution, we prove that the sampling efficiency of RHMC, unlike that of constant duration HMC, behaves in a regular way. This regularity is also verified numerically in non-Gaussian target distributions. Finally we suggest variants of RHMC for which the time step size is not required to be small.

A nonnegativity preserved efficient chemical solver applied to the air pollution forecast

Air pollution forecast is becoming more and more important nowadays. The numerically sticky chemical ordinary differential equations (ODEs) is a critical component of air pollution models. Various solvers have been designed for the chemical ODEs in the past. However, they are either slow or imprecise. In our previous work, we have designed a nonnegativity preserved efficient chemical solver MBE, which is an acronym for Modified-Backward-Euler. In this paper, we review MBE method and prove its convergence and stability mathematically, which guarantee that MBE results converge to the exact solutions as the step-size becomes smaller and MBE results with relatively small step-size can be used as the standard. Then we apply MBE to the Nested Air Quality Prediction Modeling System (NAQPMS). Comparison between MBE and the most popular solver LSODE is also made. Considering the speed and precision, MBE is a better choice for the air pollution forecast.

Harmonics elimination PWM based direct control for 23-level multilevel distribution STATCOM using differential evolution algorithm

This work describes the application of a soft computing technique—known as the differential evolution (DE) to implement the harmonics elimination pulse-width modulation (HEPWM) for the direct control of distribution static synchronous compensator (D-STATCOM). The main contribution of the work is the application of DE to solve eleven HEPWM switching angles to synthesize a 23-level cascaded multilevel voltage source inverter (MVSI). The key feature of this algorithm is the wide modulation index (MI) range that can be produced (i.e. 5.40–8.15p.u) with very small step size (0.01), while maintaining the total harmonic distortion (THD) of the MVSI output voltage below 5%. Furthermore, the utilization of high levels MVSI allows for the output voltage of the D-STATCOM to be sufficiently high, thus avoiding the use of step-up transformer. The correctness of the computed HEPWM angles is validated using single phase 23-level cascaded MVSI experimental rig. Then, the proposed method is compared to the phase-shifted PWM (PS-PWM) using a ±6.5 MVAr/11kV D-STATCOM modelled in MATLAB-Simulink. For the same switching frequency, the HEPWM switching results in a superior harmonics spectra and lower switching losses. Furthermore, if the switching frequency is fixed to a particular value, the size of the series coupling inductor can be reduced to at least half. Dynamically, the steady state value of the reactive current (iCq) is reached in less than one mains cycle when a transition from the full inductive to full capacitive modes is imposed.

Effect of Testing Factors on Rietveld Quantitative Analysis of Cement-Matrix Materials

For cement-matrix materials, the microstructure plays a vital important role in the research. Recently, quantitative phase analysis of cementitious materials can be performed using the Rietveld method by fitting the calculated X-ray diffraction (XRD) profile with the observed one. The aim of this paper is to further perform the quantitative analysis by the Rietveld method and discuss the influence of testing factors on the Rietveld quantitative phase analysis. The factors included the collection range of pattern, step size and the scan time of per step. In this study, the chemical composition of the samples was determined by X-ray fluorescence (XRF) spectrometry. And their phase composition was calculated by X-ray powder diffraction and Rietveld analysis. The results showed that the collection range of pattern depended on the tested materials , and the scanning range should include the main diffraction peak of the sample. Smaller step size and longer scan time of each step made the fitting factor smaller, also the calculated pattern coincided with the measured pattern, better enhance the precision of the analyses.

Hybrid spiral-dynamic bacteria-chemotaxis algorithm with application to control two-wheeled machines

This paper presents the implementation of the hybrid spiral-dynamic bacteria-chemotaxis (HSDBC) approach to control two different configurations of a two-wheeled vehicle. The HSDBC is a combination of bacterial chemotaxis used in bacterial forging algorithm (BFA) and the spiral-dynamic algorithm (SDA). BFA provides a good exploration strategy due to the chemotaxis approach. However, it endures an oscillation problem near the end of the search process when using a large step size. Conversely; for a small step size, it affords better exploitation and accuracy with slower convergence. SDA provides better stability when approaching an optimum point and has faster convergence speed. This may cause the search agents to get trapped into local optima which results in low accurate solution. HSDBC exploits the chemotactic strategy of BFA and fitness accuracy and convergence speed of SDA so as to overcome the problems associated with both the SDA and BFA algorithms alone. The HSDBC thus developed is evaluated in optimizing the performance and energy consumption of two highly nonlinear platforms, namely single and double inverted pendulum-like vehicles with an extended rod. Comparative results with BFA and SDA show that the proposed algorithm is able to result in better performance of the highly nonlinear systems.

A GPU-Accelerated Fourth-Order Runge–Kutta in the Interaction Picture Method for the Simulation of Nonlinear Signal Propagation in Multimode Fibers

The nonlinear signal propagation in fibers can be described by the nonlinear Schrodinger equation and the Manakov equation. Most commonly, split-step Fourier methods (SSFM) are applied to solve these nonlinear equations. The numerical simulation of the nonlinear signal propagation is especially challenging for multimode fibers, particularly if the calculation of very small step sizes or a large number of steps is required. Instead of utilizing SSFM, the fourth-order Runge–Kutta in the Interaction Picture (RK4IP) method can be applied. This method has the potential to reduce the numerical error while simultaneously allowing an increased step size. These advantages come at the price of a higher numerical effort compared to the SSFM method for the same step size. Since the simulation of the signal propagation in multimode fibers is already quite challenging, parallelization becomes an even more interesting option. We demonstrate the adaptation of the RK4IP method to simulate the nonlinear signal propagation in multimode fibers, including its parallelization. Besides comparing the performance of a parallelized implementation for multicore CPUs and a GPU-accelerated version, we discuss efficient strategies to implement the RK4IP method on a GPU accelerator with CUDA. In addition, the RK4IP implementation is numerically compared with a conventional SSFM implementation.

A new combined-step-size normalized least mean square algorithm for cyclostationary inputs

For cyclostationary input signals, the normalized least mean square (NLMS) algorithm suffers from large steady-state errors. The average mean square deviation (MSD) analysis of the NLMS and least mean square (LMS) algorithms shows that NLMS has good transient response which is independent of the reference inputs, whereas the steady-state MSD of LMS does not depend on the periodic input power. In this paper, therefore, the combined-step-size NLMS (CSSNLMS) algorithm is proposed to reduce steady-state misalignment as compared to those in the conventional variable step-size (VSS) NLMS algorithms while achieving similar convergence rate for cyclostationary input signals. The proposed CSSNLMS algorithm combines and utilizes the different performances of the NLMS and LMS algorithms, in which NLMS is with large step-size and LMS uses small step-size. The mixing parameter is indirectly adjusted by use of the shrinkage denoising method in accordance with the estimated noise-free a priori error. The mean square performance analysis indicates that the proposed CSSNLMS algorithm can achieve the merits of NLMS and LMS under cyclostationary inputs. Finally, simulation results on system identification and echo cancellation verify the theoretical analysis and the efficiency of the proposed algorithm.

Implementation of a novel projector-splitting integrator for the multi-configurational time-dependent Hartree approach.

The variational equations of motion of the multi-configuration time-dependent Hartree (MCTDH) approach contain the inverse of reduced density matrices which are typically ill-conditioned and therefore lead to small stepsizes for numerical time integration. This problem is usually dealt with via regularization of the density matrices, which works well in many cases but still calls for systematic improvement schemes. Recently this problem, its implications and possible solutions have become the subject of increased interest. Notably, a projector splitting integrator for the MCTDH approach that does not require the inversion of reduced density matrices has been proposed [C. Lubich, Appl. Math. Res. Express 2015, 311]. Here, we present the first implementation of this integration scheme. Results for low-dimensional benchmark systems are presented, and the case of initially unoccupied single-particle functions is discussed.

Distributed methods for synchronization of orthogonal matrices over graphs

This paper addresses the problem of synchronizing orthogonal matrices over directed graphs. For synchronized transformations (or matrices), composite transformations over loops equal the identity. We formulate the synchronization problem as a least-squares optimization problem with nonlinear constraints. The synchronization problem appears as one of the key components in applications ranging from 3D-localization to image registration. The main contributions of this work can be summarized as the introduction of two novel algorithms; one for symmetric graphs and one for graphs that are possibly asymmetric. Under general conditions, the former has guaranteed convergence to the solution of a spectral relaxation to the synchronization problem. The latter is stable for small step sizes when the graph is quasi-strongly connected. The proposed methods are verified in numerical simulations.

Numerical study of fouling characteristics on two kinds of typical heat exchangers used in the waste heat recovery systems

The problem of high exhaust temperature universally exists around the industrial boilers, which causes enormous loss of energy. Different kinds of heat exchangers are widely used in the waste heat recovery systems to recover waste energy from different heat sources. However, considering the heat exchangers mainly work in dusty environment, such as in the tail of large boiler, fly ash fouling on heat exchangers is a major problem in waste heat recovery systems, which can lead to significant efficiency deterioration. Besides, the fouling may also lead to failure of the systems with serious economic consequence. The H-type finned tube heat exchanger is a new type of heat exchanger which have been widely used in waste heat recovery systems in recent years, because of their excellent anti-wear and anti-fouling performance. In this paper, a numerical model was developed to investigate the fouling characteristics of two kinds of H-type finned tube heat exchangers used in the waste heat recovery systems. Firstly, a fouling model which contains a deposition process and a removal process is developed. The deposition process is determined by the energy balance on the particles and the removal process is determined by the the local wall shear stress and the fouling thickness on the fin surface. As the flow field directly affects the particle motion, the fluid flow characteristics of the two kinds of heat exchangers are calculated, and the discrete phase model (DPM) was employed to trace the fly ash particles in the flow field during the fouling process. Since the simulation time is much smaller in magnitude than the real time in fouling process, it is impossible to simulate the whole real fouling time period (hours) with a small time step size. In order to obtain a reliable result, it is necessary to enlarge the simulation time and results to the real time scale. In this paper, a fouling time ratio is proposed by considering the volume difference between the simulated deposit layer and the real deposited particles and the concentration difference of injected particles between the simulation and real situation. According to the fouling time ratio, we can enlarge the simulation time and results to the real time scale. Then the real-time fouling characteristics of the two kind of H-type finned tube heat exchangers are investigated and the effects of inlet velocity, particle diameter on the fouling results are also studied. The results show that, the fouling mainly occurred in the flow stagnation region and the recirculation region of the H-type finned tube. The net fouling mass increases with the operating time, showing an asymptotic characteristic of growth, and finally will reach a stable value, which corresponds to the asymptotic fouling resistance. The increase in inlet velocity and particle diameter would lead to a rapidly decrease in fouling results. The anti-fouling performance of double H-type finned tubes shows slightly better than that of single ones, but the advantages decreases with the increase in inlet velocity and particle diameter.

Three rapidly convergent parareal solvers with application to time‐dependent PDEs with fractional Laplacian

Time‐dependent PDEs with fractional Laplacian ( − Δ)α play a fundamental role in many fields and approximating ( − Δ)α usually leads to ODEs' system like u′(t) + Au(t) = g(t) with A = Qα, where is a sparse symmetric positive definite matrix and α > 0 denotes the fractional order. The parareal algorithm is an ideal solver for this kind of problems, which is iterative and is characterized by two propagators and . The propagators and are respectively associated with large step size ΔT and small step size Δt, where ΔT = JΔt and J⩾2 is an integer. If we fix the ‐propagator to the Implicit‐Euler method and choose for some proper Runge–Kutta (RK) methods, such as the second‐order and third‐order singly diagonally implicit RK methods, previous studies show that the convergence factors of the corresponding parareal solvers can satisfy and , where σ(A) is the spectrum of the matrix A. In this paper, we show that by choosing these two RK methods as the ‐propagator, the convergence factors can reach , provided the one‐stage complex Rosenbrock method is used as the ‐propagator. If we choose for both and , the complex Rosenbrock method, we show that the convergence factor of the resulting parareal solver can also reach . Numerical results are given to support our theoretical conclusions. Copyright © 2017 John Wiley & Sons, Ltd.

Dynamic analysis of an impact load applied to the composite wall structure

In presented paper the composite wall structure made of concrete and Zahorski/Mooney Rivlin rubber cross pads subjected to impact loads was investigated. Mentioned rubber cross pads were assumed to be inserted or injected in specially formed holes during the production process. Moreover, two different impact loads were taken into further considerations - pressure and concentrated force. Such impact loads due to the small time step size allowed observation of mechanical wave propagation in analyzed structure in the form of stress plots. On the basis of composite wall comparison with the solid concrete wall it could be stated that the mechanical wave propagation was significantly damped via inserted/injected rubber cross pads. Discussed problem was solved with the use of finite element method using ADINA program.

Graded nanowell arrays: a fine plasmonic "library" with an adjustable spectral range.

We present an effective approach for fabricating graded plasmonic arrays based on ordered micro-/nanostructures with a geometric gradient. Ag nanowell arrays with graded geometric parameters were fabricated and systematically investigated. The order of the graded plasmonic arrays is generated by colloidal lithography, while the geometric gradient is the result of inclined reactive ion etching. The surface plasmon resonance (SPR) peaks were measured at different positions, which move gradually along the Ag nanowell arrays with a geometric gradient. Such micro-/nanostructure arrays with graded and integrated SPR peaks can work as a fine plasmonic "library" (FPL), and the spectral range can be controlled using a "coarse adjustment knob" (lattice constant) and a "fine adjustment knob" (pore diameter). Additionally, the spectral resolution of the FPL is high, which benefits from the high value of the full height/full width at half-maximum and the small step size of the wavelength shift (0.5 nm). Meanwhile, the FPL could be effectively applied as a well-defined model to verify the plasmonic enhancement in surface enhanced Raman scattering. As the FPL is an integrated optical material with graded individual SPR peaks, it can not only be a theoretical model for fundamental research, but also has great potential in high-throughput screening of optical materials, multiplex sensors, etc.