Nonuniform Stepsizes Research Articles

FAT: Frequency-Aware Transformation for Bridging Full-Precision and Low-Precision Deep Representations.

Learning low-bitwidth convolutional neural networks (CNNs) is challenging because performance may drop significantly after quantization. Prior arts often quantize the network weights by carefully tuning hyperparameters such as nonuniform stepsize and layerwise bitwidths, which are complicated since the full- and low-precision representations have large discrepancies. This work presents a novel quantization pipeline, named frequency-aware transformation (FAT), that features important benefits: 1) instead of designing complicated quantizers, FAT learns to transform network weights in the frequency domain to remove redundant information before quantization, making them amenable to training in low bitwidth with simple quantizers; 2) FAT readily embeds CNNs in low bitwidths using standard quantizers without tedious hyperparameter tuning and theoretical analyses show that FAT minimizes the quantization errors in both uniform and nonuniform quantizations; and 3) FAT can be easily plugged into various CNN architectures. Using FAT with a simple uniform/logarithmic quantizer can achieve the state-of-the-art performance in different bitwidths on various model architectures. Consequently, FAT serves to provide a novel frequency-based perspective for model quantization.

Three-dimensional forward calculation of time domain electromagnetic response stimulated by a pseudo-Gaussian pulse source

The calculation of three-dimensional (3D) electromagnetic responses using the controlled source time domain electromagnetic method suffers from low accuracy of the early response calculation and difficulty in the convergence of the later response for large-scale models. To address these problems, we propose a method of decomposing the time domain electromagnetic response into a primary field and a secondary field. This approach can easily address the above issues. In this paper, we use a time domain finite difference method based on a non-uniform step size and staggered grid and use an iterative algorithm with a non-uniform step size to calculate the late time domain electromagnetic response under high power pulse source excitation. Several example simulations were compared with analytical and spectral Lanczos decomposition solutions and demonstrate the correctness of the 3D forward modeling method. The response of a 3D anomaly excited by an electric dipole source pulse current was analyzed. The calculated time domain response revealed that the field excited by the high power pulse source diffused in the earth over time and exhibited a secondary diffusion process of the anomalous field generated by the 3D anomaly. The numerical simulation results showed that the time domain electromagnetic responses excited by the pulse source were stable and effective. The amplitude of the anomalous response was much greater than that of the step source, the signal-to-noise ratio was greater, and the detectability was better. The method may be used in inversion of the time domain electromagnetic response if the calculation speed can be further improved.

On the Accelerated Convergence of the Decentralized Event-triggered Algorithm for Convex Optimization

This article considers a problem of solving the optimal solution of the sum of locally convex cost functions over an undirected network. Each local convex cost function in the network is accessed only by each unit. To be able to reduce the amount of computation and get the desired result in an accelerated way, we put forward a fresh accelerated decentralized event-triggered algorithm, named as A-DETA, for the optimization problem. A-DETA combines gradient tracking and two momentum accelerated terms, adopts nonuniform step-sizes and emphasizes that each unit interacts with neighboring units independently only at the sampling time triggered by the event. On the premise of assuming the smoothness and strong convexity of the cost function, it is proved that A-DETA can obtain the exact optimal solution linearly in the event of sufficiently small positive step-size and momentum coefficient. Moreover, an explicit linear convergence speed is definitely shown. Finally, extensive simulation example validates the usability of A-DETA.

Angle-Based Analysis Approach for Distributed Constrained Optimization

In this article, a distributed constrained optimization problem is studied with nonconvex input constraints, nonuniform convex state constraints, and nonuniform step sizes for single-integrator multiagent systems. Due to the existence of nonconvex input constraints, the edge weights between agents are equivalently multiplied with different time-varying scaling factors, and thus, the real interaction relationship cannot be kept balanced, even if the original communication graphs are kept balanced. Due to the existence of nonuniform convex state constraints and nonuniform step sizes, the system contains strong nonlinearities, which are coupled with the unbalance of the real interaction relationship, making existing analysis approaches hard to apply in this article. The main idea of the analysis approach is to fully explore the angles between the vectors from the agent states to their own projections on the intersection set of the convex state constraint sets so as to show that the distances from the agents to the intersection set diminish to zero as time evolves. By combining the analysis approaches in this article and our previous works, all agents are proved to converge to a common point and simultaneously solve the given optimization problem as long as the union of the communication graphs is strongly connected and balanced among each time interval of certain length. Numerical examples are given to show the obtained theoretical results.

Achieving Acceleration for Distributed Economic Dispatch in Smart Grids Over Directed Networks

In this paper, the economic dispatch problem (EDP) in smart grids is investigated over a directed network, which concentrates on allocating the generation power among the generators to satisfy the load demands with minimal total generation cost while complying with all constraints of local generation capacity. Each generator possesses its own local generation cost, and the total generation cost is the sum of all local generation costs. To deal with EDP, most of the existing methods, such as strategy based on push-sum, surmount the unbalancedness induced by the directed network via employing column-stochastic weights, which might be infeasible in distributed implementation. In contrast, in order to be suitable for the directed network with row-stochastic weights, we develop a novel directed distributed Lagrangian momentum algorithm, named as D-DLM, which integrates distributed gradient tracking method with two types of momentum terms and utilizes non-uniform step-sizes with respect to the updates of Lagrangian multipliers. If the largest step-size and the maximum momentum coefficient are positive and sufficiently small, D-DLM can linearly allocate the optimal dispatch on condition that the generation costs are smooth and strongly convex. Finally, a variety of studies on EDP in smart grids are simulated.

Computational Study to Predict the Free-Surface Flow over Non-uniform Stepped Spillway Using ANSYS-CFX

Using stepped chutes as a structure for controlling flood discharges is applicable for long time. Measuring the depth of flow over that structure is essential for designing of the side walls. The aim of this paper is to determine the free-surface that flows on spillway equipped with non-uniform step sizes. For that purpose, the two-dimensional software package code of ANSYS-CFX has been utilized to run eight configurations of two moderate slopes (1V:2H and 1V:2.5H) and for four different discharges 1≤dc/hs≤2.2 to determine the effect of flow discharges, chute slopes, and step heights on the position of free surface along the structure over non-uniform stepped cascade. The hexahedral grid size of 0.015 m is selected with inflation technique close to the walls. In addition, the renormalized group of k-ε (RNG) turbulence model is implemented and the numerical volume of fluid software is employed. The results show smoother stream for higher discharges, and the free-surface drops when the slope of chutes increases. Moreover, it is found that the step size has insignificant effect on the depth of water. The results of this study are important because they provide new insight in improving the design of stepped spillways. It is recommended to perform more investigations to evaluate their effectiveness in other flow parameters including pressure distribution and energy dissipation rates.

Event-Triggering Interaction Scheme for Discrete-Time Decentralized Optimization With Nonuniform Step Sizes.

In this article, we study the discrete-time decentralized optimization problems of multiagent systems by an event-triggering interaction scheme, in which each agent privately knows its local convex cost function, and collectively minimizes the total cost functions. The underlying interaction and the corresponding weight matrix are required to be undirected connected and doubly stochastic, respectively. To resolve this optimization problem collaboratively, we propose a decentralized event-triggering algorithm (DETA) that is based on the consensus theory and inexact gradient tracking technique. DETA involves each agent interacting with its neighboring agents only at some independent event-triggering sampling time instants. Under the assumptions that the global convex cost function is coercive and has Lipschitz continuous gradient, we prove that DETA steers all agents' states to an optimal solution even with nonuniform constant step sizes. Moreover, our analysis also shows that DETA converges at a rate of O(1/√t) if the step sizes are uniform and do not exceed some upper bounds. We illustrate the effectiveness of DETA on a canonical simple decentralized parameter estimation problem.

Computational Analysis of the Stability of 2D Heat Equation on Elliptical Domain Using Finite Difference Method

Aims: The aim and objective of the study to derive and analyze the stability of the finite difference schemes in relation to the irregularity of domain.
 Study Design: First of all, an elliptical domain has been constructed with the governing two dimensional (2D) heat equation that is discretized using the Finite Difference Method (FDM). Then the stability condition has been defined and the numerical solution by writing MATLAB codes has been obtained with the stable values of time domain.
 Place and Duration of Study: The work has been jointly conducted at the MUET, Jamshoro and QUEST, Nawabshah Pakistan from January 2019 to December 2019. 
 Methodology: The stability condition over an elliptical domain with the non-uniform step size depending upon the boundary tracing function is derived by using Von Neumann method.
 Results: From the results it was revealed that stability region for the small number of mesh points remains larger and gets smaller as the number of mesh nodes is increased. Moreover, the ranges for the time steps are defined for varied spatial step sizes that help to find the stable solution.
 Conclusion: The corresponding stability range for number of nodes N=10, 20, 30, 40, 50, and 60 was found respectively. Within this range the solution remains smooth as time increases. The results of this study attempt to provide the stable solution of partial differential equations on irregular domains.

A Nesterov-Like Gradient Tracking Algorithm for Distributed Optimization Over Directed Networks

In this article, we concentrate on dealing with the distributed optimization problem over a directed network, where each unit possesses its own convex cost function and the principal target is to minimize a global cost function (formulated by the average of all local cost functions) while obeying the network connectivity structure. Most of the existing methods, such as push-sum strategy, have eliminated the unbalancedness induced by the directed network via utilizing column-stochastic weights, which may be infeasible if the distributed implementation requires each unit to gain access to (at least) its out-degree information. In contrast, to be suitable for the directed networks with row-stochastic weights, we propose a new directed distributed Nesterov-like gradient tracking algorithm, named as D-DNGT, that incorporates the gradient tracking into the distributed Nesterov method with momentum terms and employs nonuniform step-sizes. D-DNGT extends a number of outstanding consensus algorithms over strongly connected directed networks. The implementation of D-DNGT is straightforward if each unit locally chooses a suitable step-size and privately regulates the weights on information that acquires from in-neighbors. If the largest step-size and the maximum momentum coefficient are positive and small sufficiently, we can prove that D-DNGT converges linearly to the optimal solution provided that the cost functions are smooth and strongly convex. We provide numerical experiments to confirm the findings in this article and contrast D-DNGT with recently proposed distributed optimization approaches.

Distributed Continuous-Time and Discrete-Time Optimization With Nonuniform Unbounded Convex Constraint Sets and Nonuniform Stepsizes

This paper is devoted to distributed continuous-time and discrete-time optimization problems with nonuniform convex constraint sets and nonuniform stepsizes for general differentiable convex objective functions. The communication graphs are not required to be strongly connected at any time, the gradients of the local objective functions are not required to be bounded when their independent variables tend to infinity, and the constraint sets are not required to be bounded. For continuous-time multi-agent systems, a distributed continuous algorithm is first introduced where the stepsizes and the convex constraint sets are both nonuniform. It is shown that all agents reach a consensus while minimizing the team objective function even when the constraint sets are unbounded. After that, the obtained results are extended to discrete-time multi-agent systems and then the case where each agent remains in a corresponding convex constraint set is studied. To ensure all agents to remain in a bounded region, a switching mechanism is introduced in the algorithms. It is shown that the distributed optimization problems can be solved, even though the discretization of the algorithms might deviate the convergence of the agents from the minimum of the objective functions. Finally, numerical examples are included to show the obtained theoretical results.

Validation of the Computational ANSYS -CFX code for Free Surface Flow: Skimming Flow over Non-Uniform Step Size Stepped Spillways

Validation of the Computational ANSYS -CFX code for Free Surface Flow: Skimming Flow over Non-Uniform Step Size Stepped Spillways

Distributed Optimization With Nonconvex Velocity Constraints, Nonuniform Position Constraints, and Nonuniform Stepsizes

This note is devoted to the distributed optimization problem of multi-agent systems with nonconvex velocity constraints, nonuniform position constraints and nonuniform stepsizes. Two distributed constrained algorithms with nonconvex velocity constraints and nonuniform stepsizes are proposed in the absence and the presence of nonuniform position constraints by introducing a switching mechanism to guarantee all agents' position states to remain in a bounded region. The algorithm gains need not to be predesigned and can be selected by each agent using its own and neighbours' information. By a model transformation, the original nonlinear time-varying system is converted into a linear time-varying one with a nonlinear error term. Based on the properties of stochastic matrices, it is shown that the optimization problem can be solved as long as the communication topologies are jointly strongly connected and balanced. Numerical examples are given to show the obtained theoretical results.

Development of blind algorithm with automatic gain control

In this paper, the concept of blind algorithm with automatic gain control (AGC) is introduced in adaptive antenna system for signal optimization with an aim to estimate the desired response in adaptive fashion. Blind algorithm with AGC is a hybrid two-stage adaptive filtering algorithm; sequentially combining constant modulus algorithm (CMA) and Bessel least mean square (BLMS) algorithm. Blind Bessel beamformer with AGC does not require external reference signal to update its weight vectors and step size for convergence but updates itself from own reference signal obtained from the output of CMA. Similarly, step size is obtained from the correlation matrix which is the product of the signals induced in array elements of antenna. BLMS is the modified version of LMS algorithm; based on the non-uniform step size exploiting the asymptotic decay property of Bessel function of the first kind. The output of CMA provides input and reference signals for BLMS that makes it blind. The contributions of this paper include the development of novel blind theory concept and presentation of an AGC method in order to make the Bessel beamformer blind which can update itself electronically through the correlation matrix depending on the signal array vector with the aim to make the signal power constant.

Time of flight PET reconstruction using nonuniform update for regional recovery uniformity.

Time of flight (TOF) PET reconstruction is well known to statistically improve the image quality compared to non-TOF PET. Although TOF PET can improve the overall signal to noise ratio (SNR) of the image compared to non-TOF PET, the SNR disparity between separate regions in the reconstructed image using TOF data becomes higher than that using non-TOF data. Using the conventional ordered subset expectation maximization (OS-EM) method, the SNR in the low activity regions becomes significantly lower than in the high activity regions due to the different photon statistics of TOF bins. A uniform recovery across different SNR regions is preferred if it can yield an overall good image quality within small number of iterations in practice. To allow more uniform recovery of regions, a spatially variant update is necessary for different SNR regions. This paper focuses on designing a spatially variant step size and proposes a TOF-PET reconstruction method that uses a nonuniform separable quadratic surrogates (NUSQS) algorithm, providing a straightforward control of spatially variant step size. To control the noise, a spatially invariant quadratic regularization is incorporated, which by itself does not theoretically affect the recovery uniformity. The Nesterov's momentum method with ordered subsets (OS) is also used to accelerate the reconstruction speed. To evaluate the proposed method, an XCAT simulation phantom and clinical data from a pancreas cancer patient with full (ground truth) and 6× downsampled counts were used, where a Poisson thinning process was employed for downsampling. We selected tumor and cold regions of interest (ROIs) and compared the proposed method with the TOF-based conventional OS-EM and OS-SQS algorithms with an early stopping criterion. In computer simulation, without regularization, hot regions of OS-EM and OS-NUSQS converged similarly, but cold region of OS-EM was noisier than OS-NUSQS after 24 iterations. With regularization, although the overall speeds of OS-EM and OS-NUSQS were similar, recovery ratios of hot and cold regions reconstructed by the OS-NUSQS were more uniform compared to those of the conventional OS-SQS and OS-EM. The OS-NUSQS with Nesterov's momentum converged faster than others while preserving the uniform recovery. In the clinical example, we demonstrated that the OS-NUSQS with Nesterov's momentum provides more uniform recovery ratios of hot and cold ROIs compared to the OS-SQS and OS-EM. Although the cost function of all methods is equivalent, the proposed method has higher structural similarity (SSIM) values of hot and cold regions compared to other methods after 24 iterations. Furthermore, our computing time using graphics processing unit was 80× shorter than the time using quad-core CPUs. This paper proposes a TOF PET reconstruction method using the OS-NUSQS with Nesterov's momentum for uniform recovery of different SNR regions. In particular, the spatially nonuniform step size in the proposed method provides uniform recovery ratios of different SNR regions, and the Nesterov's momentum further accelerates overall convergence while preserving uniform recovery. The computer simulation and clinical example demonstrate that the proposed method converges uniformly across ROIs. In addition, tumor contrast and SSIM of the proposed method were higher than those of the conventional OS-EM and OS-SQS in early iterations.

Distributed Optimization Algorithm for Discrete-Time Heterogeneous Multi-Agent Systems With Nonuniform Stepsizes

This paper is devoted to the distributed optimization problem of heterogeneous multi-agent systems, where the communication topology is jointly strongly connected and the dynamics of each agent is the first-order or second-order integrator. A new distributed algorithm is first designed for each agent based on the local objective function and the local neighbors’ information that each agent can access. By a model transformation, the original closed-loop system is converted into a time-varying system and the system matrix of which is a stochastic matrix at any time. Then, by the properties of the stochastic matrix, it is proven that all agents’ position states can converge to the optimal solution of a team objective function provided the union communication topology is strongly connected. Finally, the simulation results are provided to verify the effectiveness of the distributed algorithm proposed in this paper.

SHAFA: sparse hybrid adaptive filtering algorithm to estimate channels in various SNR environments

The l p -norm penalised (LP) normalised least mean square algorithm converges faster than the LP normalised least mean fourth algorithm does, but the latter can achieve better steady-state performance, particularly in regions with low signal-to-noise ratios (SNRs). To simultaneously take advantage of both merits, a sparse hybrid adaptive filtering algorithm is proposed in various SNR environments. Specifically, the authors construct a cost function that uses the statistical error term and sparse penalty term. The first term is designed by a hybrid error function of the second- and fourth-order statistical errors, respectively, and the second term is obtained using a sparse constraint function. The hybrid error term can be easily balanced by a proportional parameter α ∈ 0 , 1 . Moreover, they devise a non-uniform step size in the proposed algorithm to further balance the convergence speed and estimation error. Simulation results are provided to validate the proposed algorithm in various SNR environments.

Hybrid Monte Carlo with non-uniform step size.

The Hybrid Monte Carlo method offers a rigorous and potentially efficient approach to the simulation of dense systems, by combining numerical integration of Newton's equations of motion with a Metropolis accept-or-reject step. The Metropolis step corrects for sampling errors caused by the discretization of the equations of motion. The integration is usually performed using a uniform step size. Here, we present simulations of the Lennard-Jones system showing that the use of smaller time steps in the tails of each integration trajectory can reduce errors in energy. The acceptance rate is 10-15 percentage points higher in these runs, compared to simulations with the same trajectory length and the same number of integration steps but a uniform step size. We observe similar effects for the harmonic oscillator and a coarse-grained peptide model, indicating generality of the approach.

Preservation of the growth rates of delay differential equations by Euler schemes with non-uniform step sizes

In this paper, we consider explicit Euler methods which recover the rate of growth to infinity of a highly nonlinear autonomous delay differential equation. The success of the methods rely on the step size changing in response to the state, as it has been shown that Euler methods with constant step size will systematically underestimate the growth rate. It is also shown that the computed solution converges to the true solution on any compact time interval, when a parameter which controls the step size is sent to zero. A second method, which applies to a related ordinary differential equation, and which requires less computational effort, is also presented.

Fast Beamforming for Mobile Satellite Receiver Phased Arrays: Theory and Experiment

The purpose of this paper is to present a robust and fast beamforming algorithm for the low-cost mobile phased array antennas. The proposed beamforming algorithm uses a sequentially perturbation gradient estimation method to update the control voltages of the phase shifters, with the objective of maximizing the received power by the array. This algorithm does not require either the knowledge of phase shifter characteristics or signal direction-of-arrival. Moreover, in this paper, the algorithm parameters are derived for the stationary and mobile platform configurations. For the stationary array, it is shown how the proper selection of the beamforming parameters limits the noise effects and increases the array output power. For the mobile array, a condition for the fast convergence is derived and the advantage of using nonuniform step size to update the control voltages is illustrated. When phase shifters suffer from the imbalanced insertion loss the proposed beamforming technique perturbs the phase-conjugate condition to increase the total received power. This algorithm has been implemented with the low-cost microwave components and applied to a Ku-band phased array antenna with 34 sub-arrays. The experimental results verify the broadband performance, and the fast convergence of the algorithm for different platform maneuvers.

Remarks on the rational approximation of holomorphic semigroups with nonuniform partitions

Some convergence and stability results for rational approximations of holomorphic semigroups are presented. The approximations are related to time discrete schemes with nonuniform stepsize to a linear evolution equation of parabolic type in a Banach space. Our theorems cover not only the backward Euler but also the Crank-Nicolson and forward Euler schemes.