Iterative Gradient Descent Research Articles

Iterative gradient denoising algorithm for signal recovery using analysis-based implicit prior

To recover a signal from its linear measurements, the present paper considers the relaxed analysis LASSO (RALASSO) optimization model. An algorithm involving a gradient descent operator and a soft shrinkage operator in each iteration is proposed to solve RALASSO, termed iterative gradient descent shrinkage (IGDS). Further, being not restricted to the sparsity regime, we replace the ℓ1-norm regularizer in RALASSO by a general regularization function, and derive an algorithm framework for solving this more general problem. The framework employs a denoiser as a black-box to replace the soft shrinkage operator, thus we call it iterative gradient descent denoising (IGDD). IGDS can be viewed as a special instance of IGDD. The recovery performance of IGDD armed with a good denoiser will be better than that of IGDS. Extensive experimental results based on several numerical examples and image restoration tasks show that our algorithms have advantages over some analysis-model-based algorithms and denoiser-based ones, in terms of recovery capability or run-time.

Sparse recovery under nonnegativity and sum-to-one constraints

Sparse recovery under nonnegativity and sum-to-one constraints is a special form of the linear regression problem, where the solution is required to simultaneously satisfy sparsity, nonnegativity, and sum-to-one restraints. Existing algorithms for this task mainly utilize the penalty technique to convert the sparsity constraint into a regularization term. Therefore, the sparsity is determined via tuning the associated penalty parameter, which is time-consuming in practice. This paper exploits projected gradient descent to directly tackle the constrained problem without involving the penalty parameter and ℓ0-norm approximation. The addition of the ℓ0-norm constraint with a specific upper bound enables the proposed algorithm to explicitly control sparsity. The developed method is termed as modified iterative hard thresholding (MIHT), comprised of two iterative steps, namely, gradient descent and nonconvex projection. For the latter, the constraint set consists of the ℓ0-norm, nonnegativity, and sum-to-one constraints. We devise an efficient algorithm to address the nonconvex projection and then prove that this method produces an optimal solution. Furthermore, we establish the convergence of the MIHT, including objective value and variable sequence. Numerical experiments using financial and hyperspectral data demonstrate that the MIHT is superior to state-of-the-art methods in terms of prediction error and recovery accuracy.

Research and design of image style transfer technology based on multi‐scale convolutional neural network feature fusion

AbstractTo reduce the occurrence of information loss and distortion in image style transfer, a method is proposed for researching and designing image style transfer technology based on multi‐scale convolutional neural networks (CNNs) feature fusion. Initially, the VGG19 model is designed for coarse and fine‐scale networks to achieve multi‐scale CNN feature extraction of target image information. Subsequently, while setting the corresponding feature loss function, an additional least‐squares penalty parameter is introduced to balance the optimal total loss function. Finally, leveraging the characteristics of stochastic gradient descent iteration, image features are fused and reconstructed to obtain better style transfer images. Experimental evaluations utilize peak signal‐to‐noise ratio (PSNR), structural similarity index (SSIM), information entropy (IE), and mean squared error (MSE) as metrics for assessing the transferred images, comparing them with three typical image style transfer methods. Results demonstrate that the proposed method achieves optimal performance across all metrics, realizing superior image style transfer effects.

On the Unstable Convergence Regime of Gradient Descent

Traditional gradient descent (GD) has been fully investigated for convex or L-smoothness functions, and it is widely utilized in current neural network optimization. The classical descent lemma ensures that for a function with L-smoothness, the GD trajectory converges stably towards the minimum when the learning rate is below 2 / L. This convergence is marked by a consistent reduction in the loss function throughout the iterations. However, recent experimental studies have demonstrated that even when the L-smoothness condition is not met, or if the learning rate is increased leading to oscillations in the loss function during iterations, the GD trajectory still exhibits convergence over the long run. This phenomenon is referred to as the unstable convergence regime of GD. In this paper, we present a theoretical perspective to offer a qualitative analysis of this phenomenon. The unstable convergence is in fact an inherent property of GD for general twice differentiable functions. Specifically, the forwardinvariance of GD is established, i.e., it ensures that any point within a local region will always remain within this region under GD iteration. Then, based on the forward-invariance, for the initialization outside an open set containing the local minimum, the loss function will oscillate at the first several iterations and then become monotonely decreasing after the GD trajectory jumped into the open set. This work theoretically clarifies the unstable convergence phenomenon of GD discussed in previous experimental works. The unstable convergence of GD mainly depends on the selection of the initialization, and it is actually inevitable due to the complex nature of loss function.

Mode-resolved phonon transmittance using lattice dynamics: Robust algorithm and statistical characteristics

Lattice dynamics (LD) enables the calculation of mode-resolved transmittance of phonons passing through an interface, which is essential for understanding and controlling the thermal boundary conductance (TBC). However, the original LD method may yield unphysical transmittance over 100% due to the absence of the constraint of energy conservation. Here, we present a robust LD algorithm that utilizes linear algebra transformations and projection gradient descent iterations to ensure energy conservation. Our approach demonstrates consistency with the original LD method on the atomically smooth Si/Ge interface and exhibits robustness on rough Si/Ge interfaces. The evanescent modes and localized effects at the interface are revealed. In addition, bottom-up analysis of the phonon transmittance shows that the anisotropy in the azimuth angle can be ignored, while the dependency on the frequency and polar angle can be decoupled. The decoupled expression reproduces the TBC precisely. This work provides comprehensive insights into the mode-resolved phonon transmittance across interfaces and paves the way for further research into the mechanism of TBC and its relation to atomic structures.

Penalized wavelet nonparametric univariate logistic regression for irregular spaced data

This paper concerns the study of a non-smooth logistic regression function. The focus is on a high-dimensional binary response case by penalizing the decomposition of the unknown logit regression function on a wavelet basis of functions evaluated on the sampling design. Sample sizes are arbitrary (not necessarily dyadic) and we consider general designs. We study separable wavelet estimators, exploiting sparsity of wavelet decompositions for signals belonging to homogeneous Besov spaces, and using efficient iterative proximal gradient descent algorithms. We also discuss a level by level block wavelet penalization technique, leading to a type of regularization in multiple logistic regression with grouped predictors. Theoretical and numerical properties of the proposed estimators are investigated. A simulation study examines the empirical performance of the proposed procedures, and real data applications demonstrate their effectiveness.

On Kahan's automatic step-size control and an anadromic gradient descent iteration

Accelerating the basic gradient iteration is a critical issue in numerical optimization and deep-learning. Recently, William Kahan [Kahan, Automatic Step-Size Control for Minimization Iterations, Technical report, University of California, Berkeley CA, USA, 2019] proposed automatic step-size strategies for the gradient descent iteration without explicitly using any prior information of the Hessian; moreover, a new momentum-based gradient acceleration method, namely, an Anadromic Gradient Descent (AGD) iteration, was proposed. Besides the capability of accelerating the basic gradient descent iteration, AGD enables the iteration to return to the past iterates by just reversing steps in sign and order in the same updating formulation. Numerical performances of the automatic step-size strategies and AGD are claimed to be favourable. In this paper, for the quadratic model, through a revisit of some classical momentum-based gradient methods, we perform new analysis on the convergence and their optimal hyper-parameters. We also investigate the convergence behaviour of AGD with the optimal hyper-parameters and connect one Kahan's automatic step-size scheme with the long Barzilai–Borwein step-size. Numerical results are presented to reflect the theoretical convergence behaviours and demonstrate the practical performances of various momentum-based gradient methods.

Multi‐input fully CNN for joint pilot decontamination and symbol detection in 5G massive MIMO

AbstractThis paper presents a multi‐input deep learning‐based joint pilot decontamination and symbol detection (SD) technique for 5G massive multiple‐input multiple‐output (MAMIMO) systems. It consists of a fully convolutional neural network (FCNN) that finds the 5G channel coefficients using pre‐known sounding reference signals (SRSs) under pilot contamination and a symbol detector derived from projected gradient descent iterations. The study considers pilot contamination caused by inter‐cell interferences between SRSs during channel estimation (CE). The proposed scheme accepts entire 5G orthogonal frequency modulated (OFDM) data and least square estimates and produces the transmitted OFDM signal. Simulation experiments demonstrated that the proposed technique has better CE and SD performance with reduced trainable parameters. Moreover, it is faster due to the lowest elapsed time during end‐to‐end OFDM symbol detection. This paper proposes a joint pilot decontamination and signal detection for 5G MAMIMO systems. It achieves better detection performance with the lowest number of trainable parameters and memory requirements. It is applicable in 5G OFDM symbol detection and channel estimation under pilot contamination.

Adaptive segmentation-based evaluation of material properties of dielectric sheets using microwave NDE

ABSTRACT Composite materials/structures are rapidly advancing, offering improved product efficiency, cost-effectiveness, and superior specific properties. These materials find increasing applications in corrosion resistance and load-carrying structures across industries such as aerospace, wind turbines, transportation, and medical equipment. Therefore, ensuring reliable non-destructive evaluation of composites is crucial to address safety concerns and minimise maintenance expenses. The microwave non-destructive evaluation (NDE) method is particularly well-suited for characterising the material properties of composites. In our study, we utilise a large flange open-ended rectangular waveguide (OERW) probe, measuring 152 mm x 152 mm, to assess the complex reflection coefficient of dielectric materials. This measurement enables the extraction of essential material parameters using a forward electromagnetic model combined with an optimised method. To enhance accuracy, we introduce an iterative conjugate gradient descent (CGD) optimiser with adaptive segmentation. This approach effectively avoids local minima and reduces disparities in integrand values, resulting in precise measurements of material properties. Our proposed method provides estimations of the dielectric constant within a range of 1 to 10 and accurately determines sample thickness with a controlled limit of 3% error across all samples. Furthermore, we observe that as the loss-factor value decreases, the error in calculating the loss-factor of samples increases, which is supported by numerical analysis.

Gradient Coding With Dynamic Clustering for Straggler-Tolerant Distributed Learning

Distributed implementations are crucial in speeding up large scale machine learning applications. Distributed gradient descent (GD) is widely employed to parallelize the learning task by distributing the dataset across multiple workers. A significant performance bottleneck for the per-iteration completion time in distributed synchronous GD is straggling workers. Coded distributed computation techniques have been introduced recently to mitigate stragglers and to speed up GD iterations by assigning redundant computations to workers. In this paper, we introduce a novel paradigm of dynamic coded computation, which assigns redundant data to workers to acquire the flexibility to dynamically choose from among a set of possible codes depending on the past straggling behavior. In particular, we propose gradient coding (GC) with dynamic clustering, called GC-DC, and regulate the number of stragglers in each cluster by dynamically forming the clusters at each iteration. With time-correlated straggling behavior, GC-DC adapts to the straggling behavior over time; in particular, at each iteration, GC-DC aims at distributing the stragglers across clusters as uniformly as possible based on the past straggler behavior. For both homogeneous and heterogeneous worker models, we numerically show that GC-DC provides significant improvements in the average per-iteration completion time without an increase in the communication load compared to the original GC scheme.

A hybrid method for density power divergence minimization with application to robust univariate location and scale estimation

We develop a new globally convergent optimization method for solving a constrained minimization problem underlying the minimum density power divergence estimator for univariate Gaussian data in the presence of outliers. Our hybrid procedure combines classical Newton’s method with a gradient descent iteration equipped with a step control mechanism based on Armijo’s rule to ensure global convergence. Extensive simulations comparing the resulting estimation procedure with the more prominent robust competitor, Minimum Covariance Determinant (MCD) estimator, across a wide range of breakdown point values suggest improved efficiency of our method. Application to estimation and inference for a real-world dataset is also given.

Seismic acquisition design based on full-wavefield migration

The ultimate goal in survey design is to obtain the acquisition parameters that enable acquiring the most affordable data that fulfill certain image quality requirements. A method that allows optimization of the receiver geometry for a fixed source distribution is proposed. The former is parameterized with a receiver density function that determines the number of receivers per unit area. We optimize this receiver density function through an iterative gradient descent scheme that minimizes the difference between the image obtained with the current acquisition geometry and a reference image. The reference image is obtained from prior subsurface information that is assumed to be available. We tested the method with different subsurface models. The results indicate that the acquisition geometry is optimized according to the complexity of each subsurface model. The receivers are moved toward the areas where more data are needed for obtaining better imaging.

GANSim-surrogate: An integrated framework for stochastic conditional geomodelling

Stochastic conditional geomodelling requires effective integration of geological patterns and various types of data, which is crucial but challenging. To address this, we propose a deep-learning framework (GANSim-surrogate) for conditioning geomodels to static well facies data, facies probability maps, and non-spatial global features, as well as dynamic time-dependent pressure or flow rate data observed at wells. The framework consists of a Convolutional Neural Network (CNN) generator trained from GANSim (a Generative Adversarial Network-based geomodelling simulation approach), a CNN-based surrogate, and options for searching appropriate input latent vectors for the generator. The four search methods investigated are Markov Chain Monte Carlo, Iterative Ensemble Smoother, gradient descent, and gradual deformation. The framework is validated with channelized reservoirs. First, a generator is trained using GANSim to generate geological facies models; in addition, a flow simulation surrogate is trained using a physics-informed approach. Then, given well facies data, facies probability maps, global facies proportions, and dynamic bottomhole pressure data (BHP), the trained generator takes the first three static conditioning data and a latent vector as inputs and produces a random realistic facies model conditioned to the three static data. To condition to the dynamic data, the produced facies model is converted to permeability property and mapped to BHP data by the trained surrogate. Finally, the mismatch between the surrogate-produced and the observed BHP data is minimized to obtain appropriate input latent vectors which are further mapped into appropriate facies models through the generator. These facies models prove to be realistic and consistent with all of the conditioning data, and the framework is computationally efficient.

Hierarchical Bandwidth Allocation for Social Community-Oriented Multicast in Space-Air-Ground Integrated Networks

With the rapid advance of wireless communication technologies, the promising space-air-ground integrated networks (SAGINs) are advocated to provide ubiquitous multicast transmission services for the social community constituted by a group of mobile users that have strong social ties and similar content interests. However, due to the limited yet valuable spectrum resources, the network heterogeneity, and diverse service demands of mobile users, it is challenging to efficiently allocate bandwidth for social communities with the objective of achieving satisfactory quality of experience (QoE) in SAGINs. To address this problem, in this paper, we propose a hierarchical bandwidth allocation scheme to enable high-quality multicast services for social communities in SAGINs. Specifically, we first develop a hierarchical bandwidth allocation framework. Wherein, the low earth orbit (LEO) satellite is utilized to provide space-to-air (S2A) unicast bandwidth for unmanned aerial vehicles (UAVs) at a certain price. Each UAV is employed to provide air-to-ground (A2G) multicast bandwidth for ground social communities with a certain A2G multicast bandwidth charge. We then formulate the hierarchical bandwidth allocation problem as a four-stage Stackelberg game, where the target of each participant is to maximize its own utility. Afterward, through the game analysis by the backward induction method, the existence of the Stackelberg equilibrium is proved, where the closed-form solutions on the optimal policies of both the social communities and UAVs are derived by the convex optimization method, and the optimal pricing policies of the LEO satellite is achieved by a proposed gradient descent iteration algorithm. Finally, extensive experiments are conducted to demonstrate that the proposed scheme can greatly increase the utilities of social communities while consuming a less bandwidth compared to conventional schemes.

Survey Descent: A Multipoint Generalization of Gradient Descent for Nonsmooth Optimization

For strongly convex objectives that are smooth, the classical theory of gradient descent ensures linear convergence relative to the number of gradient evaluations. An analogous nonsmooth theory is challenging. Even when the objective is smooth at every iterate, the corresponding local models are unstable, and the number of cutting planes invoked by traditional remedies is difficult to bound, leading to convergence guarantees that are sublinear relative to the cumulative number of gradient evaluations. We instead propose a multipoint generalization of the gradient descent iteration for local optimization. While our iteration was designed with general objectives in mind, we are motivated by a “max-of-smooth” model that captures the subdifferential dimension at optimality. We prove linear convergence when the objective is itself max-of-smooth, and experiments suggest a more general phenomenon.

Improved Gradient Descent Iterations for Solving Systems of Nonlinear Equations

This research proposes and investigates some improvements in gradient descent iterations that can be applied for solving system of nonlinear equations (SNE). In the available literature, such methods are termed improved gradient descent methods. We use verified advantages of various accelerated double direction and double step size gradient methods in solving single scalar equations. Our strategy is to control the speed of the convergence of gradient methods through the step size value defined using more parameters. As a result, efficient minimization schemes for solving SNE are introduced. Linear global convergence of the proposed iterative method is confirmed by theoretical analysis under standard assumptions. Numerical experiments confirm the significant computational efficiency of proposed methods compared to traditional gradient descent methods for solving SNE.

Iterative Robust Ellipsoid Fitting Based on M-Estimator With Geometry Radius Constraint

The ellipsoid fitting technique has been popularly used in many application scenarios, especially in triaxial sensor calibration in recent years. The conventional method to achieve ellipsoid fitting is the pseudo-inverse technique with algebraic constraints to obtain proper fitting results. This technique ensures the least mean square (LMS) error relative to all measured data and prevents trivial solutions. However, this approach lacks robustness against outliers resulting in nonideal fitting performance in real calibration situations. The M-estimator technique with several robust cost functions is adopted and applied in this research to attenuate the outlier’s effect to offer a more accurate fitting performance. In contrast, the constraint with geometric meaning is also applied and examined to prevent trivial solutions. These two methods are achieved by using the iterative constrained gradient descent (G.D.) method to approach the proper ellipsoid fitting. The performances are validated through synthesis and real measured data. Through calibrated experiments, the proposed methods are found to be able to increase robustness against outliers and prevent parameters of ellipsoid degeneration toward trivial solutions.

Deep Learning for Scalable Optimal Design of Incremental Volt/VAR Control Rules

Volt/VAR control rules enable distributed energy resources (DER) to autonomously regulate voltage in distribution grids. The Volt/VAR rules provisioned by the IEEE Standard 1547 take on a piecewise-linear shape. However, its maximum slope is upper bounded to ensure stability, and that may hamper its voltage regulation performance. This limitation can be surpassed by adding a memory term to the control rule, and thus, obtaining a so-termed incremental control rule. This letter aims to optimally customize the shape of incremental rules across buses to attain desirable voltage profiles. Albeit this task can be posed as a bilevel program, we pursue a more scalable approach by reformulating it as a deep learning task. The idea is that Volt/VAR dynamics can be captured by a recursive neural network (RNN). Interestingly, the RNN weights correspond to the parameters of the control rule; the RNN input to the grid loading conditions; and the RNN output to the equilibrium voltages. Therefore, the optimal rule parameters can be found upon training the RNN so its output (equilibrium voltages) approach unity. Training is performed by feeding the RNN with representative scenarios of the anticipated grid loading conditions. The RNN depth depends on the settling time of Volt/VAR dynamics. Because the discrete-time Volt/VAR dynamics can be viewed as iterations of a proximal gradient descent (PGD) algorithm, we also leverage Nesterov’s accelerated PGD iterations to reduce the RNN depth. The RNN is never implemented in the field. Training this RNN is equivalent to solving the optimal rule design in a more computationally efficient manner. Analytical findings and numerical tests corroborate that the proposed solution can be neatly adapted to single-and multi-phase feeders. The proposed approach could be of general interest in designing piecewise-linear controllers acting on linear plants.

Deep-Unfolding Neural-Network Aided Hybrid Beamforming Based on Symbol-Error Probability Minimization

In massive multiple-input multiple-output (MIMO) systems, hybrid analog-digital (AD) beamforming can be used to attain a high directional gain without requiring a dedicated radio frequency (RF) chain for each antenna element, which substantially reduces both the hardware costs and power consumption. While massive MIMO transceiver design typically relies on the conventional mean-square error (MSE) criterion, directly minimizing the symbol error rate (SER) can lead to a superior performance. In this article, we first mathematically formulate the problem of hybrid transceiver design under the minimum SER (MSER) optimization criterion and then develop an MSER-based iterative gradient descent (GD) algorithm to find the related stationary points. We then propose a deep-unfolding neural network (NN). The iterative GD algorithm is unfolded into a multi-layer structure wherein trainable parameters are introduced to accelerate the convergence and enhance the overall system performance. To implement the training stage, we derive the relationship between adjacent layers' gradients based on the generalized chain rule (GCR). The deep-unfolding NN is developed for both quadrature phase shift keying (QPSK) and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$M$</tex-math></inline-formula> -ary quadrature amplitude modulated (QAM) signals, and its convergence is investigated theoretically. Furthermore, we analyze the transfer capability, computational complexity, and generalization capability of the proposed deep-unfolding NN. Our simulation results show that the latter significantly outperforms its conventional counterpart at a reduced complexity.

Variational Monte Carlo approach to partial differential equations with neural networks

The accurate numerical solution of partial differential equations (PDEs) is a central task in numerical analysis allowing to model a wide range of natural phenomena by employing specialized solvers depending on the scenario of application. Here, we develop a variational approach for solving PDEs governing the evolution of high dimensional probability distributions. Our approach naturally works on the unbounded continuous domain and encodes the full probability density function through its variational parameters, which are adapted dynamically during the evolution to optimally reflect the dynamics of the density. In contrast to previous works, this dynamical adaptation of the parameters is carried out using an explicit prescription avoiding iterative gradient descent. For the considered benchmark cases we observe excellent agreement with numerical solutions as well as analytical solutions for tasks that are challenging for traditional computational approaches.