Fast Initial Convergence Research Articles

A Comprehensive Analysis of Optimization Algorithms for Large-Scale Machine Learning

Abstract—Optimization algorithms determine the best model parameters that tend to offer prediction accuracy in complex machine learning (ML) problems in big data. In fact, they are very crucial while handling large-scale data because old ways of handling datasets will be computationally very inefficient. This paper provides an in-depth comparison of the two most widely used optimization techniques in machine learning: SGD, and BGD. Our simulated results indicate that though the SGD technique indicates very fast initial convergence, its efficiency eventually tends to degrade with the increasing iteration of the algorithm. On the other hand, the BGD approach might take slow initiation but is relatively consistent in a long run. We then further probe how the variations in learning rate affect its performance in the case of both the methods. Our analysis shows that adaptive learning rates drastically accelerate convergence. Finally, we show that the computational efficiency of the SGD method makes it a better choice since gradients can be computed on a per sample basis, which makes the method better for scaling. Keywords—Optimization Algorithms, Stochastic gradient de- scent, Batch gradient descent, Machine learning, Large-scale data, and Adaptive learning rate.

A robust variable-λ least logarithmic hyperbolic cosine function adaptive filtering-based control algorithm for grid-connected solar PV system

This paper implements a robust variable-λ least logarithmic hyperbolic cosine function adaptive filtering (VLlncosh)-based control algorithm for a grid-connected solar photovoltaic system (GCPVS) which includes DSTATCOM features while integrating PV power to the grid at enhanced power quality (PQ). In this paper, a single-stage topology of GCPVS is considered, which can reduce the complexity, cost, and maintenance associated with DC-DC converter employed in two-stage topology. In GCPVS, the nonlinear loads, which deteriorate the quality of the power, are attached to the point of common coupling (PCC), and maximum extracted PV power is allocated between these loads and the grid. The power transfer at the grid takes place at unity power factor (UPF). The proposed VLlncosh algorithm estimates the fundamental weight components of the distorted load current for grid reference currents generation with a variable learning parameter to ensure fast initial convergence and more steady-state accuracy. The control algorithm operates GCPVS such that it provides compensation to the load current harmonics, unbalanced load current, and load reactive power while complying with IEEE-519–2022 standard. The VLlncosh-based control algorithm is implemented for GCPVS modeled in MATLAB/Simulink to evaluate its performance through simulation results. Further, the control algorithm is validated using the experimental prototype setup of GCPVS in the laboratory. The comparative assessment of the VLlncosh algorithm is provided to confirm its superior performance over LMS, sign-error LMS, and LMF algorithms.

A Novel Influence Function M-Estimator-Based for Active Noise Control

M-estimators are widely used in active noise control (ANC) systems in order to update the adaptive FIR filter taps. ANC systems reduce the noise level by generating anti-noise signals. Up to now, the evaluation of M-estimators capabilities has shown that there exists a need for further improvements in this area. In this paper, a new improved M-estimator is proposed. The sensitivity of the proposed algorithm to the variations of its constant parameter is checked in feedforward control. The effectiveness of the algorithm in both types is proved by comparing it with previous studies. Simulation results show the steady performance and fast initial convergence of the proposed algorithm.

Developing a New Filtered-X Recursive Least Squares Adaptive Algorithm Based on a Robust Objective Function for Impulsive Active Noise Control Systems

It is well-known that performance of the classical algorithms for active noise control (ANC) systems severely degrades when implemented for controlling the impulsive sources. The objective of this paper is to propose a new recursive least squares (RLS) algorithm (and its variant) for being implemented in the framework of ANC systems. The proposed RLS-based adaptive algorithm employs an objective function designed to achieve robustness against the impulse type sources. The derivation of the algorithm is quite straightforward; however, a few modifications have been incorporated to address the application at hand. In order to improve upon the numerical stability issue of RLS-based adaptation, it is suggested to employ smoothing while updating the inverse correlation matrix. Furthermore, it is proposed to introduce a step size in the update equation of the adaptive algorithm. This results in the fixed step-size modified filtered-x (MFx) robust RLS (FSS-MFxRRLS) algorithm. As expected, a fixed value step size results in a trade-off situation for convergence speed and steady-state misalignment. In order to address this issue of a trade-off situation, the idea of a convex combined step size (CCSS) is introduced into the adaptive procedure to develop the CCSS-MFxRRLS algorithm. When the ANC is started, the CCSS strategy (automatically) selects a large-valued step size to achieve a fast initial convergence. As the ANC system converges at the steady-state, the CCSS is automatically tuned to a small value which improves the steady-state performance of the proposed CCSS-MFxRRLS algorithm. Extensive simulations have been designed to mimic many scenarios for practical applications of ANC for impulsive sources. The simulation results demonstrate that the proposed CCSS-MFxRRLS algorithm is very effective in many practical scenarios involving ANC of impulsive sources.

SlimTrain---A Stochastic Approximation Method for Training Separable Deep Neural Networks

Deep neural networks (DNNs) have shown their success as high-dimensional function approximators in many applications; however, training DNNs can be challenging in general. DNN training is commonly phrased as a stochastic optimization problem whose challenges include nonconvexity, nonsmoothness, insufficient regularization, and complicated data distributions. Hence, the performance of DNNs on a given task depends crucially on tuning hyperparameters, especially learning rates and regularization parameters. In the absence of theoretical guidelines or prior experience on similar tasks, this requires solving a series of repeated training problems which can be time-consuming and demanding on computational resources. This can limit the applicability of DNNs to problems with nonstandard, complex, and scarce datasets, e.g., those arising in many scientific applications. To remedy the challenges of DNN training, we propose \tt slimTrain, a stochastic optimization method for training DNNs with reduced sensitivity to the choice of hyperparameters and fast initial convergence. The central idea of slimTrain is to exploit the separability inherent in many DNN architectures; that is, we separate the DNN into a nonlinear feature extractor followed by a linear model. This separability allows us to leverage recent advances made for solving large-scale, linear, ill-posed inverse problems. Crucially, for the linear weights, \tt slimTrain does not require a learning rate and automatically adapts the regularization parameter. In our numerical experiments using function approximation tasks arising in surrogate modeling and dimensionality reduction, slimTrain outperforms existing DNN training methods with the recommended hyperparameter settings and reduces the sensitivity of DNN training to the remaining hyperparameters. Since our method operates on mini-batches, its computational overhead per iteration is modest and savings can be realized by reducing the number of iterations (due to quicker initial convergence) or the number of training problems that need to be solved to identify effective hyperparameters.

Comparison and optimization of deep learning-based radiosensitivity prediction models using gene expression profiling in National Cancer Institute-60 cancer cell line

BackgroundIn this study, various types of deep-learning models for predicting in vitro radiosensitivity from gene-expression profiling were compared. MethodsThe clonogenic surviving fractions at 2 Gy from previous publications and microarray gene-expression data from the National Cancer Institute-60 cell lines were used to measure the radiosensitivity. Seven different prediction models including three distinct multi-layered perceptrons (MLP), four different convolutional neural networks (CNN) were compared. Folded cross-validation was applied to train and evaluate model performance. The criteria for correct prediction were absolute error < 0.02 or relative error < 10%. The models were compared in terms of prediction accuracy, training time per epoch, training fluctuations, and required calculation resources. ResultsThe strength of MLP-based models was their fast initial convergence and short training time per epoch. They represented significantly different prediction accuracy depending on the model configuration. The CNN-based models showed relatively high prediction accuracy, low training fluctuations, and a relatively small increase in the memory requirement as the model deepens. ConclusionOur findings suggest that a CNN-based model with moderate depth would be appropriate when the prediction accuracy is important, and a shallow MLP-based model can be recommended when either the training resources or time are limited.

A novel feedback correction-adaptive Kalman filtering method for the whole-life-cycle state of charge and closed-circuit voltage prediction of lithium-ion batteries based on the second-order electrical equivalent circuit model

Accurate state of charge (SOC) and closed-circuit voltage (CCV) prediction is essential for lithium-ion batteries and their model performance. In this study, a novel feedback correction-adaptive Kalman filtering (FC-AKF) method is proposed for the online battery state co-prediction, which is adaptive to the whole-life-cycle of the lithium-ion battery based on the improved second-order equivalent circuit model (SO-ECM). For the feedback correction strategy, the optimized iterative state initialization is conducted using the uncertainty covariance matrix of the prior three-time points with the convergence of the updating process. The experimental results show that the SOC prediction error of the proposed FC-AKF method is 0.0099% and 0.975% compared with the ampere-hour integral method under the dynamic stress test (DST) and the Beijing bus dynamic stress test (BBDST) working conditions, respectively. Also, the CCV traction by the SO-ECM is 0.80 V and has fast initial convergence and quick prediction error reduction characteristics. The constructed iterative calculation model promotes the accurate SOC and CCV co-prediction effect, improving the safety and longevity of lithium-ion batteries with high precision and fast convergence advantages.

An Adaptive Upper Threshold Based Gain Function for the ZA-PNLMS Algorithm

The recently proposed Zero-Attracting Proportionate Normalized Least Mean Square (ZA-PNLMS) algorithm improves the performance of the PNLMS algorithm for identifying sparse systems. In particular, it keeps the fast initial convergence rate of the PNLMS algorithm, and improves its transient performance by arresting the fall in the convergence rate at the later stage of the adaptation process, and it also improves the steady-state mean square error (MSE). However, the improvement in the steady-state performance is marginal. In this brief, we propose a novel gain function for the ZA-PNLMS algorithm by introducing an adaptive upper threshold parameter. It has two fold implications. First, the proposed threshold parameter truncates the proportional gains of highly active taps when they approach to their steady-states and helps the lesser active taps to converge faster by providing them more gains, and thereby the overall transient performance is further improved. Secondly, it also improves steady-state MSE significantly by reducing the fluctuation of the active taps at the steady-state. Extensive simulation studies have been carried out to justify the improvement obtained by the proposed algorithm.

Robust adaptive filter with lncosh cost

In this paper, a least lncosh (Llncosh) algorithm is derived by utilizing the lncosh cost function. The lncosh cost is characterized by the natural logarithm of hyperbolic cosine function, which behaves like a hybrid of the mean square error (MSE) and mean absolute error (MAE) criteria depending on adjusting a positive parameter λ. Hence, the Llncosh algorithm performs like the least mean square (LMS) algorithm for small errors and behaves as the sign-error LMS (SLMS) algorithm for large errors. It provides comparable performance to the LMS algorithm in Gaussian noise. When compared with several existing robust approaches, the superior steady-state performance and stronger robustness can be attained in impulsive noise. The mean behavior, mean-square behavior and steady-state performance analyses of the proposed algorithm are also provided. In addition, aiming to acquire a compromise between fast initial convergence rate and satisfactory steady-state performance, we introduce a variable-λ Llncosh (VLlncosh) scheme. Lastly, in order to resist the sparsity of the acoustic echo path, an improved proportionate least lncosh (PLlncosh) algorithm is presented. The good performance against impulsive noise and theoretical results of the proposed algorithm are validated by simulations.

Convergence analysis of herded-Gibbs-type sampling algorithms: effects of weight sharing

Herded Gibbs (HG) and discretized herded Gibbs (DHG), which are Gibbs samplings combined with herding, are deterministic sampling algorithms for Markov random fields with discrete random variables. In this paper, we introduce the notion of “weight sharing” to systematically view these HG-type algorithms, and also investigate their convergence theoretically and numerically. We show that, by sharing and reducing the number of weight variables, the HG-type algorithm achieves fast initial convergence at the expense of asymptotic convergence. This means that the HG-type algorithm can be practically more efficient than conventional Markov chain Monte Carlo algorithms, although its estimate does not necessarily converge to the target asymptotically. Moreover, we decompose the numerical integration error of HG-type algorithms into several components and evaluate each of them in relation to herding and weight sharing. By using this formulation, we also propose novel variants of the HG-type algorithm that reduce the asymptotic bias.

On the regularizing property of stochastic gradient descent

Stochastic gradient descent (SGD) and its variants are among the most successful approaches for solving large-scale optimization problems. At each iteration, SGD employs an unbiased estimator of the full gradient computed from one single randomly selected data point. Hence, it scales well with problem size and is very attractive for handling truly massive dataset, and holds significant potentials for solving large-scale inverse problems. In this work, we rigorously establish its regularizing property under a priori early stopping rule for linear inverse problems, and also prove convergence rates under the canonical sourcewise condition. This is achieved by combining tools from classical regularization theory and stochastic analysis. Further, we analyze its preasymptotic weak and strong convergence behavior, in order to explain the fast initial convergence typically observed in practice. The theoretical findings shed insights into the performance of the algorithm, and are complemented with illustrative numerical experiments.

Frequency-Domain Adaptive Kalman Filter With Fast Recovery of Abrupt Echo-Path Changes

The frequency-domain Kalman filter (FDKF) was successfully applied in echo-cancelation systems due to its fast initial convergence and robustness to double-talk. However, the reconvergence ability of the FDKF was not comprehensively resolved and analyzed in the literature. This letter presents the steady-state solutions of the predicted system distance and corresponding step size of the FDKF, which is found to be related to the signal-to-noise ratio and the transition parameter. On this basis, it is pointed out that a frequently observed slow reconvergence of the FDKF, when the sudden echo-path change occurs, can be mainly attributed to the over-estimated noise power spectral density. Finally, we propose a shadow filter approach to resolve the algorithm's lack of reconvergence. Computer simulations of acoustic echo cancelation confirm the improved performance of the proposed method.

A family of gain-combined proportionate adaptive filtering algorithms for sparse system identification

The classical proportionate adaptive filtering (PAF) algorithms achieve a fast initial convergence for sparse impulse response. But the small coefficients receive very little gain so that the time needed to reach steady-state misalignment is increased. In addition, the PAF algorithms converge much slower than the original adaptive filtering (OAF) algorithms when the impulse response is dispersive. In order to address these problems, this paper proposes a family of gain-combined PAF (GC-PAF) algorithms. The gain-combined matrix of the proposed GC-PAF algorithms is implemented by using a sigmoidal activation function to adaptively combine the proportionate matrix and identity matrix, which can retain the advantages of both the PAF algorithms in the context of sparse impulse response and the OAF algorithms in the context of dispersive impulse response. Meanwhile, to be also applicable to the family of sign algorithms against impulsive noise, a general framework for the update of the sigmoidal activation function is obtained by using the gradient descent method to minimize the L1-norm of the system output error. Simulations in the contexts of three different sparsity impulse responses have shown that the proposed GC-PAF algorithms perform much better than the OAF, PAF and improved PAF (IPAF) algorithms.

Adaptive step-size strategy for noise-robust Fourier ptychographic microscopy.

The incremental gradient approaches, such as PIE and ePIE, are widely used in the field of ptychographic imaging due to their great flexibility and computational efficiency. Nevertheless, their stability and reconstruction quality may be significantly degraded when non-negligible noise is present in the image. Though this problem is often attributed to the non-convex nature of phase retrieval, we found the reason for this is more closely related to the choice of the step-size, which needs to be gradually diminishing for convergence even in the convex case. To this end, we introduce an adaptive step-size strategy that decreases the step-size whenever sufficient progress is not made. The synthetic and real experiments on Fourier ptychographic microscopy show that the adaptive step-size strategy significantly improves the stability and robustness of the reconstruction towards noise yet retains the fast initial convergence speed of PIE and ePIE. More importantly, the proposed approach is simple, nonparametric, and does not require any preknowledge about the noise statistics. The great performance and limited computational complexity make it a very attractive and promising technique for robust Fourier ptychographic microscopy under noisy conditions.

A variable neighborhood search particle filter for bearings-only target tracking

In this paper a novel filtering procedure that uses a variant of the variable neighborhood search (VNS) algorithm for solving nonlinear global optimization problems is presented. The base of the new estimator is a particle filter enhanced by the VNS algorithm in resampling step. The VNS is used to mitigate degeneracy by iteratively moving weighted samples from starting positions into the parts of the state space where peaks and ridges of a posterior distribution are situated. For testing purposes, bearings-only tracking problem is used, with two static observers and two types of targets: non-maneuvering and maneuvering. Through numerous Monte Carlo simulations, we compared performance of the proposed filtering procedure with the performance of several standard estimation algorithms. The simulation results show that the algorithm mostly performed better than the other estimators used for comparison; it is robust and has fast initial convergence rate. Robustness to modeling errors of this filtering procedure is demonstrated through tracking of the maneuvering target. Moreover, in the paper it is shown that it is possible to combine the proposed algorithm with an interacted multiple model framework.

Generalized row-action methods for tomographic imaging

Row-action methods play an important role in tomographic image reconstruction. Many such methods can be viewed as incremental gradient methods for minimizing a sum of a large number of convex functions, and despite their relatively poor global rate of convergence, these methods often exhibit fast initial convergence which is desirable in applications where a low-accuracy solution is acceptable. In this paper, we propose relaxed variants of a class of incremental proximal gradient methods, and these variants generalize many existing row-action methods for tomographic imaging. Moreover, they allow us to derive new incremental algorithms for tomographic imaging that incorporate different types of prior information via regularization. We demonstrate the efficacy of the approach with some numerical examples.

Mean-square error and stability analysis of a subband structure for the rapid identification of sparse impulse responses

In a context of supervised adaptive filtering, the sparsity of the impulse response to be identified can be employed to accelerate the convergence rate of the algorithm. This idea was first explored by the so-called proportionate NLMS (PNLMS) algorithm, where the adaptation step-sizes are made larger for the coefficients with larger magnitudes. Whereas fast initial adaptation convergence rate is obtained with the PNLMS algorithm for white-noise input, slow convergence is observed for colored input signals. The combination of the PNLMS approach and a subband structure results in an algorithm with better convergence rate for sparse systems and colored input signals. In this paper, the steady-state mean-square error (MSE) and the maximum value of the step-size @b that allows convergence of the subband PNLMS-type algorithm are analyzed. Theoretical results are confirmed by simulations.

Affine projection algorithm with selective projections

In the affine projection adaptive filtering algorithm, convergence is sped up by increasing the projection order but with an unwelcome consequence of increased steady-state misalignment. To address this unfavorable compromise, we propose a new affine projection algorithm with selective projections. This algorithm adaptively changes the projection order according to the estimated variance of the filter output error. The error variance is estimated using exponential window averaging with a variable forgetting factor and a simple moving averaging technique. The input regressors are selected according to two different criteria to update the filter coefficients at each iteration. Simulations, carried out for different adaptive filtering applications, demonstrate that the new algorithm provides fast initial convergence and low steady-state misalignment without necessarily trading off one for the other in addition to a significant reduction in average computational complexity.

On the Convergence of Primal–Dual Hybrid Gradient Algorithms for Total Variation Image Restoration

In this paper we establish the convergence of a general primal–dual method for nonsmooth convex optimization problems whose structure is typical in the imaging framework, as, for example, in the Total Variation image restoration problems. When the steplength parameters are a priori selected sequences, the convergence of the scheme is proved by showing that it can be considered as an e-subgradient method on the primal formulation of the variational problem. Our scheme includes as special case the method recently proposed by Zhu and Chan for Total Variation image restoration from data degraded by Gaussian noise. Furthermore, the convergence hypotheses enable us to apply the same scheme also to other restoration problems, as the denoising and deblurring of images corrupted by Poisson noise, where the data fidelity function is defined as the generalized Kullback–Leibler divergence or the edge preserving removal of impulse noise. The numerical experience shows that the proposed scheme with a suitable choice of the steplength sequences performs well with respect to state-of-the-art methods, especially for Poisson denoising problems, and it exhibits fast initial and asymptotic convergence.

Interacting fading memory modified input estimation for maneuvering target tracking

The modified input estimation (MIE) has been introduced recently and it provides fast initial convergence rate as well as satisfactory tracking performance in low and medium maneuvering target cases. Unfortunately, it fails to track a high maneuvering target accurately due to modeling errors relating to the target acceleration dynamics. In this paper, an improvement on the MIE is proposed for maneuvering target tracking. With the advantage of reduced sensitivity to modeling errors, a set of fading memory is selected to indicate different maneuver levels. The proposed method jumps between these different fading memories according to a Markov chain. Therefore, it can assign different fading memories different weights in different stages of the target motion to compensate for the influence of model errors. Experimental results show that the proposed method has more efficiency in tracking a maneuvering target than the MIE with conventional fading memory and the simple MIE.