Gradient-based Algorithm Research Articles

Evolutionary optimization framework to train multilayer perceptrons for engineering applications.

Training neural networks by using conventional supervised backpropagation algorithms is a challenging task. This is due to significant limitations, such as the risk for local minimum stagnation in the loss landscape of neural networks. That may prevent the network from finding the global minimum of its loss function and therefore slow its convergence speed. Another challenge is the vanishing and exploding gradients that may happen when the gradients of the loss function of the model become either infinitesimally small or unmanageably large during the training. That also hinders the convergence of the neural models. On the other hand, the traditional gradient-based algorithms necessitate the pre-selection of learning parameters such as the learning rates, activation function, batch size, stopping criteria, and others. Recent research has shown the potential of evolutionary optimization algorithms to address most of those challenges in optimizing the overall performance of neural networks. In this research, we introduce and validate an evolutionary optimization framework to train multilayer perceptrons, which are simple feedforward neural networks. The suggested framework uses the recently proposed evolutionary cooperative optimization algorithm, namely, the dynamic group-based cooperative optimizer. The ability of this optimizer to solve a wide range of real optimization problems motivated our research group to benchmark its performance in training multilayer perceptron models. We validated the proposed optimization framework on a set of five datasets for engineering applications, and we compared its performance against the conventional backpropagation algorithm and other commonly used evolutionary optimization algorithms. The simulations showed the competitive performance of the proposed framework for most examined datasets in terms of overall performance and convergence. For three benchmarking datasets, the proposed framework provided increases of 2.7%, 4.83%, and 5.13% over the performance of the second best-performing optimizers, respectively.

Optimize Non-Convex Functions within Polynomial Time: Alternative to Gradient-Based Algorithms

Optimize Non-Convex Functions within Polynomial Time: Alternative to Gradient-Based Algorithms

Development of a multi-stage model for freezing of a suspended binary solution droplet

The importance of developing spray freeze-drying technology for extending the shelf life of biological and pharmaceutical materials has never been greater, given the increasing food shortage and the strong demand for pharma processes. In particular, the optimal thermal design and application of spray freeze-drying technologies now depend on the estimation of nucleation behavior for droplet solidification (especially the droplets of binary mixtures). Although earlier nucleation models could estimate the nucleation rate and temperature of solidifying droplets, few considered extreme environmental factors, such as extremely low ambient temperatures below −60C∘. To ensure the preservation and storage of biological and pharmaceutical products, such as vaccines and protein drugs, these conditions are essential. Hence, developing an accurate and trustworthy mathematical framework for simulating nucleation is paramount. In this study, a multi-stage, hybrid analytical-numerical model for droplet solidification is developed while coupled with a gradient-based optimization algorithm. Specifically, the five-stage solidification of binary mixtures is simulated (including supercooling of liquid, nucleation, recalescence, equilibrium freezing, and subcooling of solid), which captures the dynamic behaviors of temperature and composition or solute concentration during phase change. The heterogeneous nucleation of a binary-mixture droplet in a frigid environment is predicted and validated against a series of experiments on single suspended droplets at a wide range of ambient temperatures between −20 and −160C∘. The freezing curves of different solute concentrations are also validated against experimental data. It is found that significant variations in interfacial tension lead to abrupt changes in nucleation temperature for extremely cold environments. Further, the effects of the concentration, contact angle, droplet size, and heat transfer coefficient are investigated.

A fast interpretable adaptive meta-learning enhanced deep learning framework for diagnosis of diabetic retinopathy

Gradient-based meta-learning algorithms offer promising solutions to the challenge of swift adaptation to new tasks, especially when faced with limited sample data. One pivotal concern in few-shot classification tasks is striking the right balance between interpretability and accuracy within the meta-learning framework. This study introduces a novel methodology titled Fast, Interpretable, and Adaptive meta-Learning based on Logistic Regression (FIAML-LR). Distinctively, FIAML-LR employs logistic regression to craft a meta-network in the inner loop. This design facilitates faster generation of the learning rate and weight attenuation coefficient, enhancing the interpretability of meta-learning for new task adaptation. An adaptable parameter update strategy is also embedded, initializing with a broader hyperparameter adjustment scope and fine-tuning progressively throughout the experiment. Experimental evidence reveals that, when implemented on a 4-CONV architecture, FIAML-LR not only bolsters the model's interpretability but also amplifies its accuracy for few-shot classification tasks. A focused investigation on the diabetic retinopathy dataset demonstrated that FIAML-LR, even with limited data, could boost classification accuracy by a significant 14.28% against the benchmark model. This heightened accuracy could aid medical professionals in more precisely diagnosing diabetic retinopathy.

Convergence analysis of kernel conjugate gradient for functional linear regression

In this paper, we discuss the convergence analysis of the conjugate gradient-based algorithm for the functional linear model in the reproducing kernel Hilbert space framework, utilizing early stopping results in regularization against over-fitting. We establish the convergence rates depending on the regularity condition of the slope function and the decay rate of the eigenvalues of the operator composition of covariance and kernel operator. Our convergence rates match the minimax rate available from the literature.

Neural Network-Based Hammerstein Model Identification of a Lab-Scale Batch Reactor.

This paper focuses on two types of neural network-based Hammerstein model identification methods for the acrylamide polymerization reaction of a batch reactor process. The first neural-based identification type formulates the weights of the multilayer network directly as parameters of the nonlinear static and linear dynamic blocks of the Hammerstein model and trains the weights using a gradient-based backpropagation algorithm. In the second identification type, the nonlinear static block of the Hammerstein model is framed as a single hidden-layer feedforward network and both nonlinear and linear block parameters are trained using an extreme learning machine, where the training procedure is exempted from gradient calculation. The primary focus of the paper is neural-based model identification of a complex nonlinear system, which facilitates ease of linear/nonlinear controller design with good learning speed and less computations. A future work toward the machine learning-based nonlinear model predictive controller implementation using the Jetson Orin Nano board is also described.

Automated facial characterization and image retrieval by convolutional neural networks.

Developing efficient methods to infer relations among different faces consisting of numerous expressions or on the same face at different times (e.g., disease progression) is an open issue in imaging related research. In this study, we present a novel method for facial feature extraction, characterization, and identification based on classical computer vision coupled with deep learning and, more specifically, convolutional neural networks. We describe the hybrid face characterization system named FRetrAIval (FRAI), which is a hybrid of the GoogleNet and the AlexNet Neural Network (NN) models. Images analyzed by the FRAI network are preprocessed by computer vision techniques such as the oriented gradient-based algorithm that can extract only the face region from any kind of picture. The Aligned Face dataset (AFD) was used to train and test the FRAI solution for extracting image features. The Labeled Faces in the Wild (LFW) holdout dataset has been used for external validation. Overall, in comparison to previous techniques, our methodology has shown much better results on k-Nearest Neighbors (KNN) by yielding the maximum precision, recall, F1, and F2 score values (92.00, 92.66, 92.33, and 92.52%, respectively) for AFD and (95.00% for each variable) for LFW dataset, which were used as training and testing datasets. The FRAI model may be potentially used in healthcare and criminology as well as many other applications where it is important to quickly identify face features such as fingerprint for a specific identification target.

Optimization approaches for the design and operation of open-loop shallow geothermal systems

Abstract. The optimization of open-loop shallow geothermal systems, which includes both design and operational aspects, is an important research area aimed at improving their efficiency and sustainability and the effective management of groundwater as a shallow geothermal resource. This paper investigates various approaches to address optimization problems arising from these research and implementation questions about GWHP systems. The identified optimization approaches are thoroughly analyzed based on criteria such as computational cost and applicability. Moreover, a novel classification scheme is introduced that categorizes the approaches according to the types of groundwater simulation model and the optimization algorithm used. Simulation models are divided into two types: numerical and simplified (analytical or data-driven) models, while optimization algorithms are divided into gradient-based and derivative-free algorithms. Finally, a comprehensive review of existing approaches in the literature is provided, highlighting their strengths and limitations and offering recommendations for both the use of existing approaches and the development of new, improved ones in this field.

New modified Bat algorithm for blind speech enhancement in time domain

We address the speech enhancement problem for dual convolutif mixed channel by viewing it in a blind separation sources setting. One widely used technique to separate mixed signals is to apply adaptive filtering, the challenge is to identify an unknown finite impulse response. traditionally we apply a gradient-based algorithm to adapt filter coefficients. However, such algorithm often suffers from premature convergence ,when using large filters and non-stationary inputs , leading to the so-called local minimum problem , which affects the quality of enhanced signals significatively .one alternative to overcome this problem is to apply a population-based metaheuristic algorithms in which filter coefficients are adapted iteratively by minimizing a cost function .But even with this metaheuristic based solution, local minimum problem at large filters still exists. In order to avoid local minima and improve the chance to reach the global solution, we propose in this paper, a novel algorithm called a modified bat algorithm to render the search process efficiently by enhancing its capability of exploration and exploitation. Several experiments under different noise types are carried out using our proposed modified bat algorithm in comparison with some of the popular state-of-the-art algorithms. The enhanced signals obtained by each algorithm at the separation process outputs, show good behavior and superiority of our proposed algorithm. In terms of system misalignment, as well as a segmental signal-to-noise ratio.

Multi-agent deep reinforcement learning based real-time planning approach for responsive customized bus routes

Customized bus can meet many passengers’ personalized travel demand in a public transportation system by providing an innovative shared travel service. Customized bus offers multiple bus routes that jointly form a route network to serve its passengers. It must frequently adjust the station sequences of each bus route in response to trip cancellations and new trip bookings during its operation. Different from relevant research works in the literature, this paper proposes a multi-agent deep reinforcement learning based real-time planning approach for tackling the multiple customized bus routes planning problem. We model the problem as a multi-agent Markov decision process for the first time in literature where a separate agent is assigned to each route to plan its station sequence. We then develop a new multi-agent system. Each agent in the system is powered by an encoder–decoder neural network that consolidates the station sequence decision policy for each bus route. We employ a policy gradient-based reinforcement learning algorithm to train the network parameters of the multi-agent system so as to maximize the number of passengers served while ensuring the customized bus service quality and minimizing the operating cost of all customized bus routes. On three (six) problem instances in offline (online) scenarios, the trained multi-agent system can significantly outperform several existing algorithms in terms of the total cost, adaptiveness and computation time.

Iterative Approach for Traffic Assignment with Capacity Limits using Taylor Series Approximation

This study presents a method for reaching user equilibrium in network traffic assignment problems with capacity constraints. The presented approach minimizes each user’s own travel time using a gradient-based algorithm based on the Taylor series. The algorithm is shown to converge efficiently to user equilibrium after a limited number of iterations. A numerical example is provided to demonstrate the effectiveness of the presented approach, and comparisons are made with other algorithms available in the literature. The obtained results show that the presented method is capable of reaching user equilibrium for capacity network traffic assignment problems.

Value-Function-Based Sequential Minimization for Bi-Level Optimization.

Gradient-based Bi-Level Optimization (BLO) methods have been widely applied to handle modern learning tasks. However, most existing strategies are theoretically designed based on restrictive assumptions (e.g., convexity of the lower-level sub-problem), and computationally not applicable for high-dimensional tasks. Moreover, there are almost no gradient-based methods able to solve BLO in those challenging scenarios, such as BLO with functional constraints and pessimistic BLO. In this work, by reformulating BLO into approximated single-level problems, we provide a new algorithm, named Bi-level Value-Function-based Sequential Minimization (BVFSM), to address the above issues. Specifically, BVFSM constructs a series of value-function-based approximations, and thus avoids repeated calculations of recurrent gradient and Hessian inverse required by existing approaches, time-consuming especially for high-dimensional tasks. We also extend BVFSM to address BLO with additional functional constraints. More importantly, BVFSM can be used for the challenging pessimistic BLO, which has never been properly solved before. In theory, we prove the convergence of BVFSM on these types of BLO, in which the restrictive lower-level convexity assumption is discarded. To our best knowledge, this is the first gradient-based algorithm that can solve different kinds of BLO (e.g., optimistic, pessimistic, and with constraints) with solid convergence guarantees. Extensive experiments verify the theoretical investigations and demonstrate our superiority on various real-world applications.

Incorporating gradient information into dimension perturbation mutation for high-dimensional expensive optimization

Surrogate-assisted evolutionary algorithms (SAEAs) have recently shown excellent abilities in solving computationally expensive optimization problems (EOPs), but most of them are not designed for high-dimensional EOPs (HEOPs). A new algorithm where the gradient information is well-incorporated into dimension perturbation mutation called as GIDPM for HEOPs, is proposed and fully investigated in this paper. Specifically, for each solution, a local candidate offspring pool is obtained by combining the gradient descent information and dimension perturbation mutation, and the DE/current-to-rand/2/bin is employed to generate a global candidate offspring pool. Then an offspring selection strategy is used to identify the exact offspring solutions, in which the overall qualities of two pools, the directions of near gradient descent of candidate solutions, and the distributions and RBF predictions of candidate solutions are considered in turn, to achieve appropriate local exploitation or global exploration. In addition, an Autoencoder-embedded search strategy is used to quickly jump out of the local optima caused by some small ripples or oscillations of the landscape after each several iterations to achieve leapfrog search. The GIDPM maintains a good balance between exploration and exploitation. Experimental studies on benchmark problems demonstrate that the GIDPM achieves better convergence performance when the problem dimension is larger than or equal to 30, and the better performance of GIDPM can be achieved with the increase of dimension. The GIDPM is also used for the design of the sandwich panel with truss cores. Compared with gradient-based algorithm, the GIDPM obtains the design with better structural performance.

NISQ-compatible approximate quantum algorithm for unconstrained and constrained discrete optimization

Quantum algorithms are getting extremely popular due to their potential to significantly outperform classical algorithms. Yet, applying quantum algorithms to optimization problems meets challenges related to the efficiency of quantum algorithms training, the shape of their cost landscape, the accuracy of their output, and their ability to scale to large-size problems. Here, we present an approximate gradient-based quantum algorithm for hardware-efficient circuits with amplitude encoding. We show how simple linear constraints can be directly incorporated into the circuit without additional modification of the objective function with penalty terms. We employ numerical simulations to test it on MaxCut problems with complete weighted graphs with thousands of nodes and run the algorithm on a superconducting quantum processor. We find that for unconstrained MaxCut problems with more than 1000 nodes, the hybrid approach combining our algorithm with a classical solver called CPLEX can find a better solution than CPLEX alone. This demonstrates that hybrid optimization is one of the leading use cases for modern quantum devices.

The Weighted, Relaxed Gradient-Based Iterative Algorithm for the Generalized Coupled Conjugate and Transpose Sylvester Matrix Equations

By applying the weighted relaxation technique to the gradient-based iterative (GI) algorithm and taking proper weighted combinations of the solutions, this paper proposes the weighted, relaxed gradient-based iterative (WRGI) algorithm to solve the generalized coupled conjugate and transpose Sylvester matrix equations. With the real representation of a complex matrix as a tool, the necessary and sufficient conditions for the convergence of the WRGI algorithm are determined. Also, some sufficient convergence conditions of the WRGI algorithm are presented. Moreover, the optimal step size and the corresponding optimal convergence factor of the WRGI algorithm are given. Lastly, some numerical examples are provided to demonstrate the effectiveness, feasibility and superiority of the proposed algorithm.

Enhanced decomposition-based hybrid evolutionary and gradient-based algorithm for many-objective optimization

Enhanced decomposition-based hybrid evolutionary and gradient-based algorithm for many-objective optimization

Learning-based demand-supply-coupled charging station location problem for electric vehicle demand management

We present a learning-based, demand-supply-coupled optimization model for the charging station location problem (CSLP), aiming to integrate the concept of electric vehicle (EV) charging demand management into the planning of charging infrastructures. In stage one, a gradient boosting-based learning model is developed to predict the charging demand of a charging station based on 15 defined features. Next, in stage two, a demand–supply-coupled CSLP model is developed to optimize the total charging usage rates of both existing and newly selected charging stations. We design a gradient-based stochastic spatial search algorithm to solve the proposed model. A case study with 6-year charging event data from Kansas City Missouri is performed. Results show that the proposed method can generate satisfactory charging demand predictions, and can increase charging usage rates by 14%, outperforming two benchmark approaches. The results of this research are poised to guide agencies in identifying optimal locations for new charging stations.

Orientation optimization via topological derivatives in combination with multi-material topology optimization based on extended level set method

This paper provides an orientation angle optimization method for the design of fiber-reinforced composite materials using topology optimization. The orientation angle optimization is based on a topological derivative, which measures the sensitivity of an objective function with respect to a topological change of anisotropic materials. The sensitivity is incorporated into a new gradient-based optimization algorithm. This method improves our ability to obtain better optimal candidates. We provide some numerical examples and verify the effectiveness of the proposed method.

Optimization on the symplectic Stiefel manifold: SR decomposition-based retraction and applications

Numerous problems in optics, quantum physics, stability analysis, and control of dynamical systems can be brought to an optimization problem with matrix variable subjected to the symplecticity constraint. As this constraint nicely forms a so-called symplectic Stiefel manifold, Riemannian optimization is preferred, because one can borrow ideas from unconstrained optimization methods after preparing necessary geometric tools. Retraction is arguably the most important one which decides the way iterates are updated given a search direction. Two retractions have been constructed so far: one relies on the Cayley transform and the other is designed using quasi-geodesic curves. In this paper, we propose a new retraction which is based on an SR matrix decomposition. We prove that its domain contains the open unit ball which is essential in proving the global convergence of the associated gradient-based optimization algorithm. Moreover, we consider three applications—symplectic target matrix problem, symplectic eigenvalue computation, and symplectic model reduction of Hamiltonian systems—with various examples. The extensive numerical comparisons reveal the strengths of the proposed optimization algorithm.

An Online Joint Optimization–Estimation Architecture for Distribution Networks

In this article, we propose an optimal joint optimization–estimation architecture for distribution networks, which jointly solves the optimal power flow (OPF) problem and static state estimation (SE) problem through an online gradient-based feedback algorithm. The main objective is to enable a fast and timely interaction between the OPF decisions and state estimators with limited sensor measurements. First, convergence and optimality of the proposed algorithm are analytically established. Then, the proposed gradient-based algorithm is modified by introducing statistical information of the inherent estimation and linearization errors for an improved and robust performance of the online OPF decisions. Overall, the proposed method eliminates the traditional separation of operation and monitoring, where optimization and estimation usually operate at distinct layers and different time scales. Hence, it enables a computationally affordable, efficient, and robust online operational framework for distribution networks under time-varying settings.