Gradient-based Algorithm Research Articles

A Modified Particle Swarm Optimization Algorithm for Optimizing Artificial Neural Network in Classification Tasks

Artificial neural networks (ANNs) have achieved great success in performing machine learning tasks, including classification, regression, prediction, image processing, image recognition, etc., due to their outstanding training, learning, and organizing of data. Conventionally, a gradient-based algorithm known as backpropagation (BP) is frequently used to train the parameters’ value of ANN. However, this method has inherent drawbacks of slow convergence speed, sensitivity to initial solutions, and high tendency to be trapped into local optima. This paper proposes a modified particle swarm optimization (PSO) variant with two-level learning phases to train ANN for image classification. A multi-swarm approach and a social learning scheme are designed into the primary learning phase to enhance the population diversity and the solution quality, respectively. Two modified search operators with different search characteristics are incorporated into the secondary learning phase to improve the algorithm’s robustness in handling various optimization problems. Finally, the proposed algorithm is formulated as a training algorithm of ANN to optimize its neuron weights, biases, and selection of activation function based on the given classification dataset. The ANN model trained by the proposed algorithm is reported to outperform those trained by existing PSO variants in terms of classification accuracy when solving the majority of selected datasets, suggesting its potential applications in challenging real-world problems, such as intelligent condition monitoring of complex industrial systems.

Optimal Actuator Design via Brunovsky’s Normal Form

In this article, by using the Brunovsky normal form, we provide a reformulation of the problem consisting in finding the actuator design which minimizes the controllability cost for finite-dimensional linear systems with scalar controls. Such systems may be seen as spatially discretized linear partial differential equations with lumped controls. The change of coordinates induced by Brunovsky’s normal form allows us to remove the restriction of having to work with diagonalizable system dynamics and does not entail a randomization procedure as done in past literature on diffusion equations or waves. Instead, the optimization problem reduces to a minimization of the norm of the inverse of a change of basis matrix and allows for an easy deduction of existence of solutions, and for a clearer picture of some of the problem’s intrinsic symmetries. Numerical experiments help to visualize these properties, indicate further open problems, and also show a possible obstruction of using gradient-based algorithms—this is alleviated by using an evolutionary algorithm.

Efficient Light Probes for Real-Time Global Illumination

Reproducing physically-based global illumination (GI) effects has been a long-standing demand for many real-time graphical applications. In pursuit of this goal, many recent engines resort to some form of light probes baked in a precomputation stage. Unfortunately, the GI effects stemming from the precomputed probes are rather limited due to the constraints in the probe storage, representation or query. In this paper, we propose a new method for probe-based GI rendering which can generate a wide range of GI effects, including glossy reflection with multiple bounces, in complex scenes. The key contributions behind our work include a gradient-based search algorithm and a neural image reconstruction method. The search algorithm is designed to reproject the probes' contents to any query viewpoint, without introducing parallax errors, and converges fast to the optimal solution. The neural image reconstruction method, based on a dedicated neural network and several G-buffers, tries to recover high-quality images from low-quality inputs due to limited resolution or (potential) low sampling rate of the probes. This neural method makes the generation of light probes efficient. Moreover, a temporal reprojection strategy and a temporal loss are employed to improve temporal stability for animation sequences. The whole pipeline runs in realtime (>30 frames per second) even for high-resolution (1920×1080) outputs, thanks to the fast convergence rate of the gradient-based search algorithm and a light-weight design of the neural network. Extensive experiments on multiple complex scenes have been conducted to show the superiority of our method over the state-of-the-arts.

A distribution-free smoothed combination method to improve discrimination accuracy in multi-category classification.

Results from multiple diagnostic tests are combined in many ways to improve the overall diagnostic accuracy. For binary classification, maximization of the empirical estimate of the area under the receiver operating characteristic curve has widely been used to produce an optimal linear combination of multiple biomarkers. However, in the presence of a large number of biomarkers, this method proves to be computationally expensive and difficult to implement since it involves maximization of a discontinuous, non-smooth function for which gradient-based methods cannot be used directly. The complexity of this problem further increases when the classification problem becomes multi-category. In this article, we develop a linear combination method that maximizes a smooth approximation of the empirical Hyper-volume Under Manifolds for the multi-category outcome. We approximate HUM by replacing the indicator function with the sigmoid function and normal cumulative distribution function. With such smooth approximations, efficient gradient-based algorithms are employed to obtain better solutions with less computing time. We show that under some regularity conditions, the proposed method yields consistent estimates of the coefficient parameters. We derive the asymptotic normality of the coefficient estimates. A simulation study is performed to study the effectiveness of our proposed method as compared to other existing methods. The method is illustrated using two real medical data sets.

Recent advances in use of bio-inspired jellyfish search algorithm for solving optimization problems

The complexity of engineering optimization problems is increasing. Classical gradient-based optimization algorithms are a mathematical means of solving complex problems whose ability to do so is limited. Metaheuristics have become more popular than exact methods for solving optimization problems because of their simplicity and the robustness of the results that they yield. Recently, population-based bio-inspired algorithms have been demonstrated to perform favorably in solving a wide range of optimization problems. The jellyfish search optimizer (JSO) is one such bio-inspired metaheuristic algorithm, which is based on the food-finding behavior of jellyfish in the ocean. According to the literature, JSO outperforms many well-known meta-heuristics in a wide range of benchmark functions and real-world applications. JSO can also be used in conjunction with other artificial intelligence-related techniques. The success of JSO in solving diverse optimization problems motivates the present comprehensive discussion of the latest findings related to JSO. This paper reviews various issues associated with JSO, such as its inspiration, variants, and applications, and will provide the latest developments and research findings concerning JSO. The systematic review contributes to the development of modified versions and the hybridization of JSO to improve upon the original JSO and present variants, and will help researchers to develop superior metaheuristic optimization algorithms with recommendations of add-on intelligent agents.

Biomechanical Trajectory Optimization of Human Sit-to-Stand Motion With Stochastic Motion Planning Framework

Trajectory optimization has been an important approach in biomechanics for the analysis and prediction of the limb movement. Such approaches have paved the way for the motion planning of biped and quadruped robots as well. Most of these methods are deterministic, utilizing first-order iterative gradient-based algorithms incorporating the constrained differentiable objective functions. However, the limitation of prevailing methods concerning differentiability hinders the implementation of non-differentiable objective functions such as metabolic energy expenditure (MEE) function, which is highly relevant for physiological systems and can even be implemented across the muscular space. This paper consolidates the implementation of the prevalent direct collocation-based optimal control method with the stochastic trajectory optimization method based on Policy Improvement with Path Integral (PI2) for comprehending the human sit-to-stand (STS) motion. PI2 method, which utilizes reinforcement learning of Dynamic Movement Primitive (DMP) to learn a goal-based trajectory is implemented and validated by comparing with the experimental result in joint-space and muscle-space.

B-DeepONet: An enhanced Bayesian DeepONet for solving noisy parametric PDEs using accelerated replica exchange SGLD

The Deep Operator Network (DeepONet) is a neural network architecture used to approximate operators, including the solution operator of parametric PDEs. DeepONets have shown remarkable approximation ability. However, the performance of DeepONets deteriorates when the training data is polluted with noise, a scenario that occurs in practice. To handle noisy data, we propose a Bayesian DeepONet based on replica exchange Langevin diffusion (reLD). Replica exchange uses two particles. The first particle trains a DeepONet to exploit the loss landscape and make predictions. The other particle trains a different DeepONet to explore the loss landscape and escape local minima via swapping. Compared to DeepONets trained with state-of-the-art gradient-based algorithms (e.g., Adam), the proposed Bayesian DeepONet greatly improves the training convergence for noisy scenarios and accurately estimates the uncertainty. To further reduce the high computational cost of the reLD training of DeepONets, we propose (1) an accelerated training framework that exploits the DeepONet's architecture to reduce its computational cost up to 25% without compromising performance and (2) a transfer learning strategy that accelerates training DeepONets for PDEs with different parameter values. Finally, we illustrate the effectiveness of the proposed Bayesian DeepONet using four parametric PDE problems.

Pretreatment PET Metrics and Clinical Outcomes of Anal Cancer in Patients Living with and without HIV

<h3>Purpose/Objective(s)</h3> This study investigates the association between pre-treatment positron emission tomography (PET) metrics and clinical outcomes in anal squamous cell carcinoma (ASCC) patients treated with definitive radiation therapy (RT). We hypothesize that pre-treatment PET metrics may differentially predict clinical outcomes in ASCC patients without HIV infection (HIV-) and ASCC patients living with HIV (PLWH) with detectable or undetectable viral load (VL). <h3>Materials/Methods</h3> Patients treated with definitive RT for non-metastatic ASCC between 2005 - 2021 at a single institution were retrospectively identified, and clinical and demographic data, including HIV status and pretreatment VL, were tabulated. Pre-treatment PETs were imported to MIM, and semiautomatic gradient-based segmentation tool algorithms (PET_Edge) were used to calculate PET metrics, including maximum standardized uptake value (SUVmax), metabolic tumor volume (MTV), and total lesion glycolysis (TLG). Cox proportional hazard modeling was used to investigate the association between primary tumor PET metrics and clinical outcomes, including overall survival (OS), progression free survival (PFS), and loco-regional control (LRC) <h3>Results</h3> 141 ASCC patients were identified, 49 of whom were PLWH. 12 had detectable HIV VL. The median follow-up time was 51.8 months (IQR: 0.5-191.2 months). There were 22 loco-regional failures and 24 deaths. 3- and 5- year LRC was 85.6% and 83.3%, 3- and 5- year PFS was 75.9% and 71.6%, and 3- and 5- year OS was 89.3% and 86.6% for the entire cohort. There was no significant difference in LRC, PFS, or OS between PLWH and HIV- patients (p>0.50). PLWH with detectable VL had similar LRC and PFS but trended towards worse 5-year OS than did HIV- patients (66.3% vs 89.7%, p=0.16). MTV and TLG from primary tumors were significantly associated with LRC, PFS, and OS for all patients and in sub-group analysis for HIV-, PLWH, and PLWH with undetectable VL (p<0.05). SUVmax was significantly associated with LRC, PFS, and OS for all patients, and on sub-group analysis only for HIV- patients (p<0.05). On multivariable analysis, after adjusting for stage and Charlson Comorbidity Index, MTV split at median was significantly associated with LRC (HR 3.59 (95% CI:1.21-10.6), p=0.02) and PFS (HR 3.12 (95%CI:1.48-6.56), p=0.003), and TLG split at median was significantly associated with LRC (HR 4.36 (95%CI:1.36-14.0), p=0.01) and PFS (HR 3.13 (95%CI:1.48-6.64), p=0.003). <h3>Conclusion</h3> Patients with ASCC had similar outcomes regardless of HIV status. Advanced pre-treatment PET metrics, especially MTV and TLG, may have prognostic or risk-stratification utility in HIV- patients and undetectable PLWH with ASCC, but were not as strongly associated with clinical outcomes for PLWH with detectable VL, perhaps due to small numbers of patients with detectable VL in this cohort.

A maiden robust FPIDD2 regulator for frequency-voltage enhancement in a hybrid interconnected power system using Gradient-Based Optimizer

This paper proposes a maiden intelligent controller design that consists of a fuzzy proportional-integral-derivative-double derivative (FPIDD) controller whose parameters are fine-tuned using the Gradient-Based Optimization algorithm (GBO). The proposed FPIDD2 regulator is employed as a secondary regulator for stabilizing the combined voltage and frequency loops in a two-area interconnected power system. The proposed FPIDD2 controller is tested in a two-area hybrid system, with each area comprising a mix of traditional (thermal, gas, and hydraulic power plants) and renewable generation units (wind and solar power). Additionally, the proposed controller takes into account system nonlinearities (such as generation rate limitations and governor deadband), system uncertainties, and load/renewables changes. The dynamic responses of the system demonstrate that FPIDD2 has superior ability to attenuate the deviations in voltage and frequency in both areas of the system. In the investigated hybrid system, the suggested FPIDD2 is compared to a GBO-tuned integral derivative tilted (ID-T) controller and FPID controller. As a fitness function for the GBO, the criteria of minimizing the integral time absolute error (ITAE) are applied. The results are presented in the form of MATLAB/SIMULINK time-domain simulations. The simulation outcomes prove that the presented controller has an outstanding performance compared with the other control strategies in the dynamic response of the system in terms of rising and settling times, maximum overshoot, undershoot values and ITAE. The ITAE value of the FPIDD2 regulator tuned by the GBO technique was enhanced by 90.9% with the GBO-based ID-T and 55.4% with GBO-based FPID controllers.

Triangularized Orthogonalization-Free Method for Solving Extreme Eigenvalue Problems

A novel orthogonalization-free method together with two specific algorithms is proposed to address extreme eigenvalue problems. On top of gradient-based algorithms, the proposed algorithms modify the multicolumn gradient such that earlier columns are decoupled from later ones. Locally, both algorithms converge linearly with convergence rates depending on eigengaps. Momentum acceleration, exact linesearch, and column locking are incorporated to accelerate algorithms and reduce their computational costs. We demonstrate the efficiency of both algorithms on random matrices with different spectrum distributions and matrices from computational chemistry.

An efficient conjugate gradient-based algorithm for unconstrained optimization and its projection extension to large-scale constrained nonlinear equations with applications in signal recovery and image denoising problems

In this paper, a hybrid version of the Rivaie–Mustafa–Ismail–Leong, RMIL for short, conjugate gradient method is proposed for unconstrained optimization problem. The search direction fulfills the sufficient descent condition and trust region property at each iteration, independent of any line search. We establish the global convergence of the proposed algorithm under normal assumptions and Wolfe line search. To move forward, incorporating with derivative-free projection technique, the proposed conjugate coefficient is extended to deal with nonlinear monotone equations. Without Lipschitz continuity assumption, the global convergence is proved with mild conditions. The numerical results for both unconstrained optimization instances and large-scale nonlinear equations prove the efficiency of the proposed methods with less computation cost. In particular, we also explore the practical applications on signal recovery and image denoising problems.

Optimal cross-layer resource allocation in fog computing: A market-based framework

The potential upsides of fog computing are undeniable, including better near-real-time response, reduced transmission costs and analysis for IoT, which will further have a significant impact on operation of factories, enterprises, organizations and even the quality of our life. However, due to the resource-constrained nature, heterogeneity and distributed nodes, how to manage the extraordinarily complex fog computing resources is necessary to improve the QoE of users. This paper investigates the rational resource allocation for fog computing from the cross-layer point of perspective, and formulates the resource allocation scenario as Cloud–Fog–User market which is composed of cloud center devices, fog nodes, and user devices. Firstly, the resource market is divided into Cloud–Fog market containing cloud layer and fog layer, and Fog–User market containing fog layer and user layer based on different scenarios separately. The paper proposes a market-based framework for fog computing networks, which enables the cloud layer and the fog layer to allocate their resources in the form of pricing, payment and supply–demand relationship, while analyzes the relationship among resource allocation participants in the market. Then the paper investigates the utility optimization models from the viewpoints of both the participants of respective markets, so as to optimize optimal payment and optimal resource allocation via convex optimization techniques, and proposes a gradient-based iterative algorithm to optimize the utilities, respectively. Finally, experimental results have displayed the performance of convex analysis results and effectiveness of the proposed algorithm.

Parallel-in-time adjoint-based optimization – application to unsteady incompressible flows

Gradient-based optimization algorithms, where gradient information is extracted using adjoint equations, are efficient but can quickly slow down when applied to unsteady and nonlinear flow problems. This is mainly due to the sequential nature of the algorithm, where the primal problem is first integrated forward in time, providing the initial condition for the adjoint problem, which is then integrated backward. In order to address the sequential nature of this optimization procedure parallel-in-time algorithms can be employed. However, the characteristics of the governing equations of interest in this work, and in particular, the divergence-free constraint (incompressibility effect) as well as the nonlinearity and the unsteadiness of the flow, make direct application of existing parallel-in-time algorithms less than straightforward. In this work, we introduce a parallel-in-time procedure, applied to the integration of the adjoint problem, which addresses all the existing constraints and allows quick access to local gradients. The performance of the proposed algorithm is assessed for both steady and unsteady actuation; in both cases it readily outperforms the sequential algorithm.

A Gradient-Based Phase-Only Synthesis for Solving Q-N Problem of Planar Clustered Phased Arrays for Sidelobe Suppression

In this article, a gradient-based phase-only synthesis approach for solving the Q-N clustering problem of planar clustered phased arrays (CPAs) for sidelobe suppression is presented. Through establishing an appropriate parameterization scheme, the synthesis problem is transformed to an unconstrained continuous problem which is suitable for solving with the gradient-based algorithm. The synthesis is carried out through simultaneously optimizing the clustering variables and the phase variables, and the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> - <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$N$ </tex-math></inline-formula> problem is addressed without reducing the feasible region of the problem. Typical examples are provided to validate the effectiveness of the proposed approach. In addition, by considering several steering angles in the proposed synthesis approach, the scanning issue in the clustering problem is clarified and discussed.

Multimuscle Functional-Electrical-Stimulation-Based Wrist Tremor Suppression Using Repetitive Control

Wrist motion is produced by a group of muscles acting in a coordinated way. However, existing functional electrical stimulation (FES)-based wrist tremor suppression methods just stimulate one pair of muscles, which can limit the tremor suppression performance and cause muscle fatigue. To address these problems, this article proposes a multimuscle FES-based wrist tremor suppression method by fully considering the properties of wrist motion. First, with the consideration of the mainly involved two pairs of muscles in wrist flexion and extension motion, a multimuscle wrist musculoskeletal model with a Hammerstein structure is developed, and the parameters are identified. Then, a feedback repetitive controller combined with a feedforward linearization controller is proposed for tremor suppression. A frequency-modified inverse repetitive control algorithm and a gradient-based repetitive control algorithm are put forward to regulate the FES level properly. Finally, experiments on both unimpaired subjects and intention tremor patients verify that compared to the existing single-muscle-pair FES-based methods, the proposed methods can substantially improve the performance of tremor suppression and effectively reduce the level of electrical stimulation significantly, thereby reducing muscle fatigue.

Conjugate gradient-based iterative algorithm for solving generalized periodic coupled Sylvester matrix equations

This paper focuses on constructing a conjugate gradient-based (CGB) method to solve the generalized periodic coupled Sylvester matrix equations in complex space. The presented method is developed from a point of conjugate gradient methods. It is proved that the presented method can find the solution of the considered matrix equations within finite iteration steps in the absence of round-off errors by theoretical derivation. Some numerical examples are provided to verify the convergence performance of the presented method, which is superior to some existing numerical algorithms both in iteration steps and computation time.

Scalable estimation and inference for censored quantile regression process

Censored quantile regression (CQR) has become a valuable tool to study the heterogeneous association between a possibly censored outcome and a set of covariates, yet computation and statistical inference for CQR have remained a challenge for large-scale data with many covariates. In this paper, we focus on a smoothed martingale-based sequential estimating equations approach, to which scalable gradient-based algorithms can be applied. Theoretically, we provide a unified analysis of the smoothed sequential estimator and its penalized counterpart in increasing dimensions. When the covariate dimension grows with the sample size at a sublinear rate, we establish the uniform convergence rate (over a range of quantile indexes) and provide a rigorous justification for the validity of a multiplier bootstrap procedure for inference. In high-dimensional sparse settings, our results considerably improve the existing work on CQR by relaxing an exponential term of sparsity. We also demonstrate the advantage of the smoothed CQR over existing methods with both simulated experiments and data applications.

A self-adaptive gradient-based particle swarm optimization algorithm with dynamic population topology

The aggregation of individuals facilitates local information exchange, and the migration of individuals from one population to another leads to a dynamic community structure. In addition, the negative feedback regulation mechanism of organisms helps them in good living conditions. Based on the above knowledge, a novel particle swarm optimization algorithm with a self-organizing topology structure and self-adaptive adjustable parameters is proposed (KGPSO). During the optimization process, the K-Means clustering method periodically divides the particle swarm into multiple distance-based sub-swarms, and the optimal number of sub-swarms is determined by maximizing the Calinski-Harabasz index. This strategy helps maintain the population diversity and gives particles the ability to perceive the surrounding environment. The parameters used to update the particle velocity are adjusted based on the gradient descent of its fitness error, ensuring a dynamic balance between exploration and exploitation. The hyperparameters of KGPSO are tuned by Bayesian optimization method to improve the algorithm performance further. Two benchmark suites are used to evaluate the performance of KGPSO. Both ranking results and Wilcoxon signed-rank tests show that KGPSO performs best among the PSO algorithms tested. Moreover, the excellent optimization capability of KGPSO are proven in the process of X-ray CT image enhancement, making it possible to analyze the structure and motion of heterogeneous granular materials efficiently and robustly. In conclusion, the proposed KGPSO can provide a stable and powerful support for the frontier experimental research of granular materials and expand the research scope.

Topology optimization of thin-walled structures with directional straight stiffeners

Continuum topology optimization is a powerful method to find the optimal design of structures. However, it usually leads to complex nonregular configurations of stiffeners for thin-walled structures, which are difficult to manufacture using traditional production techniques. Therefore, this study proposes a novel topology representation and optimization method for thin-walled structures with directional straight stiffeners using the material-field series expansion and multifield superposition. The material field function with an anisotropic correlation is introduced to describe the straight stiffeners in a given direction, and the entire stiffeners’ topology is represented by the superposition of multiple material fields. Moreover, the stiffener's direction can be designed because of the inherent merit of separating the material field and the finite element mesh. The proposed method connects the characteristics of regular shape and topology of directional straight stiffeners into the optimization model without introducing additional manufacturing constraints. The optimization model is effectively solved using a gradient-based optimization algorithm incorporating the design sensitivity information. Several numerical examples are presented to demonstrate that the proposed method can successfully obtain the optimized topology while being easy to manufacture in stiffened structures design problems.

An aeroelastic optimisation framework for manufacturable variable stiffness composite wings including critical gust loads

This work presents a new aeroelastic optimisation framework for the preliminary design of variable stiffness composite wing structures. The framework is constructed by sequentially and iteratively solving two sub-problems: aeroelastic tailoring and lay-up retrieval, using gradient-based algorithms with full-analytical sensitivities provided. During aeroelastic tailoring, the wing mass is minimised by optimising the lamination parameters and thickness of wing laminates together with wing jig twist distribution. The load cases cover not only static loads, but also the critical gust loads that are identified across the entire flight envelop at every iteration of optimisation. Further, a cruise shape constraint is included in addition to other aerostructural constraints, so that the optimal aircraft performance can be ensured. During lay-up retrieval, the manufacturable stacking sequence is retrieved according to the optimal lamination parameters with the consideration of minimal steering radius constraint. Moreover, to fix the possible constraint violations caused by lay-up retrieval, a correction strategy is incorporated to tighten the violated constraints for repeating aeroelastic tailoring. Finally, several case studies on the design of NASA common research model wing are carried out and investigated. The results indicate that the critical gust loads and cruise shape constraint have a large influence on the design of tow-steered composite wing structures, which therefore demonstrate the usefulness and benefits of the proposed optimisation framework.