Gradient-based Optimization Method Research Articles

An online framework for survival analysis: reframing Cox proportional hazards model for large data sets and neural networks.

In many biomedical applications, outcome is measured as a "time-to-event" (e.g., disease progression or death). To assess the connection between features of a patient and this outcome, it is common to assume a proportional hazards model and fit a proportional hazards regression (or Cox regression). To fit this model, a log-concave objective function known as the "partial likelihood" is maximized. For moderate-sized data sets, an efficient Newton-Raphson algorithm that leverages the structure of the objective function can be employed. However, in large data sets this approach has two issues: (i) The computational tricks that leverage structure can also lead to computational instability; (ii) The objective function does not naturally decouple: Thus, if the data set does not fit in memory, the model can be computationally expensive to fit. This additionally means that the objective is not directly amenable to stochastic gradient-based optimization methods. To overcome these issues, we propose a simple, new framing of proportional hazards regression: This results in an objective function that is amenable to stochastic gradient descent. We show that this simple modification allows us to efficiently fit survival models with very large data sets. This also facilitates training complex, for example, neural-network-based, models with survival data.

Shape-constrained geophysical inversion

Two long-standing challenges for geophysical inverse problems are nonuniqueness (especially in the multiparameter setting) and the slow convergence of gradient-based optimization methods. We introduce shape-based inversion algorithms to address these challenges. The shape-based algorithms determine, based on the available geophysical data, the underlying subsurface structures (e.g., stratigraphic layers) and the spatially varying unknown physical parameters (e.g., velocity and density). Inverting simultaneously for geologic structures and physical parameters eliminates degeneracies between parameters and also provides a multiscale framework for decomposing the inverse problem. First, we provide a rigorous mathematical formulation of shape-based algorithms through diffuse interface approximations. Then, we demonstrate the applicability of these algorithms to a range of synthetic model problems in traveltime tomography, full-waveform inversion, gradiometry, and history matching of single-phase flow. These examples highlight the ability of shape-based inversion to mitigate nonuniqueness and improve the rate of convergence to a physically realistic solution.

Toward Evaluating the Reliability of Deep-Neural-Network-Based IoT Devices

Nowadays, the impressive performance of deep neural networks (DNNs) greatly advances the development of Internet of Things (IoT) in diverse scenarios. However, the exceptional vulnerability of DNNs to adversarial attack leads IoT devices to be exposed to potential security issues. Up to now, since adversarial training empirically remains robust against gradient-based adversarial attacks, it is believed to be the most effective defense method. In this article, we find that adversarial examples generated by gradient-based adversarial attacks tend to be less imperceptible induced by the gradient-based optimization methods (adopted in the attacks) being difficult on searching the most effective adversarial examples (i.e., the global extreme points), which may lead to an inaccurate estimation for the effectiveness of the adversarial training. To overcome the inherent defect of gradient-based adversarial attacks, we propose a novel adversarial attack named nongradient attack (NGA), of which search strategy is effective but no longer depends on gradients to enhance the threat of adversarial examples. In detail, NGA first initializes the adversarial examples outside, rather than inside, of decision boundary to make them misclassified by the model and then, under without violation of misclassified condition, adjusts the adversarial examples toward the crafted direction to close the original examples. Extensive experiments show that NGA significantly outperforms the state-of-the-art adversarial attacks on attack success rate (ASR) by 2%–7%. Moreover, we propose a new evaluation metric, i.e., composite criterion (CC) based on both ASR and accuracy, to better measure the effectiveness of adversarial training. In the experiments, CC has shown to be a more comprehensive yet appropriate evaluation metric.

Sequential adaptive switching time optimization technique for optimal control problems

The control parameterization method used together with time-scaling transformation is an effective approach to transform optimal control problems into optimal parameter selection problems. The approximate problems can then be solved by gradient-based optimization methods. However, the conventional time-scaling transformation requires that all the control component functions switch simultaneously, which can be undesirable in practice. In this paper, we introduce a novel technique where the switching times for each of the control component functions can be adaptively selected. To demonstrate the effectiveness and flexibility of the proposed approach, two example problems with control input and terminal state constraints are solved using the method proposed. Numerical results show that the new method achieves much better objective values with some increase in computation time compared to the conventional time-scaling transformation technique.

LINES: Log-Probability Estimation via Invertible Neural Networks for Enhanced Sampling.

It is very challenging to sample a molecular process with large activation energies using molecular dynamics simulations. Current enhanced sampling methodologies, such as umbrella sampling and metadynamics, rely on the identification of appropriate reaction coordinates for a system. In this paper, we developed a method for log-probability estimation via invertible neural networks for enhanced sampling (LINES). This iterative scheme utilizes a normalizing flow machine learning model to learn the underlying free energy surface (FES) of a system as a function of molecular coordinates and then applies a gradient-based optimization method to the learned normalizing flow to identify reaction coordinates. A biasing potential is then evaluated over a tabulated grid of the reaction coordinate values, which can be applied to the next round of simulations for enhanced sampling, resulting in more efficient sampling. We tested the accuracy and efficiency of the LINES method in sampling the FES using the alanine dipeptide system. We also demonstrated the effectiveness of identification of reaction coordinates through simulation of cyclobutanol unbinding from β-cyclodextrin and the folding/unfolding of CLN025─a variant of the peptide Chignolin. The LINES method can be extended to the study of large-scale protein systems with complex nonlinear reaction pathways.

Efficient Design of Thermoelastic Structures Using a Krylov Subspace Preconditioner and Parallel Sensitivity Computation

The repeated updating of parametric designs is computationally challenging, especially for large-scale multi-physics models. This work is focused on proposing an efficient modal modification method for gradient-based topology optimization of thermoelastic structures, which is essential when dealing with their complex eigenproblems and global sensitivity analysis for a huge number of design parameters. The degrees of freedom of the governing equation of thermoelastic structures is very huge when its parametric partial differential equation is discretized using the numerical technique. A Krylov subspace preconditioner is constructed based on the Neumann series expansion series so that the thermoelastic eigenproblem can be solved in an efficient low-dimension solver, rather than its original high-fidelity solver. In the construction of Krylov reduced-basis vectors, the computational cost of the systemic matrix inverse becomes a critical issue, which is solved efficiently by means of constructing a diagonal systemic matrix with the lumped mass and heat generation submatrices. Then, the reduced-basis preconditioner can provide an efficient optimal solver for both the thermoelastic eigenproblem and its eigen sensitivity. Furthermore, a master-slave pattern parallel method is developed to reduce the computational time of computing the global sensitivity numbers, and therefore, the global sensitivity problem can be efficiently discretized into element-scale problems in a parallel way. The sensitivity numbers can thus be solved at the element scale and aggregated to the global sensitivity number. Finally, two case studies of the iterative topology optimization process, in which the proposed modal modification method and the traditional method are implemented, are used to illustrate the effectiveness of the proposed method. Numerical examples show that the proposed method can reduce the computational cost remarkably with acceptable accuracy.

Deep and Convolutional Neural Networks for identifying vertically-propagating incoming seismic wave motion into a heterogeneous, damped soil column

Identification of the incoming seismic wave motion at a geotechnical site plays an integral role in the seismic analysis and design of the critical components of the civil infrastructure. Current practice relies on using one-dimensional models for the soil medium at the site, and a process called deconvolution, which identifies seismic input due to measurements made at the ground surface. Using multi-dimensional models for the underlying soil medium becomes important when a site exhibits considerable heterogeneity or change in topography. These situations necessitate using a gradient-based optimization method for inverting the multi-dimensional incoming seismic-wave motion. However, such methods are computationally expensive and time consuming.We explore the effectiveness and robustness of a data-informed framework for the inverse-source problem, due to its potential in reducing the computational cost, compared to a gradient-based approach. We design deep and convolutional neural network architectures to predict the incoming wave motion based on measurements made at the ground surface. We demonstrate their effectiveness and robustness on blind test examples, where measured data are contaminated with noise, and when the incident signals in the training data set may or may not resemble a realistic seismic signal. Lastly, the presented artificial neural networks are shown to be effective in predicting incoming wave motion when the subsurface material properties lack accuracy, or are uncertain, which is likely the case in realistic situations. While only one-dimensional problems are considered here, generalization of our data-informed approach to handle multi-dimensional problems appears to be straightforward. Overall, our data-informed approach seems to be robust, fast, and promising for identifying the incoming seismic wave motion.

Gradient-based shape optimization for unsteady turbulent simulations using field inversion and machine learning

This paper presents a method for gradient-based shape optimization using unsteady models of turbulent flowfields, for which forward simulations are already expensive and adjoint calculations diverge or require costly regularization. The proposed method targets an objective and constraints computed from the time-averaged flowfield and does not use either an unsteady adjoint nor an input-to-output surrogate. The method relies on the field-inversion, machine-learning (FIML) approach, in which a correction field modifies the production term in a Reynolds-averaged Navier-Stokes (RANS) model of the flow. Steady, adjoint-based, field inversion yields this correction field, and a neural-network model is trained, only from readily-available unsteady primal data, to reproduce the field from local flow quantities. Gradient-based shape optimization is then performed using the corrected RANS model, which must include a linearization of the correction field calculation for accurate gradients. The complete design optimization loop consists of iterations of unsteady simulation, FIML, and steady adjoint-based optimization. Low and moderate Reynolds number airfoil optimization problems demonstrate the performance of the proposed method, and comparisons to RANS-alone designs illustrate the importance of accounting for the unsteady flow effects in the optimization.

Gradient-based optimisation of rectangular honeycomb core sandwich panels

When subjected to bending loads, sandwich panels are highly efficient structural components with the potential to achieve substantial weight reduction. A successful design methodology for sandwich panels should aim at maximising this potential for weight reduction while considering the various possible failure mode in a simple yet accurate manner. This paper investigates the application of steel sandwich panels as two-way deck systems. Near-optimal designs for all-steel Rectangular Honeycomb Core Sandwich Panels (RHCSPs) under general out-of-plane loading are achieved using a gradient-based optimisation method. The method relies on continuously optimising the design limit state constraints while the response constraints are considered a priori in the analysis stage using simplified analytical assessment. Plate bending solutions and sandwich bending solutions are used as alternatives to estimate the internal stresses on each layer of the sandwich panel under out-of-plane loads, where comparisons are made between these two analysis methods in terms of computational efficiency and accuracy. The internal stresses are then used to formulate design limit state equations for each relevant failure mode, including material yielding, plate buckling and deformation control. The Method of Moving Asymptotes is used for the optimisation of RHCSPs, considering the limit states as the constraints of the optimisation problem and weight as the objective function to be minimised. The proposed methodology for simplified assessment is verified against detailed nonlinear finite element models for optimal design solutions. The implications of the results of the proposed optimisation strategy on the development of a systematic design methodology for RHCSPs are also highlighted, making specific reference to critical failure modes.

Multiscale design of nonlinear materials using reduced-order modeling

A two-step optimization method is proposed for the efficient multiscale design of heterogeneous materials whose nonlinear macroscopic response is driven by volumetric and interfacial damage taking place at the microstructural level. The Eigendeformation-based reduced-order Homogenization Method (EHM) is used in a reduced-order design phase, allowing for a preliminary design that combines multiple initial configurations with a gradient-based optimization method. The preliminary design is then adopted to initialize a high-fidelity optimization phase based on the Interface-Enriched Generalized Finite Element Method (IGFEM) to arrive at the final design. The analytic sensitivities of EHM with respect to nonlinear material properties are derived to drive the gradient-based algorithm. Several multiscale optimization problems of increasing complexity are presented to illustrate the efficiency of the proposed method using an objective function based on stress error relative to a desired stress–strain response.

Huberized one-class support vector machine with truncated loss function in the primal space

One-class support vector machine (OCSVM) is an important tool in machine learning and has been extensively used for one-class classification problems. The traditional OCSVM solves the primal problem by solving the dual problem, which is a quadratic programming problem. However, the computation of the quadratic programming is cubic and the storage complexity is quadratic with problem scale, so it is inefficient for training large-scale problems. In this paper, we propose to train OCSVM in primal space directly. Unfortunately, owing to the non-differentiability of hinge loss used in OCSVM, the OCSVM cannot be solved by the gradient-based optimization method which is first-order method that converges fast. On the other hand, the hinge loss is unbounded which makes the OCSVM less robust to outliers. The outliers will make the decision boundary severely deviate from the optimal hyperplane. To overcome the drawbacks, a huberized truncated loss function which is a nonconvex differentiable function is proposed to improve the robustness of the OCSVM. The huberized truncated loss function is insensitive to outliers as a substitute for hinge loss in traditional OCSVM. In contrast to traditional OCSVM, the primal objective function of robust OCSVM is differentiable. Considering the non-convexity of the optimization problem, we employ an accelerated proximal gradient algorithm to solve the robust OCSVM in the primal space. The numerical experiments on benchmark datasets and handwritten digit datasets show that the proposed method not only improves the robustness of the OCSVM , but also can reduce the computational complexity.

Reflection-based traveltime and waveform inversion with second-order optimization

Reflection-based inversion that aims to reconstruct the low-to-intermediate wavenumbers of the subsurface model, can be a complementary to refraction-data-driven full-waveform inversion (FWI), especially for the deep target area where diving waves cannot be acquired at the surface. Nevertheless, as a typical nonlinear inverse problem, reflection waveform inversion may easily suffer from the cycle-skipping issue and have a slow convergence rate, if gradient-based first-order optimization methods are used. To improve the accuracy and convergence rate, we introduce the Hessian operator into reflection traveltime inversion (RTI) and reflection waveform inversion (RWI) in the framework of second-order optimization. A practical two-stage workflow is proposed to build the velocity model, in which Gauss-Newton RTI is first applied to mitigate the cycle-skipping problem and then Gauss-Newton RWI is employed to enhance the model resolution. To make the Gauss-Newton iterations more efficiently and robustly for large-scale applications, we introduce proper preconditioning for the Hessian matrix and design appropriate strategies to reduce the computational costs. The example of a real dataset from East China Sea demonstrates that the cascaded Hessian-based RTI and RWI have good potential to improve velocity model building and seismic imaging, especially for the deep targets.

Improvement of Maneuverability Within a Multiagent Fuzzy Transportation System With the Use of Parallel Biobjective Real-Coded Genetic Algorithm

Over the past two decades, several simulation-based approaches have been developed to seek optimal solutions in complex multiagent systems (MASs). One example of these complex systems is the proposed multiagent fuzzy transportation system (FTS), in which agents, such as manned ground vehicles (MGVs) and unmanned ground vehicles (UGVs), use fuzzy logic for maneuvering while interacting with each other. The complexity of an FTS as an optimization problem is caused by two major factors, viz., inability to apply gradient-based optimization methods and uncertainty present in the FTS model. In these MASs, the objective functions are computed as an outcome of multiple agent-to-agent interactions. For the proposed FTS, two interconnected objectives are of prime importance while modeling the system: the number of potential traffic accidents, which should be minimized, and the output traffic that needs to be maximized. The resulting model is a simulation-based biobjective optimization problem. In an attempt to improve the maneuverability at various road network configurations (RNCs), in this work, a new parallel real-coded genetic algorithm based on fuzzy clustering (FCGA) is proposed that considers the specific requirements of the FTS model. Use of the FCGA aggregated with the FTS overcomes the difficulties associated with the optimization model and the limitations caused by simulation-based optimization (the large dimensionality of decision space, uncertainty present in the MAS environment, etc.). After showing superior performance in approximating the Pareto-optimal solutions for known test instances, the proposed FCGA is applied to the proposed multiagent FTS to approximate the optimal parameters. Because of the optimization of FTS parameters with the FCGA and the use of the proposed fuzzy clustering algorithm (FCA) for the traffic density estimate, the suggested approach substantially improves the maneuverability of UGVs and MGVs to change lanes while considering turning based on fuzzy rules and overtaking.

Bi-objective design-for-control for improving the pressure management and resilience of water distribution networks

This paper investigates control and design-for-control strategies to improve the resilience of sectorized water distribution networks (WDN), while minimizing pressure induced pipe stress and leakage. Both evolutionary algorithms (EA) and gradient-based mathematical optimization approaches are investigated for the solution of the resulting large-scale non-linear (NLP) and bi-objective mixed-integer non-linear programs (BOMINLP). While EAs have been successfully applied to solve discrete network design problems for large-scale WDNs, gradient-based mathematical optimization methods are more computationally efficient when dealing with large search spaces associated with continuous variables in optimal network control problems. Considering the advantages of each method, we propose a sequential hybrid method for the optimal design-for-control of large-scale WDNs, where a refinement stage relying on gradient-based mathematical optimization is used to solve continuous optimal control problems corresponding to design solutions returned by an initial EA search. The proposed method is applied to compute the Pareto front of a bi-objective design-for-control problem for the operational network BWPnet, where we consider reopening closed connections between isolated supply areas. The results show that the considered design-for-control strategy increases the resilience of BWPnet while minimizing pressure induced leakage. Moreover, the refinement stage of the proposed hybrid method efficiently improves the coarse approximation computed by the initial EA search, returning a continuous and even Pareto front approximation.

Loss landscapes and optimization in over-parameterized non-linear systems and neural networks

The success of deep learning is due, to a large extent, to the remarkable effectiveness of gradient-based optimization methods applied to large neural networks. The purpose of this work is to propose a modern view and a general mathematical framework for loss landscapes and efficient optimization in over-parameterized machine learning models and systems of non-linear equations, a setting that includes over-parameterized deep neural networks. Our starting observation is that optimization landscapes corresponding to such systems are generally not convex, even locally around a global minimum, a condition we call essential non-convexity . We argue that instead they satisfy PL ⁎ , a variant of the Polyak-Łojasiewicz condition [32] , [25] on most (but not all) of the parameter space, which guarantees both the existence of solutions and efficient optimization by (stochastic) gradient descent (SGD/GD). The PL ⁎ condition of these systems is closely related to the condition number of the tangent kernel associated to a non-linear system showing how a PL ⁎ -based non-linear theory parallels classical analyses of over-parameterized linear equations. We show that wide neural networks satisfy the PL ⁎ condition, which explains the (S)GD convergence to a global minimum. Finally we propose a relaxation of the PL ⁎ condition applicable to “almost” over-parameterized systems.

An Efficient Algorithm for Economic Load Dispatch Considering Valve Point Loading Effect

A simple and efficient algorithm is proposed for solving the economic dispatch problem of power system with valve point discontinuities employing a particle swarm optimization based approach. Evolutionary methods such as GA and PSO are known to perform better than conventional gradient based optimization methods for non convex optimization problems. This reactive power management in economic load dispatches plays a vital role in improving power quality of the system. The power compensation and economic load dispatches is major problem in distribution network. The power is maintain the state of the UPQC (Unified power quality conditioner). The UPAC controlled by the STATCOM or DSTATCOM. The performance of the proposed method has been compared with conventional PSO. The effectiveness of the algorithm has been tested on a test system having three generating units.

Crashworthiness analysis and shape design optimization of corrugated tubes for railway application

Thin-walled tubes have been widely used as energy-absorbing devices since they are light and have high energy absorption efficiency. However, the downside is that conventional thin-walled tubes usually exhibit an excessive peak crushing force (PCF) and large fluctuations in the force-displacement curve. Corrugated tubes are introduced to reduce the PCF and to increase the stability of energy-absorbing devices. Since the performance of corrugated tubes is highly influenced by their geometry, design optimization methods can be utilized to optimize the performance of corrugated tubes. It is difficult to apply conventional gradient-based optimization methods in crashworthiness analysis because of the strong nonlinearity and large computational time. In this paper, we utilize shape design optimization based on an adaptive surrogate model. The EA is maximized in transition crushing mode (T-mode), which is the boundary between progressive crushing mode (P-mode) and simultaneous crushing mode (S-mode), while simultaneously optimizing the contact force efficiency (CFE).

Efficient online motion planning method for the robotic arm to pick-up moving objects smoothly with temporal constraints

This paper proposes a new efficient online motion planning method for the manipulator to grasp moving objects smoothly. The algorithm framework consists of front-end pathfinding and back-end nonlinear trajectory optimization. The sample-based pathfinding algorithm can predict the intersection location and find a safe initial path in the dynamic environment. The gradient-based continuous-time trajectory optimization method converts the Euclidean Signed Distance Field information into the joint space, uses the B-spline curve to represent the joint trajectories, optimizes the initial path using convex hull characteristics of B-spline, and generates a smooth and kinematics feasible trajectory in joint space. Finally, we use an adjustment method of iterative trajectory fragment length to guarantee kinodynamic feasibility. Simulation comparisons and real-world experiments verify our planning framework’s high performance.

Simultaneous topology and fiber path optimization of composite structures with MAC constraints

The pursuit of stiffer lightweight structures has led to the development of new concepts such as the Variable Stiffness Composite Laminates (VSCL), which further enhance the design potential of these highly tailored materials. As the number of design variables rise, more robust topology optimization formulations are necessary to achieve optimal solutions. In this work, a design optimization framework is developed by assigning material and fiber orientation on discrete patches of the VSCL structures. A gradient-based optimization method is used in the framework by applying the Discrete Material Optimization (DMO) formulation. The main goal is to design structures to meet dynamic performance objectives, with particular emphasis on the synthesis of modes with a target design. Accordingly, the objective function depends on the eigenvalues (natural frequencies), while one set of constraints depends on the eigenvectors (mode shapes). The Modal Assurance Criterion (MAC) is simultaneously used to perform mode tracking between the different design iterations and to evaluate the correlation between the synthetized modes and the pre-defined objective shapes. In this study, the calculation of the weight derivatives is also revised for the DMO formulation to show the contribution of the missing terms in the final optimal results. The optimization framework is examined for some potential applications related to dynamic decoupling and dynamic synthesis. It is shown that the design approach tailors the dynamic response of the composite structure and it can effectively be used to design resonators based on a reference model.

“Deep reinforcement learning for engineering design through topology optimization of elementally discretized design domains”

Advances in machine learning algorithms and increased computational efficiencies give engineers new capabilities and tools to apply to engineering design. Machine learning models can approximate complex functions and, therefore, can be useful for various tasks in the engineering design workflow. This paper investigates using reinforcement learning (RL), a subset of machine learning that teaches an agent to complete a task through accumulating experiences in an interactive environment, to automate the designing of 2D discretized topologies. RL agents use past experiences to learn sequential sets of actions to best achieve some objective. In the proposed environment, an RL agent can make sequential decisions to design a topology by removing elements to best satisfy compliance minimization objectives. After each action, the agent receives feedback by evaluating how well the current topology satisfies the design objectives. After training, the agent was tasked with designing optimal topologies under various load cases. The agent's proposed designs had similar or better compliance minimization performance to those produced by traditional gradient-based topology optimization methods. These results show that a deep RL agent can learn generalized design strategies to satisfy multi-objective design tasks and, therefore, shows promise as a tool for arbitrarily complex design problems across many domains.