Second-order Optimality Research Articles

Explicit-by-Implicit Treatment of Natural Orbital Occupations Using First- and Second-Order Optimization Algorithms: A Comparative Study.

To address the convergence issues in the natural occupation optimization of reduced density matrix functional theory (RDMFT), we recently proposed the explicit-by-implicit (EBI) idea to handle the ensemble N-representability constraint (Yao et al. J. Phys. Chem. Lett. 2021, 12, 6788). This work continues to focus on these issues that can affect the reliability of the electronic structure description in RDMFT; further explores the combination of EBI, as well as the (augmented) Lagrangian methods (both LM and ALM), with both first- and second-order numerical optimization algorithms; and carefully evaluates their performances in natural occupation optimizations of various systems, including strongly correlated systems and large molecules. By comparing both converged energies and elapsed times, it can be seen that the LM and ALM have serious convergence issues for systems of different sizes. In contrast, the optimizations of EBI can converge to better energies with fewer iterations. However, due to the local convergence nature of the Newton's Method (NM) algorithm, EBI@NM still suffers from the local minimum issue for both strongly correlated systems and large molecules. Overall, the combination of EBI with the simple first-order algorithm of gradient descent (GD), namely EBI@GD, consistently provides the lowest converged energies for different types of systems, with the lowest computational scaling. These tests demonstrate the advantages of EBI in the calculations of transition states, strongly correlated systems, and large molecules. Meanwhile, the insights gained from this work are helpful to further develop more efficient algorithms for RDMFT.

Reflection-based traveltime and waveform inversion with second-order optimization

Reflection-based inversion that aims to reconstruct the low-to-intermediate wavenumbers of the subsurface model, can be a complementary to refraction-data-driven full-waveform inversion (FWI), especially for the deep target area where diving waves cannot be acquired at the surface. Nevertheless, as a typical nonlinear inverse problem, reflection waveform inversion may easily suffer from the cycle-skipping issue and have a slow convergence rate, if gradient-based first-order optimization methods are used. To improve the accuracy and convergence rate, we introduce the Hessian operator into reflection traveltime inversion (RTI) and reflection waveform inversion (RWI) in the framework of second-order optimization. A practical two-stage workflow is proposed to build the velocity model, in which Gauss-Newton RTI is first applied to mitigate the cycle-skipping problem and then Gauss-Newton RWI is employed to enhance the model resolution. To make the Gauss-Newton iterations more efficiently and robustly for large-scale applications, we introduce proper preconditioning for the Hessian matrix and design appropriate strategies to reduce the computational costs. The example of a real dataset from East China Sea demonstrates that the cascaded Hessian-based RTI and RWI have good potential to improve velocity model building and seismic imaging, especially for the deep targets.

Application of Second-Order Optimization Methods for Solving an Inverse Coefficient Problem in the Three-Dimensional Statement

Application of Second-Order Optimization Methods for Solving an Inverse Coefficient Problem in the Three-Dimensional Statement

Sketched Newton--Raphson

We propose a new globally convergent stochastic second-order method. Our starting point is the development of a new sketched Newton--Raphson (SNR) method for solving large scale nonlinear equations of the form $F(x)=0$ with $F:\mathbb{R}^p \rightarrow \mathbb{R}^m$. We then show how to design several stochastic second-order optimization methods by rewriting the optimization problem of interest as a system of nonlinear equations and applying SNR. For instance, by applying SNR to find a stationary point of a generalized linear model, we derive completely new and scalable stochastic second-order methods. We show that the resulting method is very competitive as compared to state-of-the-art variance reduced methods. Furthermore, using a variable splitting trick, we also show that the stochastic Newton method (SNM) is a special case of SNR and use this connection to establish the first global convergence theory of SNM. We establish the global convergence of SNR by showing that it is a variant of the online stochastic gradient descent (SGD) method, and then leveraging proof techniques of \textttSGD. As a special case, our theory also provides a new global convergence theory for the original Newton--Raphson method under strictly weaker assumptions as compared to the classic monotone convergence theory.

Compatible intrinsic triangulations

Finding distortion-minimizing homeomorphisms between surfaces of arbitrary genus is a fundamental task in computer graphics and geometry processing. We propose a simple method utilizing intrinsic triangulations, operating directly on the original surfaces without going through any intermediate domains such as a plane or a sphere. Given two models A and B as triangle meshes, our algorithm constructs a Compatible Intrinsic Triangulation (CIT), a pair of intrinsic triangulations over A and B with full correspondences in their vertices, edges and faces. Such a tessellation allows us to establish consistent images of edges and faces of A's input mesh over B (and vice versa) by tracing piecewise-geodesic paths over A and B. Our algorithm for constructing CITs, primarily consisting of carefully designed edge flipping schemes, is empirical in nature without any guarantee of success, but turns out to be robust enough to be used within a similar second-order optimization framework as was used previously in the literature. The utility of our method is demonstrated through comparisons and evaluation on a standard benchmark dataset.

On the second-order optimality conditions for multi-objective optimal control problems with mixed pointwise constraints

On the second-order optimality conditions for multi-objective optimal control problems with mixed pointwise constraints

A combined first- and second-order optimization method for improving convergence of Hartree–Fock and Kohn–Sham calculations

In this work, we investigate the optimization of Hartree-Fock (HF) orbitals with our recently proposed combined first- and second-order (SO-SCI) method, which was originally developed for multi-configuration self-consistent field (MCSCF) and complete active space SCF (CASSCF) calculations. In MCSCF/CASSCF, it unites a second-order optimization of the active orbitals with a Fock-based first-order treatment of the remaining closed-virtual orbital rotations. In the case of the single-determinant wavefunctions, the active space is replaced by a preselected "second-order domain," and all rotations involving orbitals in this subspace are treated at second-order. The method has been implemented for spin-restricted and spin-unrestricted Hartree-Fock (RHF, UHF), configuration-averaged Hartree-Fock (CAHF), as well as Kohn-Sham (KS) density functional theory (RKS, UKS). For each of these cases, various choices of the second-order domain have been tested, and appropriate defaults are proposed. The performance of the method is demonstrated for several transition metal complexes. It is shown that the SO-SCI optimization provides faster and more robust convergence than the standard SCF procedure but requires, in many cases, even less computation time. In difficult cases, the SO-SCI method not only speeds up convergence but also avoids convergence to saddle-points. Furthermore, it helps to find spin-symmetry broken solutions in the cases of UHF or UKS. In the case of CAHF, convergence can also be significantly improved as compared to a previous SCF implementation. This is particularly important for multi-center cases with two or more equal heavy atoms. The performance is demonstrated for various two-center complexes with different lanthanide atoms.

Second-order conditions for non-uniformly convex integrands: quadratic growth in $L^1$

We study no-gap second-order optimality conditions for a non-uniformly convex and non-smooth integral functional. The integral functional is extended to the space of measures. The obtained second-order derivatives contain integrals on lower-dimensional manifolds. The proofs utilize the convex pre-conjugate, which is an integral functional on the space of continuous functions. Applications to non-smooth optimal control problems are given.

Soft-sensing of effluent total phosphorus using adaptive recurrent fuzzy neural network with Gustafson-Kessel clustering

To address the issue of soft-sensing of effluent total phosphorus in wastewater treatment processes (WWTPs), a soft-sensing system based on an adaptive recursive fuzzy neural network with Gustafson-Kessel (GK) clustering and hierarchical adaptive second-order optimization algorithm (HAS) is proposed in this paper. In GK-ARFNN, first, the GK clustering algorithm was utilized to cluster the input–output dataset. Thus, the establishment of the initial fuzzy rule base and the determination of the parameter value of the fuzzy set membership function was realized. Then, the recursive layer was added into FNN to improve the dynamic mapping ability of the system. Finally, the HAS algorithm was developed based on the improved Levenberg-Marquardt (LM) optimization algorithm, and all the free parameters of the GK-ARFNN were adjusted online using HAS to improve the generalization capability and prediction accuracy of the soft-sensing system. In addition, the convergence of the proposed GK-ARFNN algorithm was also analyzed in this paper, which can ensure the effectiveness of the solutions to modelling issues for practical industrial processes. The simulation results demonstrate that the GK-ARFNN-based soft-sensing system introduced in this paper achieved satisfactory accuracy in the prediction of effluent total phosphorus in WWTPs. The source codes of GK-ARFNN and some competitors can be downloaded from https://github.com/hyitzhb/GK-ARFNN.

On the generalized trace ratio problem

The generalized trace ratio problem (GTRP) is to maximize the quotient over the Stiefel manifold. We establish the first-order necessary and sufficient optimality conditions for the global maximizer. Based on the classical second-order necessary optimality condition, we prove that GTRP has no local non-global maximizer. Finally we reveal the hidden convexity property of GTRP by applying the convex hull representation of the orthogonal similarity set.

Second-order conic programming for data envelopment analysis models

Data envelopment analysis (DEA) is a widely used benchmarking technique. Its strength stems from the fact that it can include several inputs and outputs of not necessarily the same type to evaluate efficiency scores. Indeed, the aforesaid method is based on mathematical optimization. This paper constructs a second-order conic optimization problem unifying several DEA models. Moreover, it presents an algorithm that solves the former problem, and provides a MATLAB function associated with it. As far as known, no MATLAB function solves DEA models. Among different types of DEA, this function can handle deterministic, Malmquist index, and stochastic models. In fact, DEA is involved in various practical applications, thus, this work will provide some possible future extensions, not only for MATLAB but also for any programming software in applications of decision science and efficiency analysis.

Convergence to a second-order critical point by a primal-dual interior point trust-region method for nonlinear semidefinite programming

In this paper, we propose a primal-dual interior point trust-region method for solving nonlinear semidefinite programming problems, in which the iterates converge to a point that satisfies the first-order and second-order optimality conditions. The method consists of the outer iteration (SDPIP-revised) that finds a Karush-Kuhn-Tucker (KKT) point which satisfies the second-order optimality condition, and the inner iteration (SDPTR-revised) that calculates an approximate barrier KKT point. Algorithm SDPTR-revised uses a commutative class of Newton-like directions within the framework of the trust-region method in the primal-dual space. In addition, we also use a direction of negative curvature when it exists. The proposed algorithm employs a new method that generates negative-curvature directions in the existence of -type penalty term for equality constraints. It is proved that there exists a limit point of the generated sequence which satisfies the second-order optimality condition along with the barrier KKT conditions.

AC Network-Constrained Unit Commitment via Relaxation and Decomposition

We propose an approach to solving the AC network-constrained unit commitment problem (AC-NCUC) of large-scale power systems. This approach relies on solving convex (continuous and mixed-integer) optimization problems. First, we formulate a second-order conic relaxation of the original optimization problem, which renders a mixed-integer second-order conic optimization problem. To solve this problem, we use Benders’ decomposition, which decomposes it into a master problem, which is mixed-integer linear, and a subproblem, which is continuous and second-order conic. To improve the convergence of the Benders’ algorithm, we add several linear feasibility constraints to the master problem. Since the second-order relaxation of the original problem cannot guarantee AC feasibility of the solution found, we ensure such feasibility by solving a sequence of continuous convex optimization problems. We numerically validate the proposed approach using a Texas 2000-bus system.

First-order and second-order necessary optimality conditions for discrete-time stochastic systems with delay

Abstract This paper is concerned with the first-order and second-order necessary optimality conditions for discrete-time stochastic systems with delay under weak assumptions. Based on a characteristic method and a discrete-time backward stochastic equation, we establish the linearized discrete-time stochastic maximum principle and Euler-type necessary optimality condition. Moreover, by a new discrete-time backward stochastic matrix equation, we also obtain the necessary optimality condition of quasi-singular control and the pointwise second-order optimality condition.

Application of Pontryagin's maximum principle to quantum metrology in dissipative systems

Optimal control theory, also known as Pontryagin's Maximum Principle, is applied to the quantum parameter estimation in the presence of decoherence. An efficient procedure is devised to compute the gradient of quantum Fisher information with respect to the control parameters and is used to construct the optimal control protocol. The proposed procedure keeps the control problem in the time-invariant form so that both first-order and second-order optimality conditions derived from Pontryagin's Maximum Principle apply; the second-order condition turns out to be crucial when the optimal control contains singular arcs. Concretely we look for the optimal control that maximizes quantum Fisher information for "twist and turn" problem. We find that the optimal control is singular without dissipation but can become unbounded once the quantum decoherence is introduced. An amplitude constraint is needed to guarantee a bounded solution. With quantum decoherence, the maximum quantum Fisher information happens at a finite time due to the decoherence, and the asymptotic value depends on the specific decoherence channel and the control of consideration.

Acoustic-Electrical Joint Localization Method of Partial Discharge in Power Transformer Considering Multi-Path Propagation Impact

Efficient and accurate localization of partial discharge (PD) is of paramount importance to ensure the safe operation of power transformers. However, the multi-path propagation effect introduced by the reflection, refraction and diffraction of the ultrasonic signal may add significant computational complexity to the localization process and degrade the localization accuracy. This paper proposes an acoustic- electrical joint method for partial discharge location in the power transformer with the full consideration of the multi-path propagation impact. Unlike the conventional error analysis methods, a partial discharge localization model is proposed for characterizing the multipath propagation impact without the prior knowledge of the transcendental error probability. Based on the matrix inequality transformation and relaxation, the high-dimensional nonlinear localization equations are transformed into a set of second-order convex optimization equations that can be solved using the convex second-order cone program (SOCP). The proposed solution can significantly reduce the computational complexity and improve the localization accuracy as well as avoid the local optimum and slow convergence. The solution is assessed through extensive experiments based on simulations, testbed and trial deployment in comparison with the existing solutions with the localization error of about 0.1 m.

A model-free method for identifying time-delay characteristics of HVAC system based on multivariate transfer entropy

The time delays in the HVAC system are common, which seriously affect time series prediction and pre-control command implementation, so identifying them is promising research. The existing methods of identifying time-delays mainly focus on establishing physical or dynamical models first and then conducting control variable experiments, which don't facilitate engineering. Therefore, we develop a model-free identification method rooted in an information-theoretic framework by introducing transfer entropy (TE) into the HVAC field. The suggested multivariate TE method can pick out each time-delay characteristic by mining monitoring data. Compared with correlation coefficients, it can filter redundant information between variables and discern the nonlinearity. Among them, to estimate multivariate TE well, a kernel estimator is improved. It performs the precise detection ability, low computational burden and strong robustness, compared with the original and k-nearest-neighbor (KNN) estimator. Besides, for an unacquainted HVAC system, a hierarchical optimization algorithm combining the Nash-optimization algorithm with a second-order oscillatory particle swarm optimization (SOPSO) algorithm is proposed to identify its time delays, where the accuracy and time cost are improved. Lastly, the above-mentioned methods are validated with simulated and real time series. This work is enlightening and has a further reference to identifying time delays in HVAC systems.

Simulation of the 3D hyperelastic behavior of ventricular myocardium using a finite-element based neural-network approach

High-fidelity cardiac models using attribute-rich finite element based models have been developed to a very mature stage. However, such finite-element based approaches remain time consuming, which have limited their clinical use. There remains a need for alternative methods for novel cardiac simulation methods capable of high fidelity simulations in clinically relevant time frames. Surrogate models are one approach, which traditionally use a data-driven approach for training, requiring the generation of a sufficiently large number of simulation results as the training dataset. Alternatively, a physics-informed neural network can be trained by minimizing the PDE residuals or energy potentials. However, this approach does not provide a general method to easily using existing finite element models. To address these challenges, we developed a hybrid approach that seamlessly bridged a neural network surrogate model with a differentiable finite element domain representation (NNFE). Given its importance in cardiac simulations, we applied this approach to simulations of the hyperelastic mechanical behavior of ventricular myocardium from recent 3D kinematic constitutive model (J Mech Behav Biomed Mater, 2020 doi: 10.1016/j.jmbbm.2019.103508). We utilized a cuboidal domain and conducted numerical studies of individual myocardium specimens discretized by a finite element mesh and assigned with experimentally obtained myofiber architectures. Both parameterized Dirichlet and Neumann boundary conditions were studied. We developed a second-order Newton optimization method, instead of using a stochastic gradient descent method, to train the neural network efficiently. The resulting trained neural network surrogate model demonstrated excellent agreement with the corresponding “ground truth” finite element solutions over the entire physiological deformation range. More importantly, the NNFE approach provided a significantly decreased computational time for a range of finite element mesh sizes for online predictions. For example, as the finite element mesh size increased from 2744 to 175615 elements, the NNFE computational time increased from 0.1108 s to 0.1393 s, while the “ground truth” FE model increased from 4.541 s to 719.9 s. These results suggest that NNFE run times can be significantly reduced compared with the traditional large-deformation based finite element solution methods. The trade-off is to train the NNFE off-line within a range of anticipated physiological responses. However, training time would only have to be performed once before any number of application uses. Moreover, since the NNFE is an analytical function its computational performance will be amplified when the corresponding problem becomes more complex.

Influence of electric vehicle distributed energy storage access on safety and stability of distribution network

This paper proposes a distributed energy storage control strategy for electric vehicles to improve the security and stability of distribution network when electric vehicles are connected. Firstly, the load characteristics of electric vehicles are investigated, and the optimal power flow model including energy storage power station, electric vehicle charging station considering V2G and distribution network is established. The objective function is established to minimize network loss, voltage deviation and load peak to peak. The problem is transformed into a mixed integer second-order cone optimization problem for solution, and based on the analysis of distributed energy storage model and constraints, the distributed energy storage control strategy of electric vehicle is established. Through the solution analysis in IEEE33 node system, it can be concluded that using this strategy can improve the power quality of distribution network and effectively improve the security and stability of distribution network.

First- and Second-Order Optimality Conditions for Quadratically Constrained Quadratic Programming Problems

We consider a quadratic programming problem with quadratic cone constraints and an additional geometric constraint. Under suitable assumptions, we establish necessary and sufficient conditions for optimality of a KKT point and, in particular, we characterize optimality by using strong duality as a regularity condition. We consider in details the case where the feasible set is defined by two quadratic equality constraints and, finally, we analyse simultaneous diagonalizable quadratic problems, where the Hessian matrices of the involved quadratic functions are all diagonalizable by means of the same orthonormal matrix.