Second-order Optimization Methods Research Articles

EV charging load forecasting and optimal scheduling based on travel characteristics

Accurate prediction of EV charging load distribution is not only the basis of evaluating the capacity of distribution network to receive EV, but also the necessary premise to promote the research of charging station planning and vehicle-network interaction. Based on this, firstly, according to the travel characteristics of electric private car and taxi users, this paper constructs the models of speed-flow and EV power consumption per mileage, thus, the real-time changes of vehicle speed and power consumption per mileage are simulated. Secondly, consider factors such as charging price, time, and power consumption along the way, as well as multiple sources of information such as traffic networks, vehicles, public quick charging stations, and distribution networks, the OD matrix analysis method is introduced to simulate EV moving characteristics for EV trip assignment matrix, and the EV charging load forecasting system with real-time interaction of multi-source information is established. Finally, the second-order cone optimization method is used to optimize the space-time distribution of regional charging load. The model and method proposed in this paper are more consistent with the actual situation and can improve the accuracy of prediction.

A Second-Order Numerical Method for a Class of Optimal Control Problems

The numerical solution of optimal control problems through second-order methods is examined in this paper. Controlled processes are described by a system of nonlinear ordinary differential equations. There are two specific characteristics of the class of control actions used. The first one is that controls are searched for in a given class of functions, which depend on unknown parameters to be found by minimizing an objective functional. The parameter values, in general, may be different at different time intervals. The second feature of the considered problem is that the boundaries of time intervals are also optimized with fixed values of the parameters of the control actions in each of the intervals. The special cases of the problem under study are relay control problems with optimized switching moments. In this work, formulas for the gradient and the Hessian matrix of the objective functional with respect to the optimized parameters are obtained. For this, the technique of fast differentiation is used. A comparison of numerical experiment results obtained with the use of first- and second-order optimization methods is presented.

Physics-informed Neural Network for Force-free Magnetic Field Extrapolation

In this paper, we propose a physics-informed neural network extrapolation method that leverages machine learning techniques to reconstruct coronal magnetic fields. We enhance the classical neural network structure by introducing the concept of a quasi-output layer to address the challenge of preserving physical constraints during the neural network extrapolation process. Furthermore, we employ second-order optimization methods for training the neural network, which are more efficient compared to the first-order optimization methods commonly used in classical machine learning. Our approach is evaluated on the widely recognized semi-analytical model proposed by Low and Lou. The results demonstrate that the deep learning method achieves high accuracy in reconstructing the semi-analytical model across multiple evaluation metrics. In addition, we validate the effectiveness of our method on the observed magnetogram of active region.

Decentralized Over-the-Air Federated Learning by Second-Order Optimization Method

Decentralized Over-the-Air Federated Learning by Second-Order Optimization Method

Potential Energy Optimization Approach in the Form-Finding of Tensegrities and Cable–Strut Systems

A form-finding process for tensegrities and cable–strut systems is determined as a minimum problem of potential energy that is constructed considering initial estimated pre-stress states. Both the Jacobian and Hessian matrices of a potential energy function are derived explicitly regarding free nodal coordinates. Based on these derivations, both the first-order geodesic dynamic relaxation optimization method and the second-order optimization method of Newton can be applied to solve form-finding problems involving tensegrities and cable–strut structures. It should be noted that both the pre-stress state and the configuration of the systems are adjusted during the optimization process to satisfy self-equilibrium conditions. The geometrical constraints are then included in the proposed work as equality constraint conditions to find the final forms satisfying not only the self-equilibrium states but also the designers’ demands. Two well-known and complex systems are investigated to assess the efficiency of the proposed method, i.e. free-standing truncated icosahedral tensegrity and the Kiewitt Dome.

A Quasi-Newton Matrix Factorization-Based Model for Recommendation

Solving large-scale non-convex optimization problems is the fundamental challenge in the development of matrix factorization (MF)-based recommender systems. Unfortunately, employing conventional first-order optimization approaches proves to be an arduous endeavor since their curves are very complex. The exploration of second-order optimization methods holds great promise. They are more powerful because they consider the curvature of the optimization problem, which is captured by the second-order derivatives of the objective function. However, a significant obstacle arises when directly applying Hessian-based approaches: their computational demands are often prohibitively high. Therefore, the authors propose AdaGO, a novel quasi-Newton method-based optimizer to meet the specific requirements of large-scale non-convex optimization problems. AdaGO can strike a balance between computational efficiency and optimization performance. In the comparative studies with state-of-the-art MF-based models, AdaGO demonstrates its superiority by achieving higher prediction accuracy.

Joint Communication and Sensing Design for Multihop RIS-Aided Communication Systems in Underground Coal Mines

How to achieve reliable communication and safety monitoring is very important in coal mines. However, most of the existing transmission strategies and sensing-based monitoring approaches assume a single objective and neglect Non-Line-of-Sight (NLOS) problems brought by winding tunnels or mine collapses. To this end, we firstly propose a multi-hop reconfigurable intelligent surfaces (RISs) aided joint communication and sensing (JCAS) approach to maximize the energy efficiency of the JCAS access point and the sum sensing rates in order to improve the sensing accuracy. Specifically, we formulate an energy-efficient optimization problem by jointly designing both the phase-shift matrix and the switches status of the RISs as well as the transmit power of the access point. The problem is solved by adopting the successive convex approximation-based alternating optimization algorithm, the second-order optimization method, the Lambert-w function, and the Newton’s method. Moreover, a sensing-based rate optimization problem is also solved via the Lagrange relaxation method and the Gradient descent method. Simulation results demonstrate that the proposed algorithm has better robustness and higher energy efficiency.

Research on acoustic fault diagnosis of bearings based on spatial filtering and time-frequency domain filtering

In conventional methods, vibration-based diagnosis (VBD) is commonly used for identifying rolling bearing faults. In contrast, acoustical-based fault diagnosis (ABD) has been extensively studied as an alternative due to its non-contact measurement capabilities, which are superior to VBD. Nonetheless, the accuracy of machine diagnosis may be compromised by multisource aliasing in the recorded acoustic signals. This research integrates spatial domain filtering and time-frequency domain filtering to address the issue of bearing fault diagnosis. By utilizing the second-order cone optimization method, differential power response beamformers have been established to attain a high signal-to-noise ratio of bearing sound while effectively controlling the noise power level. The output of differential beamformers is analyzed using the parametric adaptive MOMEDA approach. The method presented in this paper enables bearing acoustic fault diagnosis and identification of the faulty bearing location. The validity of the proposed ABD method has been demonstrated through experimental and simulation results.

Scalable K-FAC Training for Deep Neural Networks With Distributed Preconditioning

The second-order optimization methods, notably the D-KFAC (Distributed Kronecker Factored Approximate Curvature) algorithms, have gained traction on accelerating deep neural network (DNN) training on GPU clusters. However, existing D-KFAC algorithms require to compute and communicate a large volume of second-order information, i.e., Kronecker factors (KFs), before preconditioning gradients, resulting in large computation and communication overheads as well as a high memory footprint. In this paper, we propose DP-KFAC, a novel distributed preconditioning scheme that distributes the KF constructing tasks at different DNN layers to different workers. DP-KFAC not only retains the convergence property of the existing D-KFAC algorithms but also enables three benefits: reduced computation overhead in constructing KFs, no communication of KFs, and low memory footprint. Extensive experiments on a 64-GPU cluster show that DP-KFAC reduces the computation overhead by 1.55×-1.65×, the communication cost by 2.79×-3.15×, and the memory footprint by 1.14×-1.47× in each second-order update compared to the state-of-the-art D-KFAC methods. Our codes are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/lzhangbv/kfac\_pytorch</uri> .

Compressive Gate Set Tomography

Flexible characterization techniques that provide a detailed picture of the experimental imperfections under realistic assumptions are crucial to gain actionable advice in the development of quantum computers. Gate set tomography self-consistently extracts a complete tomographic description of the implementation of an entire set of quantum gates, as well as the initial state and measurement, from experimental data. It has become a standard tool for this task but comes with high requirements on the number of sequences and their design, making it already experimentally challenging for only two qubits. In this work, we show that low-rank approximations of gate sets can be obtained from significantly fewer gate sequences and that it is sufficient to draw them at random. This coherent noise characterization however still contains the crucial information for improving the implementation. To this end, we formulate the data processing problem of gate set tomography as a rank-constrained tensor completion problem. We provide an algorithm to solve this problem while respecting the usual positivity and normalization constraints of quantum mechanics. For this purpose, we combine methods from Riemannian optimization and machine learning and develop a saddle-free second-order geometrical optimization method on the complex Stiefel manifold. Besides the reduction in sequences, we numerically demonstrate that the algorithm does not rely on structured gate sets or an elaborate circuit design to robustly perform gate set tomography. Therefore, it is more flexible than traditional approaches. We also demonstrate how coherent errors in shadow estimation protocols can be mitigated using estimates from gate set tomography.5 MoreReceived 25 May 2022Revised 10 January 2023Accepted 25 January 2023DOI:https://doi.org/10.1103/PRXQuantum.4.010325Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.Published by the American Physical SocietyPhysics Subject Headings (PhySH)Research AreasQuantum channelsQuantum tomographyQuantum verificationQuantum Information

Eigenvalue-Corrected Natural Gradient Based on a New Approximation

Using second-order optimization methods for training deep neural networks (DNNs) has attracted many researchers. A recently proposed method, Eigenvalue-corrected Kronecker Factorization (EKFAC), proposed an interpretation by viewing natural gradient update as a diagonal method and corrects the inaccurate re-scaling factor in the KFAC eigenbasis. What’s more, a new method to approximate the natural gradient called Trace-restricted Kronecker-factored Approximate Curvature (TKFAC) is also proposed, in which the Fisher information matrix (FIM) is approximated as a constant multiplied by the Kronecker product of two matrices and the traces can be kept equal before and after the approximation. In this work, we combine the ideas of these two methods and propose Trace-restricted Eigenvalue-corrected Kronecker Factorization (TEKFAC). The proposed method not only corrects the inexact re-scaling factor under the Kronecker-factored eigenbasis, but also considers the new approximation method and the effective damping technique adopted by TKFAC. We also discuss the differences and relationships among the related Kronecker-factored approximations. Empirically, our method outperforms SGD with momentum, Adam, EKFAC and TKFAC on several DNNs.

LSOS: Line-search second-order stochastic optimization methods for nonconvex finite sums

We develop a line-search second-order algorithmic framework for minimizing finite sums. We do not make any convexity assumptions, but require the terms of the sum to be continuously differentiable and have Lipschitz-continuous gradients. The methods fitting into this framework combine line searches and suitably decaying step lengths. A key issue is a two-step sampling at each iteration, which allows us to control the error present in the line-search procedure. Stationarity of limit points is proved in the almost-sure sense, while almost-sure convergence of the sequence of approximations to the solution holds with the additional hypothesis that the functions are strongly convex. Numerical experiments, including comparisons with state-of-the art stochastic optimization methods, show the efficiency of our approach.

Establishment of a Fuzzy Comprehensive Evaluation Random Matrix Model for the Assessment of Foreign Language Translation Level in Universities

In this paper, the random matrix method of a fuzzy comprehensive evaluation is used to conduct in-depth research and analysis on the assessment of foreign language translation levels in universities, and the corresponding model is designed for the automatic assessment of translation levels in university education. The Hessian matrix of the second-order optimization method is used to analyze the geometric properties of the loss plane of the deep neural network, and it is proved that the Hessian matrix can be constructed as a sample covariance matrix, which is the classical Wishart matrix in the random matrix theory. Using the study on the asymptotic distribution properties of the Wishart matrix in random matrix theory, the Hessian matrix is analyzed and the limiting spectral distribution, the eigenvalue extreme value distribution, and the standard conditional number distribution of the matrix are given. And the scores are basically distributed above 240, but some indicators have relatively low scores, all below 120 points, so the indicators distributed above 240 points are selected as reserved indicators. The analysis considers that the evaluation of foreign language translation in learning in colleges and universities is itself very fuzzy and constructs an evaluation model based on the fuzzy comprehensive judgment method, which provides a research strategy for the future evaluation of foreign language translation in learning in colleges and universities. Finally, the evaluation system of foreign language translation in learning in colleges and universities is designed and implemented. It is feasible to apply the fuzzy comprehensive judgment to the comprehensive evaluation of college students, the combined judgment of multiple methods has higher reliability and stability, and its research conclusions have certain reference value in the specific education management practice. The value, objectivity, feasibility, and wide adaptability of the fuzzy comprehensive evaluation method are analyzed, and some references are expected to be provided for translation quality assessment research and translation practice.

Application of Second-Order Optimization Methods for Solving an Inverse Coefficient Problem in the Three-Dimensional Statement

Application of Second-Order Optimization Methods for Solving an Inverse Coefficient Problem in the Three-Dimensional Statement

Sketched Newton--Raphson

We propose a new globally convergent stochastic second-order method. Our starting point is the development of a new sketched Newton--Raphson (SNR) method for solving large scale nonlinear equations of the form $F(x)=0$ with $F:\mathbb{R}^p \rightarrow \mathbb{R}^m$. We then show how to design several stochastic second-order optimization methods by rewriting the optimization problem of interest as a system of nonlinear equations and applying SNR. For instance, by applying SNR to find a stationary point of a generalized linear model, we derive completely new and scalable stochastic second-order methods. We show that the resulting method is very competitive as compared to state-of-the-art variance reduced methods. Furthermore, using a variable splitting trick, we also show that the stochastic Newton method (SNM) is a special case of SNR and use this connection to establish the first global convergence theory of SNM. We establish the global convergence of SNR by showing that it is a variant of the online stochastic gradient descent (SGD) method, and then leveraging proof techniques of \textttSGD. As a special case, our theory also provides a new global convergence theory for the original Newton--Raphson method under strictly weaker assumptions as compared to the classic monotone convergence theory.

A combined first- and second-order optimization method for improving convergence of Hartree–Fock and Kohn–Sham calculations

In this work, we investigate the optimization of Hartree-Fock (HF) orbitals with our recently proposed combined first- and second-order (SO-SCI) method, which was originally developed for multi-configuration self-consistent field (MCSCF) and complete active space SCF (CASSCF) calculations. In MCSCF/CASSCF, it unites a second-order optimization of the active orbitals with a Fock-based first-order treatment of the remaining closed-virtual orbital rotations. In the case of the single-determinant wavefunctions, the active space is replaced by a preselected "second-order domain," and all rotations involving orbitals in this subspace are treated at second-order. The method has been implemented for spin-restricted and spin-unrestricted Hartree-Fock (RHF, UHF), configuration-averaged Hartree-Fock (CAHF), as well as Kohn-Sham (KS) density functional theory (RKS, UKS). For each of these cases, various choices of the second-order domain have been tested, and appropriate defaults are proposed. The performance of the method is demonstrated for several transition metal complexes. It is shown that the SO-SCI optimization provides faster and more robust convergence than the standard SCF procedure but requires, in many cases, even less computation time. In difficult cases, the SO-SCI method not only speeds up convergence but also avoids convergence to saddle-points. Furthermore, it helps to find spin-symmetry broken solutions in the cases of UHF or UKS. In the case of CAHF, convergence can also be significantly improved as compared to a previous SCF implementation. This is particularly important for multi-center cases with two or more equal heavy atoms. The performance is demonstrated for various two-center complexes with different lanthanide atoms.

Simulation of the 3D hyperelastic behavior of ventricular myocardium using a finite-element based neural-network approach

High-fidelity cardiac models using attribute-rich finite element based models have been developed to a very mature stage. However, such finite-element based approaches remain time consuming, which have limited their clinical use. There remains a need for alternative methods for novel cardiac simulation methods capable of high fidelity simulations in clinically relevant time frames. Surrogate models are one approach, which traditionally use a data-driven approach for training, requiring the generation of a sufficiently large number of simulation results as the training dataset. Alternatively, a physics-informed neural network can be trained by minimizing the PDE residuals or energy potentials. However, this approach does not provide a general method to easily using existing finite element models. To address these challenges, we developed a hybrid approach that seamlessly bridged a neural network surrogate model with a differentiable finite element domain representation (NNFE). Given its importance in cardiac simulations, we applied this approach to simulations of the hyperelastic mechanical behavior of ventricular myocardium from recent 3D kinematic constitutive model (J Mech Behav Biomed Mater, 2020 doi: 10.1016/j.jmbbm.2019.103508). We utilized a cuboidal domain and conducted numerical studies of individual myocardium specimens discretized by a finite element mesh and assigned with experimentally obtained myofiber architectures. Both parameterized Dirichlet and Neumann boundary conditions were studied. We developed a second-order Newton optimization method, instead of using a stochastic gradient descent method, to train the neural network efficiently. The resulting trained neural network surrogate model demonstrated excellent agreement with the corresponding “ground truth” finite element solutions over the entire physiological deformation range. More importantly, the NNFE approach provided a significantly decreased computational time for a range of finite element mesh sizes for online predictions. For example, as the finite element mesh size increased from 2744 to 175615 elements, the NNFE computational time increased from 0.1108 s to 0.1393 s, while the “ground truth” FE model increased from 4.541 s to 719.9 s. These results suggest that NNFE run times can be significantly reduced compared with the traditional large-deformation based finite element solution methods. The trade-off is to train the NNFE off-line within a range of anticipated physiological responses. However, training time would only have to be performed once before any number of application uses. Moreover, since the NNFE is an analytical function its computational performance will be amplified when the corresponding problem becomes more complex.

A Trace-restricted Kronecker-Factored Approximation to Natural Gradient

Second-order optimization methods have the ability to accelerate convergence by modifying the gradient through the curvature matrix. There have been many attempts to use second-order optimization methods for training deep neural networks. In this work, inspired by diagonal approximations and factored approximations such as Kronecker-factored Approximate Curvature (KFAC), we propose a new approximation to the Fisher information matrix (FIM) called Trace-restricted Kronecker-factored Approximate Curvature (TKFAC), which can hold the certain trace relationship between the exact and the approximate FIM. In TKFAC, we decompose each block of the approximate FIM as a Kronecker product of two smaller matrices and scaled by a coefficient related to trace. We theoretically analyze TKFAC's approximation error and give an upper bound of it. We also propose a new damping technique for TKFAC on convolutional neural networks to maintain the superiority of second-order optimization methods during training. Experiments show that our method has better performance compared with several state-of-the-art algorithms on some deep network architectures.

Second-Order Orbital Optimization with Large Active Spaces Using Adaptive Sampling Configuration Interaction (ASCI) and Its Application to Molecular Geometry Optimization.

Recently, selected configuration interaction (SCI) methods that enable calculations with several tens of active orbitals have been developed. With the SCI subspace embedded in the mean field, molecular orbitals with an accuracy comparable to that of the complete active space self-consistent field method can be obtained. Here, we implement the analytical gradient theory for the single-state adaptive sampling CI (ASCI) SCF method to enable molecular geometry optimization. The resulting analytical gradient is inherently approximate due to the dependence on the sampled determinants, but its accuracy was sufficient for performing geometry optimizations with large active spaces. To obtain the tight convergence needed for accurate analytical gradients, we combine the augmented Hessian (AH) and Werner-Meyer-Knowles (WMK) second-order orbital optimization methods with the ASCI-SCF method. We test these algorithms for orbital and geometry optimizations, demonstrate applications of the geometry optimizations of polyacenes and periacenes, and discuss the geometric dependence of the characteristics of singlet ASCI wave functions.

Low-Rank Riemannian Optimization for Graph-Based Clustering Applications.

With the abundance of data, machine learning applications engaged increased attention in the last decade. An attractive approach to robustify the statistical analysis is to preprocess the data through clustering. This paper develops a low-complexity Riemannian optimization framework for solving optimization problems on the set of positive semidefinite stochastic matrices. The low-complexity feature of the proposed algorithms stems from the factorization of the optimization variable X=YYT and deriving conditions on the number of columns of Y under which the factorization yields a satisfactory solution. The paper further investigates the embedded and quotient geometries of the resulting Riemannian manifolds. In particular, the paper explicitly derives the tangent space, Riemannian gradients and Hessians, and a retraction operator allowing the design of efficient first and second-order optimization methods for the graph-based clustering applications of interest. The numerical results reveal that the resulting algorithms present a clear complexity advantage as compared with state-of-the-art euclidean and Riemannian approaches for graph clustering applications.