Second-order Optimality Research Articles

Research on acoustic fault diagnosis of bearings based on spatial filtering and time-frequency domain filtering

In conventional methods, vibration-based diagnosis (VBD) is commonly used for identifying rolling bearing faults. In contrast, acoustical-based fault diagnosis (ABD) has been extensively studied as an alternative due to its non-contact measurement capabilities, which are superior to VBD. Nonetheless, the accuracy of machine diagnosis may be compromised by multisource aliasing in the recorded acoustic signals. This research integrates spatial domain filtering and time-frequency domain filtering to address the issue of bearing fault diagnosis. By utilizing the second-order cone optimization method, differential power response beamformers have been established to attain a high signal-to-noise ratio of bearing sound while effectively controlling the noise power level. The output of differential beamformers is analyzed using the parametric adaptive MOMEDA approach. The method presented in this paper enables bearing acoustic fault diagnosis and identification of the faulty bearing location. The validity of the proposed ABD method has been demonstrated through experimental and simulation results.

MAML-SR: Self-adaptive super-resolution networks via multi-scale optimized attention-aware meta-learning

The deep-learning-based super-resolution (SR) methods require an avalanche of training images. However, they do not adapt model parameters in test time to cope with the novel blur kernel scenarios. Even though the recent meta-learning-based SR techniques adapt trained model parameters leveraging a test image’s internal patch recurrence, they need heavy pre-training on an external dataset to initialize. Also, the shallow internal information exploration and failure to amplify the salient edges impacts blurry SR image generation. Besides, model inferencing gets delayed due to a threshold-dependent adaptation phase. In this paper, we present Multi-scale Optimized Attention-aware Meta-Learning framework for SR (MAML-SR) to explore the multi-scale hierarchical self-similarity of recurring patches in a test image. Precisely, without any pre-training, we directly meta-train our model with a second-order optimization having the first-order adapted parameters from the intermediate scales, which are again directly supervised with the ground-truth HR images. At each scale, non-local self-similarity is maximized along with the amplification of salient edges using a novel cross-scale spectro-spatial attention learning unit. Also, we drastically reduce the inference delay by putting a metric-dependent constraint on the gradient updates for a test image. We demonstrate our method’s superior super-resolving capability over four benchmark SR datasets.

Scalable K-FAC Training for Deep Neural Networks With Distributed Preconditioning

The second-order optimization methods, notably the D-KFAC (Distributed Kronecker Factored Approximate Curvature) algorithms, have gained traction on accelerating deep neural network (DNN) training on GPU clusters. However, existing D-KFAC algorithms require to compute and communicate a large volume of second-order information, i.e., Kronecker factors (KFs), before preconditioning gradients, resulting in large computation and communication overheads as well as a high memory footprint. In this paper, we propose DP-KFAC, a novel distributed preconditioning scheme that distributes the KF constructing tasks at different DNN layers to different workers. DP-KFAC not only retains the convergence property of the existing D-KFAC algorithms but also enables three benefits: reduced computation overhead in constructing KFs, no communication of KFs, and low memory footprint. Extensive experiments on a 64-GPU cluster show that DP-KFAC reduces the computation overhead by 1.55×-1.65×, the communication cost by 2.79×-3.15×, and the memory footprint by 1.14×-1.47× in each second-order update compared to the state-of-the-art D-KFAC methods. Our codes are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/lzhangbv/kfac\_pytorch</uri> .

A Scalable Method to Exploit Screening in Gaussian Process Models with Noise

A common approach to approximating Gaussian log-likelihoods at scale exploits the fact that precision matrices can be well-approximated by sparse matrices in some circumstances. This strategy is motivated by the screening effect, which refers to the phenomenon in which the linear prediction of a process Z at a point x 0 depends primarily on measurements nearest to x 0 . But simple perturbations, such as iid measurement noise, can significantly reduce the degree to which this exploitable phenomenon occurs. While strategies to cope with this issue already exist and are certainly improvements over ignoring the problem, in this work we present a new one based on the EM algorithm that offers several advantages. While in this work we focus on the application to Vecchia’s approximation (Vecchia), a particularly popular and powerful framework in which we can demonstrate true second-order optimization of M steps, the method can also be applied using entirely matrix-vector products, making it applicable to a very wide class of precision matrix-based approximation methods. Supplementary materials for this article are available online.

Second-Order Optimality Conditions and Duality for Multiobjective Semi-Infinite Programming Problems on Hadamard Manifolds

This paper is devoted to the study of multiobjective semi-infinite programming problems on Hadamard manifolds. We consider a class of multiobjective semi-infinite programming problems (abbreviated as MSIP) on Hadamard manifolds. We use the concepts of second-order Karush–Kuhn–Tucker stationary point and second-order Karush–Kuhn–Tucker geodesic pseudoconvexity of the considered problem to derive necessary and sufficient second-order conditions of efficiency, weak efficiency and proper efficiency for MSIP along with certain generalized geodesic convexity assumptions. Moreover, we formulate the second-order Mond–Weir-type dual problem related to MSIP and deduce weak and strong duality theorems relating MSIP and the dual problem. The significance of our results is demonstrated with the help of non-trivial examples. To the best of our knowledge, this is the first time that second-order optimality conditions for MSIP have been studied in Hadamard manifold setting.

A simple proof of second-order sufficient optimality conditions in nonlinear semidefinite optimization

In this note, we present an elementary proof for a well-known second-order sufficient optimality condition in nonlinear semidefinite optimization which does not rely on the enhanced theory of second-order tangents. Our approach builds on an explicit elementary computation of the so-called second subderivative of the indicator function associated with the semidefinite cone which recovers the best curvature term known in the literature.

A second-order and slice-specific linear shimming technique to improve spinal cord fMRI

PurposeTo develop a second-order and slice-specific linear shimming technique and investigate its efficiency in the mitigation of signal loss and distortions, and the increase of temporal signal-to-noise ratio (tSNR) within the spinal cord during functional Magnetic Resonance Imaging (fMRI) of the human cervical spinal cord. MethodsAll scans were performed on a General Electric Discovery MR750 3 T scanner, using a head, neck and spine coil and a neurovascular array. To improve B0 homogeneity, a field map was acquired, and second-order shims (SOS) were optimized over manually defined regions of interest (ROIs). Signal loss from dephasing by susceptibility-induced gradients was reduced by optimizing slice-specific x-, y- and z-shims to maximize signal within the spinal cord. Spectral-spatial excitation pulses were used in both the slice-specific linear shimming calibration scan and fMRI acquisitions. The shimming technique's efficiency was initially tested on eight healthy volunteers by comparing tSNR between images acquired with the manufacturer's standard linear shimming and with our SOS and xyz-shimming technique. Subsequently, using an increased spatial resolution as needed for fMRI of the spinal cord, tSNR measurements were performed on resting-state fMRI images from 14 healthy participants. ResultsSpinal fMRI images acquired with only the standard linear shimming suffered from severe signal loss below the C5 vertebral level. The developed shimming technique compensated for this loss especially at levels C6 and C7, while tSNR was significantly higher at all vertebral levels with SOS and xyz-shimming than without it. ConclusionA comprehensive shimming approach which includes the use of spectral-spatial excitation pulses along with both second-order and slice-specific linear shim optimization reduces regional signal loss and increases tSNR along the c-spine (C3-C7), improving the ability to record functional signals from the human spinal cord.

Rolling optimization of integrated electric and cooling systems considering heating inertia of pipelines

In integrated energy systems, large fluctuations of renewable energies can be lowered effectively through optimal scheduling for the district heating and electric power systems. In this paper, an optimal scheduling scheme is proposed with the following steps: 1) A pipeline decomposition-based method is developed for improving the accuracy of the heating system model; 2) Based on the improved heating system model, a rolling optimization model for the operation of district heating and power systems is established considering the dynamic virtual energy storage of heating inertia in heating pipelines and buildings; 3) The inequality equations of electric power flows obtained by the second-order cone optimization are taken as parts of the constraints; 4) A cost-function that can evaluate the minimization of the differences between exchange power and day-ahead scheduling power in the future time is set. In theory, the proposed method can accurately predict the powers of renewable energies and outdoor temperature several steps ahead. For verification, an integrated energy system composed of an IEEE-33 bus and a 13-pipeline heating system is applied to testing. The results show that the method has the capability of smoothing out the power fluctuations caused by renewable energies and load uncertainty.

Fréchet Second-Order Subdifferentials of Lagrangian Functions and Optimality Conditions

We establish some new results on second-order (necessary and sufficient) optimality conditions for minimization problems with abstract constraints in infinite-dimensional spaces, where the objective functions are only assumed to be -smooth. For doing so, we apply the concept of Fréchet (regular) second-order subdifferential from variational analysis to the Lagrangian function of the problem under investigation. Our results extend and refine several existing ones.

Locally Hölder Continuity of the Solution Map to a Boundary Control Problem with Finite Mixed Control-State Constraints

The local stability of the solution map to a parametric boundary control problem governed by semilinear elliptic equations with finite mixed pointwise constraints is considered in this paper. We prove that the solution map is locally Hölder continuous in -norm of control variable when the strictly nonnegative second-order optimality conditions are satisfied for the unperturbed problem.

Second order optimality conditions for minimization on a general set. Part 1: Applications to mathematical programming

This paper is devoted to second-order optimality conditions for minimization of a C2 function f on a general set K in a Banach space X. We consider both necessary and sufficient conditions of the second-order which differ by the strengthening of inequalities in their formulations. The conditions use first and second order approximations (first and second-order tangents) of the set K. The no gap sufficient conditions need additional assumptions in comparison with necessary conditions. We show that these assumptions hold true in the case when the set K is an intersection of a finite number of sets described by smooth inequalities and equalities, like in problems of the mathematical programming. Moreover, we illustrate the new conditions by deducing some mathematical programming results. In this sense the paper is partly a survey. One non-trivial illustrative example in an infinite dimensional space concerns the case when K can not be represented as an intersection described above. The novelty of our approach is due, on one hand, to the arbitrariness of the set K, and on the other hand, to quite straightforward proofs.

FlexiBERT: Are Current Transformer Architectures too Homogeneous and Rigid?

The existence of a plethora of language models makes the problem of selecting the best one for a custom task challenging. Most state-of-the-art methods leverage transformer-based models (e.g., BERT) or their variants. However, training such models and exploring their hyperparameter space is computationally expensive. Prior work proposes several neural architecture search (NAS) methods that employ performance predictors (e.g., surrogate models) to address this issue; however, such works limit analysis to homogeneous models that use fixed dimensionality throughout the network. This leads to sub-optimal architectures. To address this limitation, we propose a suite of heterogeneous and flexible models, namely FlexiBERT, that have varied encoder layers with a diverse set of possible operations and different hidden dimensions. For better-posed surrogate modeling in this expanded design space, we propose a new graph-similarity-based embedding scheme. We also propose a novel NAS policy, called BOSHNAS, that leverages this new scheme, Bayesian modeling, and second-order optimization, to quickly train and use a neural surrogate model to converge to the optimal architecture. A comprehensive set of experiments shows that the proposed policy, when applied to the FlexiBERT design space, pushes the performance frontier upwards compared to traditional models. FlexiBERT-Mini, one of our proposed models, has 3% fewer parameters than BERT-Mini and achieves 8.9% higher GLUE score. A FlexiBERT model with equivalent performance as the best homogeneous model has 2.6× smaller size. FlexiBERT-Large, another proposed model, attains state-of-the-art results, outperforming the baseline models by at least 5.7% on the GLUE benchmark.

Faster Riemannian Newton-type optimization by subsampling and cubic regularization

This work is on constrained large-scale non-convex optimization where the constraint set implies a manifold structure. Solving such problems is important in a multitude of fundamental machine learning tasks. Recent advances on Riemannian optimization have enabled the convenient recovery of solutions by adapting unconstrained optimization algorithms over manifolds. However, it remains challenging to scale up and meanwhile maintain stable convergence rates and handle saddle points. We propose a new second-order Riemannian optimization algorithm, aiming at improving convergence rate and reducing computational cost. It enhances the Riemannian trust-region algorithm that explores curvature information to escape saddle points through a mixture of subsampling and cubic regularization techniques. We conduct rigorous analysis to study the convergence behavior of the proposed algorithm. We also perform extensive experiments to evaluate it based on two general machine learning tasks using multiple datasets. The proposed algorithm exhibits improved computational speed, e.g., a speed improvement from 12% :text {to} :227%, and improved convergence behavior, e.g., an iteration number reduction from mathcal{O}left(maxleft(epsilon_g^{-2}epsilon_H^{-1},epsilon_H^{-3}right)right) ,text {to}: mathcal{O}left(maxleft(epsilon_g^{-2},epsilon_H^{-3}right)right), compared to a large set of state-of-the-art Riemannian optimization algorithms.

On the Weak Second-order Optimality Condition for Nonlinear Semidefinite and Second-order Cone Programming

On the Weak Second-order Optimality Condition for Nonlinear Semidefinite and Second-order Cone Programming

Normality and second-order optimality conditions in state-constrained optimal control problems with bounded minimizers

A smooth optimal control problem with endpoint, mixed and state constraints is considered. The controllability matrix is constructed and the normality condition is presented as the condition of full rank for this matrix. The approach based on the controllability matrix allows one to examine the case of state-constrained control problems with fixed endpoints. As an application of normality, the second-order necessary optimality conditions are derived. The connection between different forms of second-order conditions is indicated.

Combining High-Order Metric Interpolation and Geometry Implicitization for Curved r-Adaption

We detail how to use Newton's method for distortion-based curved $r$-adaption to a discrete high-order metric field while matching a target geometry. Specifically, we combine two terms: a distortion measuring the deviation from the target metric; and a penalty term measuring the deviation from the target boundary. For this combination, we consider four ingredients. First, to represent the metric field, we detail a log-Euclidean high-order metric interpolation on a curved (straight-edged) mesh. Second, for this metric interpolation, we detail the first and second derivatives in physical coordinates. Third, to represent the domain boundaries, we propose an implicit representation for 2D and 3D NURBS models. Fourth, for this implicit representation, we obtain the first and second derivatives. The derivatives of the metric interpolation and the implicit representation allow minimizing the objective function with Newton's method. For this second-order minimization, the resulting meshes simultaneously match the curved features of the target metric and boundary. Matching the metric and the geometry using second-order optimization is an unprecedented capability in curved (straight-edged) $r$-adaption. This capability will be critical in global and cavity-based curved (straight-edged) high-order mesh adaption.

An Irregular Tiled Array Technique for Microwave Wireless Power Transmission

An irregular tiled array technique with the maximization of beam collection efficiency (BCE) for microwave wireless power transmission (WPT) is addressed in this paper, where the radiating elements are irregularly partitioned into multiple subarrays for reducing the number of radio frequency (RF) chain. The system architectures of WPT based on irregular tiled arrays are analyzed for the exhibition of its low-cost advantages. To achieve a good trade-off between the manufacturing cost and beampattern, a mixed-integer optimization problem (MIOP) accounting for the maximization of BCE satisfying the subarray tiling configuration, scanning direction and peak sidelobe level (SLL) constraints is established. By equivalently reformulating the beampattern, the MIOP in terms of subarray tiling configuration and subarray excitations is decoupled, and a two-stage solution method can be implemented. Firstly, resort to ‘generalized BCE’, the binary quadratic optimization problem (BQOP) for maximizing the BCE at considered scanning angles while guaranteeing the full aperture coverage is reduced to a binary second-order cone optimization problem. After the subarray tiling configuration is determined, the non-convex Rayleigh entropy maximization problem at a given scanning angle with respect to subarray complex excitations while satisfying peak SLL constraint is globally solved by using iterative convex approximation algorithm. Owing to the high efficiency of convex optimization, the BCE is able to monotonically increase to a stable value. Several numerical results with different array size and subarray shapes are provided to assess the effectiveness of the proposed method by comparing to the competitive counterparts in the open literatures.

First- and second-order optimality conditions for second-order cone and semidefinite programming under a constant rank condition

First- and second-order optimality conditions for second-order cone and semidefinite programming under a constant rank condition

On the Strong Subregularity of the Optimality Mapping in an Optimal Control Problem with Pointwise Inequality Control Constraints

This paper presents sufficient conditions for strong metric subregularity (SMsR) of the optimality mapping associated with the local Pontryagin maximum principle for Mayer-type optimal control problems with pointwise control constraints given by a finite number of inequalities G_j(u)le 0. It is assumed that all data are twice smooth, and that at each feasible point the gradients G_j'(u) of the active constraints are linearly independent. The main result is that the second-order sufficient optimality condition for a weak local minimum is also sufficient for a version of the SMSR property, which involves two norms in the control space in order to deal with the so-called two-norm-discrepancy.

On strong second-order necessary optimality conditions under relaxed constant rank constraint qualification

We discuss the (first- and second-order) optimality conditions for nonlinear programming under the relaxed constant rank constraint qualification (RCRCQ). Although the optimality conditions are well established in the literature, the proofs presented here are based solely on the well-known inverse function theorem. This is the only prerequisite from real analysis used to establish two auxiliary results needed to prove the optimality conditions. To be precise, we provide a simple and alternative proof that RCRCQ is a constraint qualification that implies strong second-order optimality conditions.