First-order Optimization Techniques Research Articles

Efficient Vertical Federated Unlearning via Fast Retraining

Vertical federated learning (VFL) revolutionizes privacy-preserved collaboration for small businesses, that have distinct but complementary feature sets. However, as the scope of VFL expands, the constant entering and leaving of participants, as well as the subsequent exercise of the “right to be forgotten” pose a great challenge in practice. The question of how to efficiently erase one’s contribution from the shared model remains largely unexplored in the context of vertical federated learning. In this paper, we introduce a vertical federated unlearning framework, which integrates model checkpointing techniques with a hybrid, first-order optimization technique. The core concept is to reduce backpropagation time and improve convergence/generalization by combining the advantages of the existing optimizers. We provide in-depth theoretical analysis and time complexity to illustrate the effectiveness of the proposed design. We conduct extensive experiments on 6 public datasets and demonstrate that our method could achieve up to 6.3 × speed-up compared to the baseline, with negligible influence on the original learning task.

Optimal inexactness schedules for tunable oracle-based methods

Several recent works address the impact of inexact oracles in the convergence analysis of modern first-order optimization techniques, e.g. Bregman Proximal Gradient and Prox-Linear methods as well as their accelerated variants, extending their field of applicability. In this paper, we consider situations where the oracle's inexactness can be chosen upon demand, more precision coming at a computational price counterpart. Our main motivations arise from oracles requiring the solving of auxiliary subproblems or the inexact computation of involved quantities, e.g. a mini-batch stochastic gradient as a full-gradient estimate. We propose optimal inexactness schedules according to presumed oracle cost models and patterns of worst-case guarantees, covering among others convergence results of the aforementioned methods under the presence of inexactness. Specifically, we detail how to choose the level of inexactness at each iteration to obtain the best trade-off between convergence and computational investments. Furthermore, we highlight the benefits one can expect by tuning those oracles' accuracy instead of keeping it constant throughout. Finally, we provide extensive numerical experiments that support the practical interest of our approach, both in offline and online settings, applied to the Fast Gradient algorithm.

Mixture of experts distributional regression: implementation using robust estimation with adaptive first-order methods

In this work, we propose an efficient implementation of mixtures of experts distributional regression models which exploits robust estimation by using stochastic first-order optimization techniques with adaptive learning rate schedulers. We take advantage of the flexibility and scalability of neural network software and implement the proposed framework in mixdistreg, an R software package that allows for the definition of mixtures of many different families, estimation in high-dimensional and large sample size settings and robust optimization based on TensorFlow. Numerical experiments with simulated and real-world data applications show that optimization is as reliable as estimation via classical approaches in many different settings and that results may be obtained for complicated scenarios where classical approaches consistently fail.

Rethinking data-driven point spread function modeling with a differentiable optical model

In astronomy, upcoming space telescopes with wide-field optical instruments have a spatially varying point spread function (PSF). Specific scientific goals require a high-fidelity estimation of the PSF at target positions where no direct measurement of the PSF is provided. Even though observations of the PSF are available at some positions of the field of view (FOV), they are undersampled, noisy, and integrated into wavelength in the instrument’s passband. PSF modeling represents a challenging ill-posed problem, as it requires building a model from these observations that can infer a super-resolved PSF at any wavelength and position in the FOV. Current data-driven PSF models can tackle spatial variations and super-resolution. However, they are not capable of capturing PSF chromatic variations. Our model, coined WaveDiff, proposes a paradigm shift in the data-driven modeling of the point spread function field of telescopes. We change the data-driven modeling space from the pixels to the wavefront by adding a differentiable optical forward model into the modeling framework. This change allows the transfer of a great deal of complexity from the instrumental response into the forward model. The proposed model relies on efficient automatic differentiation technology and modern stochastic first-order optimization techniques recently developed by the thriving machine-learning community. Our framework paves the way to building powerful, physically motivated models that do not require special calibration data. This paper demonstrates the WaveDiff model in a simplified setting of a space telescope. The proposed framework represents a performance breakthrough with respect to the existing state-of-the-art data-driven approach. The pixel reconstruction errors decrease six-fold at observation resolution and 44-fold for a 3x super-resolution. The ellipticity errors are reduced at least 20 times, and the size error is reduced more than 250 times. By only using noisy broad-band in-focus observations, we successfully capture the PSF chromatic variations due to diffraction. WaveDiff source code and examples associated with this paper are available at this link .

A new efficient algorithm based on feedforward neural network for solving differential equations of fractional order

Artificial neural network (ANN) have shown great success in various scientific fields over several decades. Recently, one of its variants known as deep feedforward neural network (FNN) led to dramatic improvement in many tasks, including getting more accurate approximation solution for integer-order differential equations. However, its capability on solving FDEs is still remain questionable. Thus, this paper aims to design new scheme based on deep feedforward neural network (FNN) with vectorized algorithm (FNNVA) using selected first-order optimization techniques which are gradient descent (GD), momentum method (MM) and adaptive moment estimation method (Adam) to solve Caputo FDEs. At the first stage, a detailed method formulations on solving Caputo FDEs using FNN are presented. Then, a vectorized algorithm is developed for the scheme to be computationally efficient. The effectiveness and applicability of the scheme were validate on linear and nonlinear FDEs through comparison based on different number of hidden layers with varying learning rates and number of neurons. The results show that FNNVA with Adam technique with one and two hidden layers outperformed among others by appropriate selection value of learning rates and number of neurons respectively. This scheme also provide high accuracy and low computational cost compared to several existing numerical methods.

Input initialization for inversion of neural networks using k-nearest neighbor approach

Inversion of neural networks aims to find optimal input variables given a target output, and is widely applicable in an industrial field such as optimizing control variables of complex systems in manufacturing facilities. To achieve optimal inputs using a standard first-order optimization technique, proper initialization of input variables is essential. This paper presents a new initialization method for input variables of neural networks based on k-nearest neighbor (k-NN) approach. The proposed method finds inputs which resulted in an output close to a target output in a training dataset, and combine them to form initial input variables. Experiments on a toy dataset demonstrate that our method outperforms random initialization. Also, we introduce an exhaustive case study on power scheduling of a heating, ventilation, and air conditioning (HVAC) system in a building to support the effectiveness of the algorithm.

Parameter estimation of an improved nonlinear Muskingum model using a new hybrid method

Although various techniques have been proposed to estimate the parameters of different versions of the Muskingum model, more rigorous techniques and models are still required to improve the computational precision of the calibration process. In this research, a new hybrid technique was proposed for Muskingum parameter estimation. Based on the conducted comprehensive literature review on the Muskingum flood routing models, a new improved Muskingum model with nine constant parameters was presented. Since the inflow-weighted parameter in the proposed model is a function of inflow hydrograph, it varies during the flood period and consequently can also be considered as a variable-parameter Muskingum model. The new hybrid technique was successfully applied for parameter estimation of the new version of Muskingum model for two case studies selected from the literature. Results were compared with those of other methods using several common performance evaluation criteria. The new Muskingum model significantly reduces the sum of the square of the deviations between the observed and routed outflows (SSQ) value for the double-peak case study. Finally, the obtained results indicate that not only the hybrid modified honey bee mating optimization-generalized reduced gradient algorithm somehow overcomes the shortcomings of both zero and first-order optimization techniques, but also the new Muskingum model appears to be the most reliable Muskingum version compared with the other methods considered in this study.

Optimizing the design of bio-inspired functionally graded material (FGM) layer in all-ceramic dental restorations

Due to elastic modulus mismatch between the different layers in all-ceramic dental restorations, high tensile stress concentrates at the interface between the ceramic core and cement. In natural tooth structure, stress concentration is reduced by the functionally graded structure of dentin-enamel junction (DEJ) which interconnects enamel and dentin. Inspired by DEJ, the aim of this study was to explore the optimum design of a bio-inspired functionally graded material (FGM) layer in all-ceramic dental restorations to achieve excellent stress reduction and distribution. Three-dimensional finite element model of a multi-layer structure was developed, which comprised bilayered ceramic, bio-inspired FGM layer, cement, and dentin. Finite element method and first-order optimization technique were used to realize the optimal bio-inspired FGM layer design. The bio-inspired FGM layer significantly reduced stress concentration at the interface between the crown and cement, and stresses were evenly distributed in FGM layer. With the optimal design, an elastic modulus distribution similar to that in DEJ occurred in the FGM layer.

On first-order algorithms forl1/nuclear norm minimization

In the past decade, problems related tol1/nuclear norm minimization have attracted much attention in the signal processing, machine learning and optimization communities. In this paper, devoted tol1/nuclear norm minimization as ‘optimization beasts’, we give a detailed description of two attractive first-order optimization techniques for solving problems of this type. The first one, aimed primarily at lasso-type problems, comprises fast gradient methods applied to composite minimization formulations. The second approach, aimed at Dantzig-selector-type problems, utilizes saddle-point first-order algorithms and reformulation of the problem of interest as ageneralized bilinear saddle-point problem. For both approaches, we give complete and detailed complexity analyses and discuss the application domains.

Topology Optimization of Flywheel Rotors Using SIMP Method: A Preliminary Study

Flywheels are kinetic energy storage and retrieval devices as chemical batteries. However, the high charge and discharge rates, as well as the high cycling capability make flywheels attractive as compared to other energy storage devices. This research serves as a preliminary study that aims for developing a technique in designing a flywheel rotor based on the solid isotropic method with penalization (SIMP) method. Examples are presented to illustrate the optimum structural layouts obtained given various design objectives. For a static rotor, the objectives are maximizing the first torsional natural frequency, maximizing the moment of inertia and maximizing both of them, respectively. The problem is formulated using bound formulation and the method of moving asymptotes (MMA), a first-order optimization technique, was employed. Therefore the design sensitivity becomes a necessity. The so-called checkerboard problem in the topology optimization is avoided using the nodal design variable. Also, a threshold is used to reduce the numerical imperfection in each iteration. For the topology design of a rotating rotor, the centrifugal force induced in the high-speed rotation is considered. The objective is to maximize the rotor stiffness and is demonstrated in the last example. Results show clear topology layout of flywheel was obtained using proposed method.