Gradient-based Optimization Method Research Articles

On the fusion of soft-decision-trees and concept-based models

In the field of eXplainable Artificial Intelligence (XAI), the generation of interpretable models that are able to match the performance of state-of-the-art deep learning methods is one of the main challenges. In this work, we present a novel interpretable model for image classification that combines the power of deep convolutional networks and the transparency of decision trees. We explore different training techniques where convolutional networks and decision trees can be trained together using gradient-based optimization methods as usually done in deep learning environments. All of this results in a transparent model in which a soft decision tree makes the final classification based on human-understandable concepts that are extracted by a convolutional neural network. We tested the proposed solution on two challenge image classification datasets and compared them with the state-of-the-art approaches, achieving competitive results.

Inverse analysis of granular flows using differentiable graph neural network simulator

Inverse problems in granular flows, such as landslides and debris flows, involve estimating material parameters or boundary conditions based on a target runout profile. Traditional high-fidelity simulators for these inverse problems are computationally expensive, restricting the number of possible simulations. These simulators are also non-differentiable, making efficient gradient-based optimization methods for high-dimensional spaces inapplicable. Machine learning-based surrogate models offer computational efficiency and differentiability. However, they often struggle to generalize beyond their training data due to relying on low-dimensional input–output mappings that fail to capture the complete physics of granular flows. We propose a novel differentiable graph neural network simulator (GNS) that combines reverse mode automatic differentiation of graph neural networks with gradient-based optimization for solving inverse problems. GNS learns the dynamics of granular flow by representing the system as a graph and predicts the evolution of the graph at the next timestep, given the current state. The differentiable GNS demonstrates optimization capabilities beyond the training data. We demonstrate the effectiveness of our method for inverse estimation across single and multi-parameter optimization problems, including evaluating material properties and boundary conditions for a target runout distance and designing baffle locations to limit a landslide runout. Our proposed differentiable GNS framework solves inverse problems with orders of magnitude faster convergence than the conventional gradient-based optimization approach using finite difference.

A gradient aerodynamic optimization method based on deep learning

Gradient-based aerodynamic optimization methods involve complex gradient calculations with high computational costs. To address these problems, this paper proposes a rapid deep learning-based method for predicting the aerodynamic force gradients and establishes an aerodynamic optimization framework. A deep neural network is used to determine the latent mapping relationship between the near-wall flow field information and the aerodynamic force gradients, and the resulting gradient prediction model is integrated into an aerodynamic optimization platform. Combining grid deformation techniques with gradient-based optimization enables the aerodynamic optimization of airfoils. The deep neural network model can rapidly predict gradients that are basically consistent with those from the traditional discrete adjoint method, and decouples the gradient computation from the flow field solution process. The proposed method is validated through the aerodynamic optimization of the airfoil. Under various optimization objectives and initial conditions, the deep neural network model significantly improves the aerodynamic performance of the airfoil and substantially enhances the efficiency of optimization compared with traditional methods.

Ensemble variational Bayesian approximation for the inversion and uncertainty quantification of Darcy flows in heterogeneous porous media with random parameters

The inversion and uncertainty quantification of physical parameters are important for many science and engineering problems. For nonlinear parameter inversion problems of flows in hydrocarbon reservoirs, estimating the true posterior probability distribution function for the random parameter is difficult when the dimension of the parameter is relatively high. Inversion methods such as gradient-based or heuristic optimization methods searching for the maximum a posterior are generally point-estimate, while uncertainty quantification using MCMC is computationally expensive. In the current study, a new ensemble variational Bayesian approximation method is developed to estimate the posterior probability density using an ensemble of realizations regarded as a trial sample distribution. The objective is to optimize the trial sample distribution such that the evidence lower bound is maximized indicating that the distance between the trial and true posterior distribution is minimized. A subspace is built using principle component analysis based on finite samples to obtain a diagonal positive-definite covariance matrix, and ensemble-based data assimilation is used to optimize the trial distribution efficiently. Heterogeneous 2D and 3D test cases of transient single- and two-phase Darcy flows are presented for validation.

Symbolic Cognitive Diagnosis via Hybrid Optimization for Intelligent Education Systems

Cognitive diagnosis assessment is a fundamental and crucial task for student learning. It models the student-exercise interaction, and discovers the students' proficiency levels on each knowledge attribute. In real-world intelligent education systems, generalization and interpretability of cognitive diagnosis methods are of equal importance. However, most existing methods can hardly make the best of both worlds due to the complicated student-exercise interaction. To this end, this paper proposes a symbolic cognitive diagnosis~(SCD) framework to simultaneously enhance generalization and interpretability. The SCD framework incorporates the symbolic tree to explicably represent the complicated student-exercise interaction function, and utilizes gradient-based optimization methods to effectively learn the student and exercise parameters. Meanwhile, the accompanying challenge is that we need to tunnel the discrete symbolic representation and continuous parameter optimization. To address this challenge, we propose to hybridly optimize the representation and parameters in an alternating manner. To fulfill SCD, it alternately learns the symbolic tree by derivative-free genetic programming and learns the student and exercise parameters via gradient-based Adam. The extensive experimental results on various real-world datasets show the superiority of SCD on both generalization and interpretability. The ablation study verifies the efficacy of each ingredient in SCD, and the case study explicitly showcases how the interpretable ability of SCD works.

Fast Coherent Video Style Transfer via Flow Errors Reduction

For video style transfer, naively applying still image techniques to process a video frame-by-frame independently often causes flickering artefacts. Some works adopt optical flow into the design of temporal constraint loss to secure temporal consistency. However, these works still suffer from incoherence (including ghosting artefacts) where large motions or occlusions occur, as optical flow fails to detect the boundaries of objects accurately. To address this problem, we propose a novel framework which consists of the following two stages: (1) creating new initialization images from proposed mask techniques, which are able to significantly reduce the flow errors; (2) process these initialized images iteratively with proposed losses to obtain stylized videos which are free of artefacts, which also increases the speed from over 3 min per frame to less than 2 s per frame for the gradient-based optimization methods. To be specific, we propose a multi-scale mask fusion scheme to reduce untraceable flow errors, and obtain an incremental mask to reduce ghosting artefacts. In addition, a multi-frame mask fusion scheme is designed to reduce traceable flow errors. In our proposed losses, the Sharpness Losses are used to deal with the potential image blurriness artefacts over long-range frames, and the Coherent Losses are performed to restrict the temporal consistency at both the multi-frame RGB level and Feature level. Overall, our approach produces stable video stylization outputs even in large motion or occlusion scenarios. The experiments demonstrate that the proposed method outperforms the state-of-the-art video style transfer methods qualitatively and quantitatively on the MPI Sintel dataset.

Modeling and multi-objective optimization of forward osmosis process

In order to ensure efficient wastewater treatment and seawater desalination, adequate modeling and optimization of the forward osmosis (FO) process has the potential to be very helpful. This paper deals with the FO model parameters calibration and FO process optimization by a gradient-based optimization method. For this purpose, an upgraded FO model, which involves temperature- and agent-dependent parameters, was developed. The FO model calibration was done using NaCl as agents in draw solution, while MgCl2 was used for model validation. The agreements between simulated and measured FO performance were satisfactory; relative index of agreement are higher than 0.99. By using the proposed FO model, the optimization of FO process conditions was performed with various definitions of the objective and constraint functions. In case of maximizing the water flux, minimizing reverse solute flux, and fulfilling the required constraints, the ratio of water flux and reverse solute flux increased up to 40 % for NaCl and up to 20 % for MgCl2; meanwhile the effective osmotic pressure difference was improved 2-times for NaCl and up to 3.8-times for MgCl2. The optimization process proved to be stable and efficient and can easily be adapted or upgraded for more complex dynamic FO modeling.

Manifold-guided multi-objective gradient algorithm combined with adjoint method for supersonic aircraft shape design

The aerodynamic design of supersonic aircraft necessitates the comprehensive consideration of multiple conflicting design objectives and multi-disciplinary characteristics. However, the efficiency of heuristic algorithms in solving multi-objective aerodynamic problems significantly decreases with the increase in the number of design variables and objectives. Although gradient-based adjoint optimization methods can eliminate the constraints of design variables on aerodynamic optimization and have high optimization efficiency, they are prone to getting stuck in local optima and are difficult to apply in multi-objective optimization. To overcome these shortcomings, a manifold-guided multi-objective gradient-based algorithm (MG-MOGBA) is proposed. This algorithm enhances the optimization efficiency of the algorithm by coupling the multi-objective gradient operator with the predicted Pareto front (PF) manifold structure to guide the population search direction and reduce the search space. A multi-objective aerodynamic optimization framework based on the discrete adjoint method is developed to automate the aerodynamic optimization process. Test functions and airfoil optimization results confirm that the proposed algorithm achieves superior results with fewer iterations and is more efficient than heuristic algorithms. The four-objective supersonic civil aircraft example demonstrates that the algorithm can effectively handle high-dimensional multi-objective aerodynamic optimization problems. A potential equilibrium solution increased the cruise factor by 17.5% and 12.8% under supersonic and transonic conditions respectively, reduced the maximum overpressure of the sonic boom by 25.1%, and decreased the root bending moment by 20.3%. Overall, MG-MOGBA can efficiently solve high-dimensional multi-objective aerodynamic optimization problems and has good scalability for the dimensions of design objectives and variables.

Improvement of Biogas Production Utilizing a Complex Anaerobic Digestion Model and Gradient-Based Optimization

Anaerobic digestion (AD) is a promising technology for renewable energy production from organic waste. In order to maximize the produced biogas quantity and quality, this paper deals with the optimization of the AD process in a CSTR bioreactor of a full-scale biogas plant. For this purpose, a novel approach was adopted coupling, a highly complex BioModel for AD simulation, and a gradient-based optimization method. In order to improve AD performance, the dosages of various types of biological additives, the dosages of inorganic additives, and the temperature in the bioreactor were optimized in three different scenarios. The best biogas quality was obtained using multi-objective optimization, where the objective function involves the following two conflicting objectives: the maximization of biogas production and minimization of the needed heating energy. The obtained results show that, potentially, the content of CH4 can be increased by 11%, while the contents of H2, H2S, and NH3 can be reduced by 30%, 20%, and 81% when comparing the simulation results with the experimental data. The obtained results confirm the usefulness of the proposed approach, which can easily be adapted or upgraded for other bioreactor types.

Investigating Hessian-based inversion velocity analysis

Inversion velocity analysis (IVA) is an image-domain method built upon the spatial scale separation of the model. Accordingly, the IVA method is performed with an iterative process composed of two minimization steps consisting of migration (inner loop) and tomography (outer loop), respectively, with each step accounting for its Hessian or not. The migration part provides the common-image gathers (CIGs) with extension in the horizontal subsurface offset. Then, the differential semblance optimization (DSO) misfit measures the focusing of the events in the CIGs, which indicates the quality of the velocity model. Commonly, the velocity updates are obtained from the DSO gradient. IVA is a modified version in which the approximate inverse replaces the adjoint of the inner loop process; in that case, the migration Hessian is approximately diagonal in the high-frequency regime. In this work, we report on implementation of the tomographic Hessian (i.e., the second derivative of the DSO misfit with respect to the background model) for the estimation of the background velocity model. We apply the second-order adjoint-state method to obtain the application of the tomographic Hessian on a vector. Then, we use the truncated-Newton (TN) method to obtain the update directions by computing approximately the application of the inverse of the tomographic Hessian on the descent direction. We also make a theoretical comparison between tomography in the IVA and full-waveform inversion contexts. Two numerical examples are used to compare, in terms of geophysical results and computational costs, the TN method with different gradient-based optimization methods applied to the IVA. A small model allows us to evaluate the eigenvalues of the tomographic Hessian, which explains the large damping needed in the TN case.

A Majorization-Minimization Algorithm for Neuroimage Registration.

Intensity-based image registration is critical for neuroimaging tasks, such as 3D reconstruction, times-series alignment, and common coordinate mapping. The gradient-based optimization methods commonly used to solve this problem require a careful selection of step-length. This limitation imposes substantial time and computational costs. Here we propose a gradient-independent rigid-motion registration algorithm based on the majorization-minimization (MM) principle. Each iteration of our intensity-based MM algorithm reduces to a simple point-set rigid registration problem with a closed form solution that avoids the step-length issue altogether. The details of the algorithm are presented, and an error bound for its more practical truncated form is derived. The performance of the MM algorithm is shown to be more effective than gradient descent on simulated images and Nissl stained coronal slices of mouse brain. We also compare and contrast the similarities and differences between the MM algorithm and another gradient-free registration algorithm called the block-matching method. Finally, extensions of this algorithm to more complex problems are discussed.

Multi-Physics Three-Dimensional Component Placement and Routing Optimization Using Geometric Projection

Abstract This article presents a novel three-dimensional topology optimization framework developed for 3D spatial packaging of interconnected systems using a geometric projection method (GPM). The proposed gradient-based topology optimization method simultaneously optimizes the locations and orientations of system components (or devices) and lengths, diameters, and trajectories of interconnects to reduce the overall system volume within the prescribed 3D design domain. The optimization is subject to geometric and physics-based constraints dictated by various system specifications, suited for a wide range of transportation (aerospace or automotive), heating, ventilation, air-conditioning, and refrigeration, and other complex system applications. The system components and interconnects are represented using 3D parametric shapes such as cubes, cuboids, and cylinders. These objects are then projected onto a three-dimensional finite element mesh using the geometric projection method. Sensitivities are calculated for the objective function (bounding box volume) with various geometric and physics-based (thermal and hydraulic) constraints. Several case studies were performed with different component counts, interconnection topologies, and system boundary conditions and are presented to exhibit the capabilities of the proposed 3D multi-physics spatial packaging optimization framework.

Partial quadratic eigenvalue assignment for vibration systems using receptances and system matrices

In this paper, we consider the partial quadratic eigenvalue assignment problem (PQEAP) for vibration system by active feedback control, such that some “troublesome” eigenpairs are replaced by some suitable chosen or newly measured ones, while the remaining unknown eigenpairs are kept unchanged. With the assumptions that the mass and stiffness matrices are nonsingular, we give a new proof of the spectral decomposition of quadratic symmetric pencil. And then we provide some necessary and sufficient conditions of the feedback matrices so that the closed-loop can preserve the no spill-over property. With these conditions, we characterize a set of parametric solutions to the PQEAP using receptances, system matrices and a few open-loop eigenpairs. Finally, we propose a gradient-based optimization method for solving the minimum norm solution to the PQEAP. Numerical experiments illustrate the effectiveness and robustness of the proposed algorithm.

Wind energy harvesting with building-integrated ducted openings: CFD simulation and neural network optimization

Building-integrated wind energy harvesting offers a promising local and decentralized energy generation solution, reducing transmission losses and infrastructure expenses. However, the complex urban wind patterns demand accurate aerodynamic optimization for optimal performance. This study seeks to optimize the geometric design of building-integrated ducted openings to maximize wind energy harvesting. The research aims to create a data-driven model predicting the openings' performance accurately, explore the interlinked impacts of different geometrical variables, find optimal configurations across different building heights, and understand underlying aerodynamic mechanisms. The proposed design comprises a nozzle, throat, and diffuser. To increase power density, a data-driven design optimization method is employed. The optimization process involves determining the optimum opening height from the ground, along with five geometrical variables: inlet diameter, throat diameter, outlet diameter, nozzle length, and diffuser length. A comprehensive test matrix is developed using design of experiments (DoE) to build an artificial neural network (ANN) model. This neural network is developed using computational fluid dynamics (CFD) simulations validated with experimental data. Subsequently, a gradient-based global optimization method fine-tunes the neural network and identifies optimal designs for various heights. The research concludes that nozzle and diffuser opening angles within the range of 19° ≤ ΦNoz ≤ 23°and 7.5° ≤ ΦDif ≤ 8.2°, respectively, result in the optimal converging-diverging ducted opening geometry. This optimal opening can yield a power density increase of 31.5 to 35.5 times when compared to freestream flow and 17.5 to 22.4 times in comparison to cylindrical openings, depending on the opening location along the building height.

Deep reinforcement learning with light-weight vision model for sequential robotic object sorting

Training an agent to generate cooperative joint learning policies for executing object sorting in cluttered scenes is complex. This paper presents a comparative model-free deep reinforcement learning (DRL) system built from variants of end-to-end lightweight deep neural networks that jointly complement three primitive policies (pushing, grasping, and placing) that operate from visual observations to actions. The end goal of the proposed DRL systems is to perform object sorting in very dynamic and complex environmental scenarios of cluttered and random objects (irregular and regular blocks) existing in different color variations. For this, we investigated DRL methods built from 12 custom instances of Pixel-wise Q-valued Critic Networks (PQCN) from four main backbone networks (DenseNet121, DenseNet169, MobileNetV3, and SqueezeNet) individually combined with custom fully convolutional neural networks (FCN) for learning the image-based observation space through affordance mapping paradigm while considering both dual and single transfer learning schemes. Additionally, we explored three categories of gradient-based optimization methods and considered custom reward functions with varying discount factors. The results show that the PQCN constructed by integrating DenseNet121 or MobileNetV3 with FCN and trained by a dual transfer learning scheme (with a discount factor of γ=0.70) yielded the best performance in most categories of our evaluation in the training phases. Overall, the PQCN-DenseNet121, when trained with a dual transfer learning process, yielded the best generalization (sorting success rate) in most of the evaluation metrics across all object categories in the testing phase. Furthermore, our study provides a new benchmark for validating deep reinforcement learning algorithms while sorting object blocks under varying degrees of complexity. This research finds practical relevance in industrial fields such as manufacturing and packaging, building construction, plant sorting, automobile assembly lines, and waste sorting. Here is the article’s source code: https://github.com/Emmanuel-Okafor/DRL_for_Object_Sorting.git.

Credit Risk Modeling Using Interpreted XGBoost

Purpose: The aim of the paper is to develop a credit risk assessment model usingb the XGBoost classifier supported by interpretation issues. Design/methodology/approach: The risk modeling is based on Extreme Gradient Boosting (XGBoost) in the research. It is a method used for regression and classification problems. It is based on a sequence of decision trees using a gradient-based optimization method of the loss function to minimize the errors of weak estimators. We use also methods for performing local and global interpretability: ceteris paribus charts, SHAP and feature importance approach. Findings: Based on the research results, it can be concluded that XGBoost achieved higher values of performance metrics than logistic regression, except sensitivity. It means that XGBoost indicated a smaller percentage of all bad client. Results of local interpretability enable a conclusion that in the case of the client in question, the credit decision is positively influenced by credit scores from external suppliers, while it is negatively influenced by minimal external scoring and short seniority. The number of years in the car and higher education are also positive. Such information helps to justify a negative credit decision. Results of global interpretability enable a conclusion that higher values of the traits associated with the z-scores are accompanied by negative Shapley values, which can be interpreted as a negative effect on the explanatory variable. Research limitations/implications: XGBoost, A ceteris paribus plot, SHAP, and feature importance methods can be used to develop a credit risk assessment model including machine learning interpretability. The main limitation of research is to compare the results of XGBoost only to the logistic regression results. Future research should focus on comparing the results of XGBoost to other machine learning methods, including neural networks. Originality/value: One of the key processes in a bank is the credit decision process, which is the evaluation of a client’s repayment risk. In the consumer finance sector, the processes are usually largely automated, and increasingly the latest machine learning methods based on neural networks and ensemble learning methods are being used for the purpose. Although machine learning models allow for achieving higher accuracy of credit risk assessment compared to traditional statistical methods, the main problem is the low interpretability of machine learning models. The models often perform as the “black box”. However, the interpretation of the results of risk assessment models is very important due to the need to explain to the client the reasons for assessing their credit risk.

Koopman modeling and optimal control for microbial fed-batch fermentation with switching operators

The modeling of microbial fed-batch fermentation with switching operators to product 1,3-propanediol (1,3-PD) still maintains a challenge because it is strongly of nonlinearity and uncertainty. Machine learning methods for learning such models have become a hot research topic, but the interpretability of existing techniques remains a challenging problem. Recently, the Koopman operator, which is a linear operator governing the eigenfunction evolution along trajectories of a nonlinear dynamical system with switching operators, has been studied for modeling complex dynamics. In this paper, we propose a Koopman modeling method based on an interpretable Koopman operator. The predominant merit of using the Koopman operator is to offer a linear infinite dimensional description of a nonlinear dynamical system with switching operators. In the proposed method, an enhanced learning-based extended dynamic mode decomposition (enhanced-EDMD) algorithm based on a novel eigenfunction construction method is proposed to obtain a finite-dimensional approximation of the Koopman operator. The convergence analysis of the enhanced-EDMD algorithm is also studied. Furthermore, to maximize the productivity of 1,3-PD and minimize the total variation in the optimal control within a time frame, an algorithm combining the model predictive control method with the enhanced learning-based EDMD (denoted by MPC-Enhanced-EDMD), based on gradient-based optimization and exact penalty function method, is proposed for devising optimal feeding rate of glycerol evolving with time. Numerical simulations are conducted by demonstrating the effectiveness of the enhanced-EDMD algorithm on the dynamics prediction and the MPC-Enhanced-EDMD method on the optimal feeding rates of glycerol.

Active uncertainty reduction for safe and efficient interaction planning: A shielding-aware dual control approach

The ability to accurately predict others’ behavior is central to the safety and efficiency of robotic systems in interactive settings, such as human–robot interaction and multi-robot teaming tasks. Unfortunately, robots often lack access to key information on which these predictions may hinge, such as other agents’ goals, attention, and willingness to cooperate. Dual control theory addresses this challenge by treating unknown parameters of a predictive model as stochastic hidden states and inferring their values at runtime using information gathered during system operation. While able to optimally and automatically trade off exploration and exploitation, dual control is computationally intractable for general interactive motion planning, mainly due to the fundamental coupling between the robot’s trajectory plan and its prediction of other agents’ intent. In this paper, we present a novel algorithmic approach to enable active uncertainty reduction for interactive motion planning based on the implicit dual control paradigm. Our approach relies on sampling-based approximation of stochastic dynamic programming, leading to a model predictive control problem that can be readily solved by real-time gradient-based optimization methods. The resulting policy is shown to preserve the dual control effect for a broad class of predictive models with both continuous and categorical uncertainty. To ensure the safe operation of the interacting agents, we use a runtime safety filter (also referred to as a “shielding” scheme), which overrides the robot’s dual control policy with a safety fallback strategy when a safety-critical event is imminent. We then augment the dual control framework with an improved variant of the recently proposed shielding-aware robust planning scheme, which proactively balances the nominal planning performance with the risk of high-cost emergency maneuvers triggered by low-probability agent behaviors. We demonstrate the efficacy of our approach with both simulated driving studies and hardware experiments using 1/10 scale autonomous vehicles.

A Functional Approach to Interpreting the Role of the Adjoint Equation in Machine Learning

The connection between numerical methods for solving differential equations and machine learning has been revealed recently. Differential equations have been proposed as continuous analogues of deep neural networks, and then used in handling certain tasks, such as image recognition, where the training of a model includes learning the parameters of systems of ODEs from certain points along their trajectories. Treating this inverse problem of determining the parameters of a dynamical system that minimize the difference between data and trajectory by a gradient-based optimization method presents the solution of the adjoint equation as the continuous analogue of backpropagation that yields the appropriate gradients. The paper explores an abstract approach that can be used to construct a family of loss functions with the aim of fitting the solution of an initial value problem to a set of discrete or continuous measurements. It is shown, that an extension of the adjoint equation can be used to derive the gradient of the loss function as a continuous analogue of backpropagation in machine learning. Numerical evidence is presented that under reasonably controlled circumstances the gradients obtained this way can be used in a gradient descent to fit the solution of an initial value problem to a set of continuous noisy measurements, and a set of discrete noisy measurements that are recorded at uncertain times.

Value-Function-Based Sequential Minimization for Bi-Level Optimization.

Gradient-based Bi-Level Optimization (BLO) methods have been widely applied to handle modern learning tasks. However, most existing strategies are theoretically designed based on restrictive assumptions (e.g., convexity of the lower-level sub-problem), and computationally not applicable for high-dimensional tasks. Moreover, there are almost no gradient-based methods able to solve BLO in those challenging scenarios, such as BLO with functional constraints and pessimistic BLO. In this work, by reformulating BLO into approximated single-level problems, we provide a new algorithm, named Bi-level Value-Function-based Sequential Minimization (BVFSM), to address the above issues. Specifically, BVFSM constructs a series of value-function-based approximations, and thus avoids repeated calculations of recurrent gradient and Hessian inverse required by existing approaches, time-consuming especially for high-dimensional tasks. We also extend BVFSM to address BLO with additional functional constraints. More importantly, BVFSM can be used for the challenging pessimistic BLO, which has never been properly solved before. In theory, we prove the convergence of BVFSM on these types of BLO, in which the restrictive lower-level convexity assumption is discarded. To our best knowledge, this is the first gradient-based algorithm that can solve different kinds of BLO (e.g., optimistic, pessimistic, and with constraints) with solid convergence guarantees. Extensive experiments verify the theoretical investigations and demonstrate our superiority on various real-world applications.