Gradient-based Algorithm Research Articles

Data Heterogeneity-Robust Federated Learning via Group Client Selection in Industrial IoT

Nowadays, the Industrial Internet of Things (IIoT) has played an integral role in Industry 4.0 and produced massive amounts of data for industrial intelligence. These data locate on decentralized devices in modern factories. To protect the confidentiality of industrial data, federated learning (FL) was introduced to collaboratively train shared machine learning (ML) models. However, the local data collected by different devices skew in class distribution and degrade industrial FL performance. This challenge has been widely studied at the mobile edge, but they ignored the rapidly changing streaming data and clustering nature of factory devices, and more seriously, they may threaten data security. In this article, we propose FED GS, which is a hierarchical cloud-edge-end FL framework for 5G empowered industries, to improve industrial FL performance on non-independent and identically distributed (non-j) data. Taking advantage of naturally clustered factory devices, FED GS uses a gradient-based binary permutation algorithm (GBP-CS) to select a subset of devices within each factory and build homogeneous super nodes participating in FL training. Then, we propose a compound-step synchronization protocol to coordinate the training process within and among these super nodes, which shows great robustness against data heterogeneity. The proposed methods are time-efficient and can adapt to dynamic environments, without exposing confidential industrial data in risky manipulation. We prove that FED GS has better convergence performance than FedAvg and give a relaxed condition under which FED GS is more communication efficient. The extensive experiments show that FED GS improves accuracy by 3.5% and reduces training rounds by 59% on average, confirming its superior effectiveness and efficiency on non-i.i.d. data.

Fed-DeepONet: Stochastic Gradient-Based Federated Training of Deep Operator Networks

The Deep Operator Network (DeepONet) framework is a different class of neural network architecture that one trains to learn nonlinear operators, i.e., mappings between infinite-dimensional spaces. Traditionally, DeepONets are trained using a centralized strategy that requires transferring the training data to a centralized location. Such a strategy, however, limits our ability to secure data privacy or use high-performance distributed/parallel computing platforms. To alleviate such limitations, in this paper, we study the federated training of DeepONets for the first time. That is, we develop a framework, which we refer to as Fed-DeepONet, that allows multiple clients to train DeepONets collaboratively under the coordination of a centralized server. To achieve Fed-DeepONets, we propose an efficient stochastic gradient-based algorithm that enables the distributed optimization of the DeepONet parameters by averaging first-order estimates of the DeepONet loss gradient. Then, to accelerate the training convergence of Fed-DeepONets, we propose a moment-enhanced (i.e., adaptive) stochastic gradient-based strategy. Finally, we verify the performance of Fed-DeepONet by learning, for different configurations of the number of clients and fractions of available clients, (i) the solution operator of a gravity pendulum and (ii) the dynamic response of a parametric library of pendulums.

Concurrent multiscale topology optimization: A hybrid approach

This paper presents a hybrid approach for multiscale topology optimization of structures. The topological shape of both macro-structure and micro-structure are concurrently optimized, based on the solid isotropic material with penalization (SIMP) technique in combination with finite element method (FEM). The material is assumed to have periodically patterned micro-structures, such that the effective properties can be evaluated via energy-based homogenization method (EBHM). In every iteration, the effective properties of material are passed to the macroscopic problem, and the macroscopic behavior (e.g. strain energy) is transferred back to the micro-scale problem, where the unit cell representing the micro-structure of material is determined for the next iteration. It is found that the update process can be done separately, i.e., the sensitivity of macro-scale design variables is not required during the update of micro-scale design variables, and vice versa. Hence, the proposal is that the macro-structure is updated by the gradient-free Proportional Topology Optimization (PTO) algorithm to utilize the computational efficiency of PTO. The micro-structure is still updated by the common gradient-based algorithm, namely Optimality Criteria (OC). Three benchmark numerical examples are investigated, demonstrating the feasibility and efficiency of the proposed hybrid approach.

Communication-Aware Orbit Design for Small Spacecraft Swarms Around Small Bodies

Exploration of small Solar System bodies has traditionally been performed by monolithic spacecraft carrying several science instruments. However, instruments often cannot be operated simultaneously due to requirements including viewing angle, illumination, power, and data constraints. This has motivated interest in architectures in which a swarm of small spacecraft, each carrying one instrument, studies a small body after being deployed by a carrier spacecraft; such architectures hold promise to yield significant improvements in mission efficiency, increased data quality, and reduced mission duration. A key difficulty is the selection of orbits for the spacecraft, which must satisfy not only instrument requirements but also communication and data storage constraints. To address this, we propose a novel gradient-based optimization algorithm that simultaneously optimizes spacecraft orbits, observations, and communications; the approach captures instrument requirements, communication bandwidth, and memory usage, and it accommodates irregular gravity field models and surface geometries. Numerical simulations of a six-spacecraft swarm studying the 433 Eros asteroid show that the approach results in a 67.9% increase in data returned as compared to a communication-agnostic approach. Collectively, these results enable system designers to quickly assess the end-to-end performance of multispacecraft constellations, and represent a first step toward communication-aware design of multispacecraft missions for small Solar System body exploration.

Thermal optimization of secondary cooling systems in the continuous steel casting process

A new process design methodology for the cooling system in the continuous steel casting process is introduced. It is based on a semi-empirical approach for choosing the primary cooling zone design parameters and using an optimization-based method to decide on the design parameters corresponding to the secondary cooling zone. The heat flux through the secondary cooling zone is the design parameter in the latter case. Unlike former studies that considered a misfit temperature function as the objective function, it is defined here as the square of the temperature field penalized with the Niyama’s criterion infeasibility to ensure the production of microporosity-free strands. This way, no information about the target temperature field is required in advance. The corresponding heat transfer problem is coupled to CALPHAD to consider commercial steel grades straightforwardly. An efficient gradient-based optimization algorithm is employed to find the optimal values of design parameters. Then, this algorithm is employed in an actual process design problem corresponding to the continuous slab casting of a commercial steel grade. A post-processing approach is suggested to convert the results of numerical optimization to industrially relevant machine parameters, and then they are exploited in practice in the foundry plant. While the optimization problem is only constrained to avoid solidification-induced defects, it produces sound and crack-free strands. Moreover, comparing numerical results to actual surface temperature measurements reveals more petite than a six percent discrepancy between the simulation and experiment. These observations illustrate the feasibility of the presented design methodology as the initial step in deciding on the cooling system parameters in this process.

Proper generalized decomposition solutions for composite laminates parametrized with fibre orientations

Composite materials are gaining popularity as an alternative to classical materials in many different applications. Moreover, their design is even more flexible due to the potential of additive manufacturing. Thus, one can produce a tailored composite laminate with the optimal values of some design parameters providing the desired mechanical performance. In this context, having a parametric numerical model for the mechanical response of the composite laminate is essential to compute the optimal parameters. In the present paper, the design parameters under consideration are the angles describing the orientation of the reinforcement fibers in different layers or patches of the composite laminates. We obtain a generalized solution using Proper Generalized Decomposition (PGD) which is adopted to provide solutions with explicit parametric dependence. The Tsai-Wu failure criterion is used to estimate first ply failure. In this context, Tsai-Wu criterion is used as the objective function for the optimization of the fibre orientations in the laminate. The PGD solution provides also sensitivities for a gradient-based optimization algorithm. The potentiality and efficiency of the presented approach is demonstrated through some numerical tests.

A Survey on Noncooperative Games and Distributed Nash Equilibrium Seeking over Multi-Agent Networks

The work gives a review on the distributed Nash equilibrium seeking of noncooperative games in multi-agent networks, which emerges as one of the frontier research topics in the area of systems and control community. Firstly, we give the basic formulation and analysis of noncooperative games with continuous action spaces, and provide the motivation and basic setting for distributed Nash equilibrium seeking. Then we introduce both the gradient-based algorithms and best-response based algorithms for various type of games, including zero-sum games, aggregative games, potential games, monotone games, and multi-cluster games. In addition, we provide some applications of noncooperative games.

Well control optimization in waterflooding using genetic algorithm coupled with Artificial Neural Networks

Optimum well controls to maximize net present value (NPV) in a waterflooding operation are often obtained from an iterative process of employing numerical reservoir simulation and optimization algorithms. It is often challenging to implement gradient-based optimization algorithms because of the large number of variables and the complexities to embed the optimization algorithm in the simulator solving workflow. Approaches based on repeated model evaluation are easier to implement but are often time-consuming and computationally expensive.

A Dielectric Parameters Reconstruction Algorithm for 2-D Plasma Media

In this article, a parameter reconstruction algorithm for 2-D plasma is proposed based on the time-domain Maxwell’s equations. The main contributions of this article are as follows: based on the Lagrange multiplier method (LMM), the partial derivatives of the resonance frequency and collision frequency are derived so that the plasma parameters can be easily reconstructed by gradient-based optimization algorithm. Unlike previous plasma parameter reconstruction algorithms that mainly focus on collisionless plasma, our algorithm also considers the impact of collision frequency and can reconstruct the distribution of collision frequencies. To further verify the stability and effectiveness of the proposed algorithm, the effects of noise and regularization terms are also analyzed. The LMM is used to obtain the augment cost function and adjoint equations. The Ploak–Ribiere conjugate gradient method (PRCGM) is used to update the parameters, such as resonance frequency, of the unknown plasma region. What is more, the finite-difference time-domain (FDTD) method is used to calculate the electromagnetic field and Lagrange multipliers value. Besides, the Thiknov regularization term is employed to improve the ill-posedness. Three numerical examples are given to indicate the accuracy and effectiveness of the proposed 2-D plasma dielectric parameters reconstruction algorithm. This method can provide a useful reference for the reconstruction of plasma distribution on the ionosphere and the surface of high-speed aircraft.

Stackelberg-Game-Based Computation Offloading Method in Cloud–Edge Computing Networks

Offloading computation tasks through cloud–edge collaboration has been a promising way to improve the Quality of Service (QoS) of applications. Usually, cloud server (CS) and edge server (ES) are selfish and rational and, therefore, it is imperative to develop incentive mechanisms, which can encourage idle ESs or the CS to participate in the task offloading process. In this article, we propose a computation offloading method based on the game theory, which is suitable for cloud–edge computing networks. It is considered that the CS has a lot of computation tasks to conduct, and ESs usually have idle computational resources. The CS can offload computation tasks to ESs with idle computational resources to reduce its own cost and pressure, and ESs can profit by selling their computational resources. The interaction between the CS and ESs is modeled as a Stackelberg game, and the proposed game is analyzed by using the backward induction method. It is proved that the game can achieve a unique Nash equilibrium. Then, a gradient-based iterative search algorithm (GISA) is proposed to obtain the optimal solution in order to maximize the utility of the CS and ESs. Finally, numerical simulation results show that our proposed method greatly outperforms other benchmark schemes under different scenarios, and can encourage ESs to trade their computational resources with the CS effectively.

Sequential versus concurrent gradient-based optimal algorithms for the robust control of quantum systems

We investigate two different formulations of gradient-based algorithms for the robust control of quantum systems. We consider the simultaneous control of an ensemble of systems which differ by the value of a constant Hamiltonian parameter. The two versions of the iterative algorithm, called concurrent and sequential, correspond respectively to a joint update of the control at each iteration for all the elements of the ensemble or to a successive correction of the control in which the control law is different for each system. We analyze the relative efficiency of the two optimization procedures on two benchmark examples, namely the control of two-level quantum systems and Bose–Einstein condensates in a one-dimensional optical lattice. Intensive numerical simulations show the superiority of the sequential-update formulation with respect to the concurrent one for a similar numerical cost.

Two-Scale Topology Optimization with Isotropic and Orthotropic Microstructures

Advances in additive manufacturing enable the fabrication of complex structures with intricate geometric details, which bring opportunities for high-resolution structure design and transform the potential of functional product development. However, the increasingly delicate designs bring computational challenges for structural optimization paradigms, such as topology optimization (TO), since the design dimensionality increases with the resolutions. Two-scale TO paves an avenue for high-resolution structural design to alleviate this challenge. This paper investigates the efficacy of introducing function-based microstructures into the two-scale TO. Both isotropic and orthotropic microstructure are considered to develop this TO framework. Implicit functions are exploited to model the two classes of cellular materials, including triply periodic minimal surfaces (TPMS) and Fourier series-based functions (FSF). The elasticity tensor of microstructures is computed with numerical homogenization. Then, a two-scale TO paradigm is formulated, and a gradient-based algorithm is proposed to simultaneously optimize the micro-scale structures and macro-scale material properties. Several engineering benchmark cases are tested with the proposed method, and experimental results reveal that using proposed microstructures leads to, at most, a 36% decrease in the compliance of optimal structures. The proposed framework provides achievable directionality and broader design flexibility for high-resolution product development.

Maximum weighted likelihood estimator for robust heavy-tail modelling of finite mixture models

Insurance claim severity data are characterized by complex distributional phenomenons, where flexible density estimation tools such as the finite mixture models (FMM) are necessary. However, maximum likelihood estimations (MLE) often produce unstable tail estimates for the FMM. Motivated by this challenge, this article presents a maximum weighted likelihood estimator (MWLE) for robust estimations of heavy-tailed FMM. Under some regularity conditions, the proposed MWLE is consistent and asymptotically normal. Since the MWLE has a probabilistic interpretation, we are able to develop two distinctive versions of the Generalized Expectation-Maximization (GEM) algorithm to estimate the MWLE parameters more efficiently and reliably than the standard gradient-based algorithms. We apply the proposed MWLE to two simulation studies and a real motor insurance dataset to demonstrate that it better extrapolates the extreme losses than the MLE, without sacrificing the flexibility of the FMM in capturing the small attritional claims.

Microstructure design using a human computation game

The design of microstructures that are optimized for a given engineering applications requires exploration of a rough and high-dimensional configuration space. Gradient-based algorithms are efficient, but suffer from a propensity to get stuck in local minima. Global-optimization algorithms are better at finding global minima, but are generally slow to converge. We developed and tested a Human Computation Game (HCG) for microstructure design where players interactively manipulate the microstructure to optimize an effective macroscopic material property. We investigate the impact of various game mechanics on solution quality and efficiency, and compare the HCG player solutions to those of a traditional global optimization algorithm—Simulated Annealing (SA). We show that organizing players into Synchronous teams performed better on more complex problems on average than players working Asynchronously or Solo. We also show that in the best cases, players can find microstructures that outperform those obtained by SA by up to 25% using the same number of computations, or achieve the same performance using up to 307 times fewer computational steps. By studying the optimization strategies employed by HCG players, we anticipate that improved optimization algorithms for microstructure design (and other configurational optimization problems) can be developed.

A novel CS-RBFs-based parameterization scheme for the optimization design of curvilinear variable-stiffness composites with manufacturing constraints

Recent advancements in additive manufacturing technology have made it convenient to design and fabricate composites with curvilinear fibers and the desired performance. For the optimization design of fiber-reinforced composite, two key issues are the description of spatially varying fiber angles and the mathematical representation of the manufacturing constraints. A novel parameterization scheme based on the compactly supported radial basis functions (CS-RBFs) is established to describe the spatially varying fiber angle vector field. Its main characters, including the nonnegativity, partition of unity, and differentiability, inherently ensure the fiber continuity, the bounds of the design variables and make it compatible with the gradient-based optimization design algorithm. Besides, a mathematical formulation is developed to construct the relationships between two common manufacturing constraints (minimum turning radius and gap/overlap) and the curl and divergence of the vector field. By the proposed parameterization scheme and p-norm aggregation function, these manufacturing constraints can be easily described in all elements and added to the optimization. Several numerical examples are conducted to investigate the influence of the varying support points, the radius of the CS-RBFs, initial design, load condition, norm parameter and the excitation frequency on the final design. The optimized results demonstrate the effectiveness of the proposed method.

On the relaxed gradient-based iterative methods for the generalized coupled Sylvester-transpose matrix equations

In this paper, we propose the full-rank and reduced-rank relaxed gradient-based iterative algorithms for solving the generalized coupled Sylvester-transpose matrix equations. We provide analytically the necessary and sufficient condition for the convergence of the proposed iterative algorithm and give explicitly the optimal step size such that the convergence rate of the algorithm is maximized. Some numerical examples are examined to confirm the feasibility and efficiency of the proposed algorithms.

Multiscale design of nonlinear materials using reduced-order modeling

A two-step optimization method is proposed for the efficient multiscale design of heterogeneous materials whose nonlinear macroscopic response is driven by volumetric and interfacial damage taking place at the microstructural level. The Eigendeformation-based reduced-order Homogenization Method (EHM) is used in a reduced-order design phase, allowing for a preliminary design that combines multiple initial configurations with a gradient-based optimization method. The preliminary design is then adopted to initialize a high-fidelity optimization phase based on the Interface-Enriched Generalized Finite Element Method (IGFEM) to arrive at the final design. The analytic sensitivities of EHM with respect to nonlinear material properties are derived to drive the gradient-based algorithm. Several multiscale optimization problems of increasing complexity are presented to illustrate the efficiency of the proposed method using an objective function based on stress error relative to a desired stress–strain response.

Outage Probability and Finite-SNR DMT Analysis for IRS-Aided MIMO Systems: How Large IRSs Need to be?

Intelligent reflecting surfaces (IRSs) are promising enablers for high-capacity wireless communication systems by constructing favorable channels between the transmitter and receiver. However, general, accurate, and tractable outage probability analysis for IRS-aided multiple-input-multiple-output (MIMO) systems is not available in the literature. In this paper, we first characterize the distribution of the mutual information (MI) for IRS-aided MIMO systems by capitalizing on large random matrix theory (RMT). Based on this result, a closed-form approximation for the outage probability is derived and a gradient-based algorithm is proposed to minimize the outage probability with statistical channel state information (CSI). We also investigate the diversity-multiplexing tradeoff (DMT) with finite signal-to-noise ratio (SNR). Based on these theoretical results, we further study the impact of the IRS size on system performance. In the high SNR regime, we provide closed-form expressions for the ergodic mutual information (EMI) and outage probability as a function of the IRS size, which analytically reveal that the benefit of increasing the IRS size saturates quickly. Simulation results validate the accuracy of the theoretical analysis and confirm the increasing cost to improve system performance by deploying larger IRSs. For example, for an IRS-aided MIMO system with 20 antennas at both the transmitter and receiver, we need to double the size of the IRS to increase the throughput from 90% to 95% of its maximum value.

Two-Stage Stochastic Program for Dynamic Coordinated Traffic Control Under Demand Uncertainty

This study develops a cell-based two-stage stochastic program to address the dynamic, spatial and stochastic characteristics of traffic flow for arterial adaptive signal control. To capture demand uncertainty, we formulate the adaptive coordinated traffic signal control as a two-stage stochastic program. To capture dynamic and spatial features of traffic flow, Cell Transmission Model (CTM) is embedded in the two-stage formulation. We incorporate the concept of Phase Clearance Reliability (PCR) to decompose the original two-stage stochastic formulation into separable sub-problems, which greatly enhances solution efficiency. A gradient-based solution algorithm is developed to solve the problem. Numerical examples are constructed to investigate the importance of capturing (or ignoring) each of the dynamic, spatial and stochastic features for traffic control. The results show that failure to account for any of these three traffic flow features will incur a certain extent of delay performance degradation, especially for heavy traffic. Finally, this study validates the findings through VISSIM, with promising results for the newly developed stochastic formulation.

Distributed LMMSE Estimation for Large-Scale Systems Based on Local Information.

This article studies the distributed linear minimum mean square error (LMMSE) estimation problem for large-scale systems with local information (LSLI). Large-scale systems are composed of numerous subsystems. Each subsystem only transmits information to its neighbors. Thus, only the local information is available to each subsystem. This implies that the information available to different subsystems is different. Using local information to design an LMMSE estimator, the gains of the estimator must satisfy the sparse structure constraint, which makes the estimator design challenging and complicates the boundedness analysis of the estimation error covariance (EEC). In this article, a framework of the distributed LMMSE estimation for LSLI is established. The gains of the LMMSE estimator are effectively constructed by solving linear matrix equations. A gradient descent algorithm is exploited to design the gains of the LMMSE estimator numerically. Sufficient conditions are derived to ensure the boundedness of the EEC. Also, a gradient-based search algorithm is developed to verify whether the sufficient conditions hold or not. Finally, an example is used to illustrate the effectiveness of the proposed results.