Speed Up Convergence Research Articles

India–Eurasia convergence speed-up by passive-margin sediment subduction

The fast increase of convergence rate between India and Eurasia around 65 million years ago (Ma)—from approximately 8 cm yr−1 to a peak rate of approximately 18 cm yr−1—remains a complex geological event to explain1–8, given the inherent uncertainty surrounding the tectonic history and the intricate interplay of forces influencing plate speed9–11. Here we use a combination of geochemical analysis and geodynamic modelling to propose that this rapid convergence can be explained by sediment subduction derived from the northern Indian passive margin. Through isotope and trace element analysis, we find an enhanced contribution of terrigenous sediment melt to the mantle source of the Gangdese magmatic rocks around 65 Ma, concurrent with the acceleration of India–Eurasia convergence. Numerical experiments suggest that subduction of sediments more than 1 km thick covering an approximately 1,000-km-wide ocean basin abutting the northern Indian passive margin starting from 65 Ma could have spurred the increased convergence rate and further led to significant crustal extension, consistent with empirical observations. Our study implies that the acceleration of India–Eurasia convergence marks the arrival of passive-margin-derived sediments, constraining the initial India–Eurasia collision to be around 60 Ma. It further suggests that temporary accelerations in subduction rates might be a common feature at the final stage of continental assembly.

Comparison and analysis of new curriculum criteria for end-to-end ASR

Traditionally, teaching a human and a Machine Learning (ML) model is quite different, but organized and structured learning has the ability to enable faster and better understanding of the underlying concepts. For example, when humans learn to speak, they first learn how to utter basic phones and then slowly move towards more complex structures such as words and sentences. Motivated by this observation, researchers have started to adapt this approach for training ML models. Since the main concept, the gradual increase in difficulty, resembles the notion of the curriculum in education, the methodology became known as Curriculum Learning (CL). In this work, we design and test new CL approaches to train Automatic Speech Recognition systems, specifically focusing on the so-called end-to-end models. These models consist of a single, large-scale neural network that performs the recognition task, in contrast to the traditional way of having several specialized components focusing on different subtasks (e.g., acoustic and language modeling). We demonstrate that end-to-end models can achieve better performances if they are provided with an organized training set consisting of examples that exhibit an increasing level of difficulty. To impose structure on the training set and to define the notion of an easy example, we explored multiple solutions that use either external, static scoring methods or incorporate feedback from the model itself. In addition, we examined the effect of pacing functions that control how much data is presented to the network during each training epoch. Our proposed curriculum learning strategies were tested on the task of speech recognition on two data sets, one containing spontaneous Finnish speech where volunteers were asked to speak about a given topic, and one containing planned English speech. Empirical results showed that a good curriculum strategy can yield performance improvements and speed-up convergence. After a given number of epochs, our best strategy achieved a 5.6% and 3.4% decrease in terms of test set word error rate for the Finnish and English data sets, respectively.

Markov-chain Monte Carlo method enhanced by a quantum alternating operator ansatz

Quantum computation is expected to accelerate certain computational tasks over classical counterparts. Its most primitive advantage is its ability to sample from classically intractable probability distributions. A promising approach to make use of this fact is the so-called quantum-enhanced Markov-chain Monte Carlo (qe-MCMC) method [D. Layden , ], which uses outputs from quantum circuits as the proposal distributions. In this paper, we propose the use of a quantum alternating operator ansatz (QAOA) for qe-MCMC and provide a strategy to optimize its parameters to improve convergence speed while keeping its depth shallow. The proposed QAOA-type circuit is designed to satisfy the specific constraint which qe-MCMC requires with arbitrary parameters. Through our extensive numerical analysis, we find a correlation in a certain parameter range between an experimentally measurable value, acceptance rate of MCMC, and the spectral gap of the MCMC transition matrix, which determines the convergence speed. This allows us to optimize the parameter in the QAOA circuit and achieve quadratic speedup in convergence. Since MCMC is used in various areas such as statistical physics and machine learning, this paper represents an important step toward realizing practical quantum advantage with currently available quantum computers through qe-MCMC. Published by the American Physical Society 2024

Distributed Discrete-Time Convex Optimization With Closed Convex Set Constraints: Linearly Convergent Algorithm Design.

The convergence rate and applicability to directed graphs with interaction topologies are two important features for practical applications of distributed optimization algorithms. In this article, a new kind of fast distributed discrete-time algorithms is developed for solving convex optimization problems with closed convex set constraints over directed interaction networks. Under the gradient tracking framework, two distributed algorithms are, respectively, designed over balanced and unbalanced graphs, where momentum terms and two time-scales are involved. Furthermore, it is demonstrated that the designed distributed algorithms attain linear speedup convergence rates provided that the momentum coefficients and the step size are appropriately selected. Finally, numerical simulations verify the effectiveness and the global accelerated effect of the designed algorithms.

DePRL: Achieving Linear Convergence Speedup in Personalized Decentralized Learning with Shared Representations

Decentralized learning has emerged as an alternative method to the popular parameter-server framework which suffers from high communication burden, single-point failure and scalability issues due to the need of a central server. However, most existing works focus on a single shared model for all workers regardless of the data heterogeneity problem, rendering the resulting model performing poorly on individual workers. In this work, we propose a novel personalized decentralized learning algorithm named DePRL via shared representations. Our algorithm relies on ideas from representation learning theory to learn a low-dimensional global representation collaboratively among all workers in a fully decentralized manner, as well as a user-specific low-dimensional local head leading to a personalized solution for each worker. We show that DePRL achieves, for the first time, a provable \textit{linear speedup for convergence} with general non-linear representations (i.e., the convergence rate is improved linearly with respect to the number of workers). Experimental results support our theoretical findings showing the superiority of our method in data heterogeneous environments.

Convergence analysis of Schwarz waveform relaxation method to compute coupled advection–diffusion–reaction equations

We study the computation of coupled advection–diffusion–reaction equations with the same or different coefficients by the Schwarz waveform relaxation method. The study starts with linear equations and analyzes the computation’s convergence with a Dirichlet condition, a Robin condition, and a combination of them as the transmission conditions. Then, it presents an optimized algorithm for the Dirichlet condition, and this algorithm leads to a substantial speedup in the convergence. Furthermore, the optimized algorithm extends to the computation of nonlinear equations, including the viscous Burgers equation, and the algorithm largely remains effective for convergence speedup. Finally, the study compares waveform relaxation and conventional methods. Numerical experiments indicate that the former tends to be much more expensive than the latter regarding the number of times to solve the involved linear systems.

Efficient Feature Selection in High Dimensional Data Based on Enhanced Binary Chimp Optimization Algorithms and Machine Learning

Feature selection with the highest performance accuracy is the biggest win for multidimensional data. The Chimpanzee Optimization Algorithm (ChOA) serves as a crucial technique for dealing with multidimensional global optimization issues. However, ChOA often lacks fast convergence and good selection of sensitive attributes leading to poor performance. To address these issues, most significant features were selected using two variants of ChOA called BChimp1 and BChimp2 (BChimp1 and BChimp are available at : https://www.mathworks.com/matlabcentral/fileexchange/133267-binary-chimpoptimization-algorithm-forfeatures-selection. September 22, 202). BChimp1 selects the optimal solution from the four best possible solutions and it applies a stochastic crossover on four moving solutions to deeply speed-up convergence level. BChimp2 uses the sigmoid function to select the significant features. Then, these features were trained using six-well known classifiers. The proposed techniques tend to select the most significant features, speed up the convergence rate and decrease training time for high-dimensional data. 23 standard datasets with six well-known classifiers were employed to assess the performance of BChimp1 and BChimp2. Experimental results validate the efficiency of BChimp1 and BChimp2 in enhancing accuracy by 83.83% and 82.02%, and reducing dimensionality by 42.77% and 72.54%, respectively. However, time-evaluation results of BChimp1 and BChimp2 in all datasets showed fast convergence and surpassed current optimization algorithms such as PSO, GWA, GOA, and GA.

Meshless Geometric Multigrid Method for Complex Geometries with Improved Cell Coarsening Algorithm

In this paper, we present an improved multicloud methodology to extend the multicloud convergence accelerator, which depends on meshless discretization on coarse level computation of the geometric multigrid method, to cell finite volume (CFV) on three-dimensional space. To achieve this, we first identify the primary challenge in the meshless coarsening application to CFV. Specifically, the challenge lies in the poorly coarsened grid, which results in a degradation of the convergence rate. The reason for this is that the coarsening stencils were coupled with the discretization stencil. To address the issue, we proposed a new coarsening strategy on CFV that adequately utilizes node information on the fine-level mesh. The proposed methodology shows superior coarsening rates compared to the basic meshless cell coarsening, regardless of the dimension and type of mesh elements, resulting in significant speedup in convergence. Various computational fluid dynamics (CFD) simulations with three-dimensional complex geometries are presented, and their results demonstrate approximately seven times faster convergence compared to the single-grid method. The overall results verify that the multicloud method with the proposed coarsening procedure provides a powerful tool for practical CFD simulations by reducing the computation time, and it enables researchers and engineers to simulate and optimize real-world systems more efficiently.

Inverse radiative transfer with goal-oriented hp-adaptive mesh refinement: adaptive-mesh inversion

The inverse problem for radiative transfer is important in many applications, such as optical tomography and remote sensing. Major challenges include large memory requirements and computational expense, which arise from high-dimensionality and the need for iterations in solving the inverse problem. Here, to alleviate these issues, we propose adaptive-mesh inversion: a goal-oriented hp-adaptive mesh refinement method for solving inverse radiative transfer problems. One novel aspect here is that the two optimizations (one for inversion, and one for mesh adaptivity) are treated simultaneously and blended together. By exploiting the connection between duality-based mesh adaptivity and adjoint-based inversion techniques, we propose a goal-oriented error estimator, which is cheap to compute, and can efficiently guide the mesh-refinement to numerically solve the inverse problem. We use discontinuous Galerkin spectral element methods to discretize the forward and the adjoint problems. Then, based on the goal-oriented error estimator, we propose an hp-adaptive algorithm to refine the meshes. Numerical experiments are presented at the end and show convergence speed-up and reduced memory occupation by the goal-oriented mesh adaptive method.

Self-Paced Learning Based Graph Convolutional Neural Network for Mixed Integer Programming (Student Abstract)

Graph convolutional neural network (GCN) based methods have achieved noticeable performance in solving mixed integer programming problems (MIPs). However, the generalization of existing work is limited due to the problem structure. This paper proposes a self-paced learning (SPL) based GCN network (SPGCN) with curriculum learning (CL) to make the utmost of samples. SPGCN employs a GCN model to imitate the branching variable selection during the branch and bound process, while the training process is conducted in a self-paced fashion. Specifically, SPGCN contains a loss-based automatic difficulty measurer, where the training loss of the sample represents the difficulty level. In each iteration, a dynamic training dataset is constructed according to the difficulty level for GCN model training. Experiments on four NP-hard datasets verify that CL can lead to generalization improvement and convergence speedup in solving MIPs, where SPL performs better than predefined CL methods.

A Hierarchical Federated Learning Algorithm Based on Time Aggregation in Edge Computing Environment

Federated learning is currently a popular distributed machine learning solution that often experiences cumbersome communication processes and challenging model convergence in practical edge deployments due to the training nature of its model information interactions. The paper proposes a hierarchical federated learning algorithm called FedDyn to address these challenges. FedDyn uses dynamic weighting to limit the negative effects of local model parameters with high dispersion and speed-up convergence. Additionally, an efficient aggregation-based hierarchical federated learning algorithm is proposed to improve training efficiency. The waiting time is set at the edge layer, enabling edge aggregation within a specified time, while the central server waits for the arrival of all edge aggregation models before integrating them. Dynamic grouping weighted aggregation is implemented during aggregation based on the average obsolescence of local models in various batches. The proposed algorithm is tested on the MNIST and CIFAR-10 datasets and compared with the FedAVG algorithm. The results show that FedDyn can reduce the negative effects of non-independent and identically distributed (IID) data on the model and shorten the total training time by 30% under the same accuracy rate compared to FedAVG.

Graph neural networks meet with distributed graph partitioners and reconciliations

Graph neural networks (GNNs) have shown great success in various applications. As real-world graphs are large, training GNNs in distributed systems is desirable. In current training schemes, their edge partitioning strategies have a strong impact on the performance of GNNs for the unbalanced influence of high-degree nodes and the damaged neighbor integrity of low-degree nodes. Meanwhile, a lack of reconciliations of different local models leads to converging up and down across workers. In this work, we design DEPR, a suitable framework for distributed GNN training. We propose a degree-sensitive edge partitioning with influence-balancing and locality-preserving to adapt distributed GNNs training by following an owner-compute rule (each partition performs all the computations involving data that it owns). And then knowledge distillation and contrastive learning are used to reconcile the fusion of local models and boost convergence. We show in extensive empirical experiments on the node classification task of three large-scale graph datasets (Reddit, Amazon, and OGB-Products) that DEPR achieves 2x speedup of convergence and get absolute up 3.97 performance improvement of F1-micro score compared to DistDGL.

FLAME

Federated Learning (FL) enables distributed training of machine learning models while keeping personal data on user devices private. While we witness increasing applications of FL in the area of mobile sensing, such as human activity recognition (HAR), FL has not been studied in the context of a multi-device environment (MDE), wherein each user owns multiple data-producing devices. With the proliferation of mobile and wearable devices, MDEs are increasingly becoming popular in ubicomp settings, therefore necessitating the study of FL in them. FL in MDEs is characterized by being not independent and identically distributed (non-IID) across clients, complicated by the presence of both user and device heterogeneities. Further, ensuring efficient utilization of system resources on FL clients in a MDE remains an important challenge. In this paper, we propose FLAME, a user-centered FL training approach to counter statistical and system heterogeneity in MDEs, and bring consistency in inference performance across devices. FLAME features (i) user-centered FL training utilizing the time alignment across devices from the same user; (ii) accuracy- and efficiency-aware device selection; and (iii) model personalization to devices. We also present an FL evaluation testbed with realistic energy drain and network bandwidth profiles, and a novel class-based data partitioning scheme to extend existing HAR datasets to a federated setup. Our experiment results on three multi-device HAR datasets show that FLAME outperforms various baselines by 4.3-25.8% higher F1 score, 1.02-2.86x greater energy efficiency, and up to 2.06x speedup in convergence to target accuracy through fair distribution of the FL workload.

Accelerated MM Algorithms for Inference of Ranking Scores from Comparison Data

Accelerated Algorithms for RankingAssigning ranking scores to items based on observed comparison data (e.g., paired comparisons, choice, and full ranking outcomes) has been of continued interest in a wide range of applications, including information search, aggregation of social opinions, electronic commerce, online gaming platforms, and more recently, evaluation of machine learning algorithms. The key problem is to compute ranking scores, which are of interest for quantifying the strength of skills, relevancies, or preferences, and prediction of ranking outcomes. One of the most popular statistical models of ranking outcomes is the Bradley–Terry model for paired comparisons and its extensions to choice and full ranking outcomes. In “Accelerated MM Algorithms for Inference of Ranking Scores from Comparison Data,” M. Vojnovic, S.-Y. Yun, and K. Zhou show that a popular MM algorithm for inference of ranking scores for generalized Bradley–Terry ranking models suffers a slow convergence issue, and they propose a new accelerated algorithm that resolves this shortcoming and can yield substantial convergence speedups.

Zeroth-order algorithms for stochastic distributed nonconvex optimization

In this paper, we consider a stochastic distributed nonconvex optimization problem with the cost function being distributed over n agents having access only to zeroth-order (ZO) information of the cost. This problem has various machine learning applications. As a solution, we propose two distributed ZO algorithms, in which at each iteration each agent samples the local stochastic ZO oracle at two points with a time-varying smoothing parameter. We show that the proposed algorithms achieve the linear speedup convergence rate O(p/(nT)) for smooth cost functions under the state-dependent variance assumptions which are more general than the commonly used bounded variance and Lipschitz assumptions, and O(p/(nT)) convergence rate when the global cost function additionally satisfies the Polyak–Łojasiewicz (P–Ł) condition, where p and T are the dimension of the decision variable and the total number of iterations, respectively. To the best of our knowledge, this is the first linear speedup result for distributed ZO algorithms. It consequently enables systematic processing performance improvements by adding more agents. We also show that the proposed algorithms converge linearly under the relatively bounded second moment assumptions and the P–Ł condition. We demonstrate through numerical experiments the efficiency of our algorithms on generating adversarial examples from deep neural networks in comparison with baseline and recently proposed centralized and distributed ZO algorithms.

A Primal-Dual SGD Algorithm for Distributed Nonconvex Optimization

The distributed nonconvex optimization problem of minimizing a global cost function formed by a sum of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$n$</tex> local cost functions by using local information exchange is considered. This problem is an important component of many machine learning techniques with data parallelism, such as deep learning and federated learning. We propose a distributed primal-dual stochastic gradient descent (SGD) algorithm, suitable for arbitrarily connected communication networks and any smooth (possibly nonconvex) cost functions. We show that the proposed algorithm achieves the linear speedup convergence rate <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathcal{O}(1/\sqrt{nT})$</tex> for general nonconvex cost functions and the linear speedup convergence rate <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathcal{O}(1/(nT))$</tex> when the global cost function satisfies the Polyak-Łojasiewicz (P-Ł) condition, where <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$T$</tex> is the total number of iterations. We also show that the output of the proposed algorithm with constant parameters linearly converges to a neighborhood of a global optimum. We demonstrate through numerical experiments the efficiency of our algorithm in comparison with the baseline centralized SGD and recently proposed distributed SGD algorithms.

An adaptive moment estimation framework for well placement optimization

In this study, we propose the use of a first-order gradient framework, the adaptive moment estimation (Adam), in conjunction with a stochastic gradient approximation, to well location and trajectory optimization problems. The Adam framework allows the incorporation of additional information from previous gradients to calculate variable-specific progression steps. As a result, this assists the search progression to be adjusted further for each variable and allows a convergence speed-up in problems where the gradients need to be approximated. We argue that under computational budget constraints, local optimization algorithms provide suitable solutions from a heuristic initial guess. Nonlinear constraints are taken into account to ensure the proposed solutions are not in violation of practical field considerations. The performance of the proposed algorithm is compared against steepest descent and generalized pattern search, using two case studies — the placement of four vertical wells and placement of 20 nonconventional (deviated, horizontal and/or slanted) wells. The results indicate that the proposed algorithm consistently outperforms the tested methods in terms computational efficiency and final optimum value. Additional discussions regarding nonconventional parameterization provide insights into simultaneous perturbation gradient approximations.

Probabilistic Models for Common Spatial Patterns: Parameter-Expanded EM and Variational Bayes

Common spatial patterns (CSP) is a popular feature extraction method for discriminating between positive andnegative classes in electroencephalography (EEG) data.Two probabilistic models for CSP were recently developed: probabilistic CSP (PCSP), which is trained by expectation maximization (EM), and variational BayesianCSP (VBCSP) which is learned by variational approx-imation. Parameter expansion methods use auxiliaryparameters to speed up the convergence of EM or thedeterministic approximation of the target distributionin variational inference. In this paper, we describethe development of parameter-expanded algorithms forPCSP and VBCSP, leading to PCSP-PX and VBCSP-PX, whose convergence speed-up and high performanceare emphasized. The convergence speed-up in PCSP-PX and VBCSP-PX is a direct consequence of parame-ter expansion methods. The contribution of this study is the performance improvement in the case of CSP,which is a novel development. Numerical experimentson the BCI competition datasets, III IV a and IV 2ademonstrate the high performance and fast convergenceof PCSP-PX and VBCSP-PX, as compared to PCSP andVBCSP.

A hyperbolic Poisson solver for wall distance computation on irregular triangular grids

This paper presents an accurate and efficient Poisson solver for wall distance computations on irregular triangular grids. It is discussed that a typical formula of computing the wall distance from a numerical solution of a Poisson equation is not very accurate when solution contours are curved and the accuracy can be improved by a second-order formula derived by Taubin. However, the improved formula, as it requires second derivatives of the numerical solution, presents a challenge for a second-order method leading to zeroth-order accurate second-order derivatives on irregular grids. To overcome the difficulty, we develop a hyperbolic Poisson solver with gradients included as additional unknowns. In this method, the solution gradient can be obtained with second-order accuracy by a second-order discretization method, and therefore the second-derivatives can be obtained with first-order accuracy. Moreover, the hyperbolic method eliminates the numerical stiffness arising from the discretization of the Laplacian operator and achieves the ever-growing O(1/h) speed-up in iterative convergence, where h is a typical mesh spacing. Superior accuracy and efficiency of the developed method are demonstrated for irregular triangular grids.

A Time–Space Multigrid Method for Efficient Solution of the Harmonic Balance Equation System

Abstract In this article, an efficient time–space multigrid (TS-MG) method for solving the harmonic balance (HB) equation system is proposed. The principle of the time–space multigrid method is to coarsen grids in both space and time dimensions simultaneously when coarse grids are formed. The inclusion of time in the time–space multigrid is to address the instability issue or diminished convergence speedup of the spatial multigrid (S-MG) due to larger grid reduced frequencies on coarse grids. With the proposed method, the unsteady flow governing equation will still be solved on all grid levels. Comparing to the finest grid, fewer harmonics and thus fewer equations will be solved consequently on coarse grids. Discrete Fourier transform (DFT) and inverse discrete Fourier transform (IDFT) are used to achieve solution prolongation and restriction between different time grid levels. Results from the proposed method are compared with those obtained from the traditional S-MG and time domain methods. It is found that the TS-MG method can increase solution stability, reduce analysis central processing unit (CPU) time cost required for convergence, save memory usage and has no adverse effect on solution accuracy.