Divide-and-conquer Approach Research Articles

Divide and conquer for accelerated failure time model with massive time‐to‐event data

AbstractBig data present new theoretical and computational challenges as well as tremendous opportunities in many fields. In health care research, we develop a novel divide‐and‐conquer (DAC) approach to deal with massive and right‐censored data under the accelerated failure time model, where the sample size is extraordinarily large and the dimension of predictors is large but smaller than the sample size. Specifically, we construct a penalized loss function by approximating the weighted least squares loss function by combining estimation results without penalization from all subsets. The resulting adaptive LASSO penalized DAC estimator enjoys the oracle property. Simulation studies demonstrate that the proposed DAC procedure performs well and also reduces the computation time with satisfactory performance compared with estimation results using the full data. Our proposed DAC approach is applied to a massive dataset from the Chinese Longitudinal Healthy Longevity Survey.

Regularized Localized Molecular Orbitals in a Divide-and-Conquer Approach for Linear Scaling Calculations.

Non-orthogonal localized molecular orbitals (NOLMOs) have been employed as building blocks for the divide-and-conquer (DC) linear scaling method. The NOLMOs are calculated from subsystems and used for constructing the density matrix (DM) of the entire system, instead of the subsystem DM in the original DC approach. Also, unlike the original DC method, the inverse electronic temperature parameter β is not needed anymore. Furthermore, a new regularized localization approach for NOLMOs has been developed, in which the localization cost function is a sum of the spatial spread function, as in the Boys method, and the kinetic energy, as a regularization measure to limit the oscillation of the NOLMOs. The optimal weight of the kinetic energy can be determined by optimization with analytical gradients. The resulting regularized NOLMOs have enhanced smoothness and better transferability because of reduced kinetic energies. Compared with the original DC, while NOLMO-DC has a similar computational linear scaling cost, the accuracy of NOLMO-DC is better by several orders of magnitude for large conjugated systems and by about 1 order of magnitude for other systems. The NOLMO-DC method is thus a promising development of the DC approach for linear scaling calculations.

A divide-and-conquer optimization paradigm for waterflooding production optimization

Production optimization technique, as a crucial step in the closed-loop reservoir management (CLRM), aims to achieve optimal development efficiency by adjusting development schemes (e.g., well-controls) with the aid of optimization methods. However, due to the unbearable computational burden brought by full-scale reservoir simulation, few optimizers can obtain satisfactory solution(s) within limited simulation calls, especially when the problem dimension is very high. This phenomenon is common in many real-world scenarios, which is also referred to as the “curse of dimensionality”. To address this issue, a novel divide-and-conquer (DAC) optimization paradigm is proposed for production optimization problems. Specifically, given a large-scale production optimization problem, it can be decomposed into a number of simpler subproblems with low dimensions. Then, to overcome the computationally expensive issue, multiple data-driven surrogates are built for the subproblems. Finally, all the subproblem surrogates are optimized cooperatively using a reuse strategy of subproblem samples. From the perspective of production optimization, the joint scheme optimization of the original problem is turned into cooperatively optimizing the schemes involved in multiple subproblems. Interestingly, the obtained subproblems always correspond to multiple flow units with weak flow interferences caused by some obstruction factors (e.g., low-permeability channel and vertical barrier layer). This indicates that the DAC method can not only serve as an optimization enhancement technique but also can be employed as an auxiliary means of connectivity analysis. In return, many connectivity analysis methods such as flow diagnostics that require fewer simulation calls can serve as the decomposition tool. More importantly, the superior flexibility of the proposed DAC-based expensive optimization framework allows it to incorporate a wide variety of state-of-the-art surrogate-assisted evolutionary solvers. In this paper, the differential evolution (DE) and two advanced surrogate-assisted evolutionary solvers are implemented under the proposed paradigm. The experimental results conducted on two 100-dimensional benchmark functions and two production optimization tasks verified the effectiveness of the proposed method.

Loop Current SSH Forecasting: A New Domain Partitioning Approach for a Machine Learning Model

A divide-and-conquer (DAC) machine learning approach was first proposed by Wang et al. to forecast the sea surface height (SSH) of the Loop Current System (LCS) in the Gulf of Mexico. In this DAC approach, the forecast domain was divided into non-overlapping partitions, each of which had their own prediction model. The full domain SSH prediction was recovered by interpolating the SSH across each partition boundaries. Although the original DAC model was able to predict the LCS evolution and eddy shedding more than two months and three months in advance, respectively, growing errors at the partition boundaries negatively affected the model forecasting skills. In the study herein, a new partitioning method, which consists of overlapping partitions is presented. The region of interest is divided into 50%-overlapping partitions. At each prediction step, the SSH value at each point is computed from overlapping partitions, which significantly reduces the occurrence of unrealistic SSH features at partition boundaries. This new approach led to a significant improvement of the overall model performance both in terms of features prediction such as the location of the LC eddy SSH contours but also in terms of event prediction, such as the LC ring separation. We observed an approximate 12% decrease in error over a 10-week prediction, and also show that this method can approximate the location and shedding of eddy Cameron better than the original DAC method.

Energy-based automatic determination of buffer region in the divide-and-conquer second-order Møller-Plesset perturbation theory.

In the linear‐scaling divide‐and‐conquer (DC) electronic structure method, each subsystem is calculated together with the neighboring buffer region, the size of which affects the energy error introduced by the fragmentation in the DC method. The DC self‐consistent field calculation utilizes a scheme to automatically determine the appropriate buffer region that is as compact as possible for reducing the computational time while maintaining acceptable accuracy (J. Comput. Chem. 2018, 39, 909). To extend the automatic determination scheme of the buffer region to the DC second‐order Møller–Plesset perturbation (MP2) calculation, a scheme for estimating the subsystem MP2 correlation energy contribution from each atom in the buffer region is proposed. The estimation is based on the atomic orbital Laplace MP2 formalism. Based on this, an automatic buffer determination scheme for the DC‐MP2 calculation is constructed and its performance for several types of systems is assessed.

A divide-and-conquer method for sparse risk prediction and evaluation.

Divide-and-conquer (DAC) is a commonly used strategy to overcome the challenges of extraordinarily large data, by first breaking the dataset into series of data blocks, then combining results from individual data blocks to obtain a final estimation. Various DAC algorithms have been proposed to fit a sparse predictive regression model in the $L_1$ regularization setting. However, many existing DAC algorithms remain computationally intensive when sample size and number of candidate predictors are both large. In addition, no existing DAC procedures provide inference for quantifying the accuracy of risk prediction models. In this article, we propose a screening and one-step linearization infused DAC (SOLID) algorithm to fit sparse logistic regression to massive datasets, by integrating the DAC strategy with a screening step and sequences of linearization. This enables us to maximize the likelihood with only selected covariates and perform penalized estimation via a fast approximation to the likelihood. To assess the accuracy of a predictive regression model, we develop a modified cross-validation (MCV) that utilizes the side products of the SOLID, substantially reducing the computational burden. Compared with existing DAC methods, the MCV procedure is the first to make inference on accuracy. Extensive simulation studies suggest that the proposed SOLID and MCV procedures substantially outperform the existing methods with respect to computational speed and achieve similar statistical efficiency as the full sample-based estimator. We also demonstrate that the proposed inference procedure provides valid interval estimators. We apply the proposed SOLID procedure to develop and validate a classification model for disease diagnosis using narrative clinical notes based on electronic medical record data from Partners HealthCare.

Efficient Implementation of Iterative Polynomial Matrix EVD Algorithms Exploiting Structural Redundancy and Parallelisation

A number of algorithms are capable of iteratively calculating a polynomial matrix eigenvalue decomposition (PEVD), which is a generalisation of the EVD and will diagonalise a parahermitian polynomial matrix via paraunitary operations. While offering promising results in various broadband array processing applications, the PEVD has seen limited deployment in hardware due to the high computational complexity of these algorithms. Akin to low complexity divide-and-conquer (DaC) solutions to eigenproblems, this paper addresses a partially parallelisable DaC approach to the PEVD. A novel algorithm titled parallel-sequential matrix diagonalisation exhibits significantly reduced algorithmic complexity and run-time when compared with existing iterative PEVD methods. The DaC approach, which is shown to be suitable for multi-core implementation, can improve eigenvalue resolution at the expense of decomposition mean squared error, and offers a trade-off between the approximation order and accuracy of the resulting paraunitary matrices.

A fast divide-and-conquer sparse Cox regression.

We propose a computationally and statistically efficient divide-and-conquer (DAC) algorithm to fit sparse Cox regression to massive datasets where the sample size $n_0$ is exceedingly large and the covariate dimension $p$ is not small but $n_0\gg p$. The proposed algorithm achieves computational efficiency through a one-step linear approximation followed by a least square approximation to the partial likelihood (PL). These sequences of linearization enable us to maximize the PL with only a small subset and perform penalized estimation via a fast approximation to the PL. The algorithm is applicable for the analysis of both time-independent and time-dependent survival data. Simulations suggest that the proposed DAC algorithm substantially outperforms the full sample-based estimators and the existing DAC algorithm with respect to the computational speed, while it achieves similar statistical efficiency as the full sample-based estimators. The proposed algorithm was applied to extraordinarily large survival datasets for the prediction of heart failure-specific readmission within 30 days among Medicare heart failure patients.

A Sharper Generalization Bound for Divide-and-Conquer Ridge Regression

We study the distributed machine learning problem where the n feature-response pairs are partitioned among m machines uniformly at random. The goal is to approximately solve an empirical risk minimization (ERM) problem with the minimum amount of communication. The divide-and-conquer (DC) method, which was proposed several years ago, lets every worker machine independently solve the same ERM problem using its local feature-response pairs and the driver machine combine the solutions. This approach is in one-shot and thereby extremely communication-efficient. Although the DC method has been studied by many prior works, reasonable generalization bound has not been established before this work.For the ridge regression problem, we show that the prediction error of the DC method on unseen test samples is at most ε times larger than the optimal. There have been constantfactor bounds in the prior works, their sample complexities have a quadratic dependence on d, which does not match the setting of most real-world problems. In contrast, our bounds are much stronger. First, our 1 + ε error bound is much better than their constant-factor bounds. Second, our sample complexity is merely linear with d.

An efficient hybrid tridiagonal divide-and-conquer algorithm on distributed memory architectures

In this paper, we propose an efficient divide-and-conquer (DC) algorithm for symmetric tridiagonal matrices based on ScaLAPACK and the hierarchically semiseparable (HSS) matrices. HSS is an important type of rank-structured matrices. The most computationally intensive part of the DC algorithm is computing the eigenvectors via matrix–matrixmultiplications (MMM). In our parallel hybrid DC (PHDC) algorithm, MMM is accelerated by using HSS matrix techniques when the intermediate matrix is large. All the HSS computations are performed via the package STRUMPACK . PHDC has been tested by using many different matrices. Compared with the DC implementation in MKL, PHDC can be faster for some matrices with few deflations when using hundreds of processes. However, the gains decrease as the number of processes increases. The comparisons of PHDC with ELPA (the Eigenvalue soLvers for Petascale Applications library) are similar. PHDC is usually slower than MKL and ELPA when using 300 or more processes on the Tianhe-2 supercomputer.

Automated error control in divide-and-conquer self-consistent field calculations.

In linear-scaling divide-and-conquer (DC) electronic structure calculations, a buffer region is used to control the error introduced by the DC approximation. In this study, an energy-based error estimation scheme is proposed for the DC self-consistent field method with a two-layer buffer region scheme. Based on this scheme, a procedure to automatically determine the appropriate buffer region in the DC method is proposed. It was confirmed that the present method works satisfactorily in calculations of water clusters and proteins, although its performance was insufficient for the calculation of a delocalized graphene system. © 2018 Wiley Periodicals, Inc.

Three pillars for achieving quantum mechanical molecular dynamics simulations of huge systems: Divide-and-conquer, density-functional tight-binding, and massively parallel computation.

The linear-scaling divide-and-conquer (DC) quantum chemical methodology is applied to the density-functional tight-binding (DFTB) theory to develop a massively parallel program that achieves on-the-fly molecular reaction dynamics simulations of huge systems from scratch. The functions to perform large scale geometry optimization and molecular dynamics with DC-DFTB potential energy surface are implemented to the program called DC-DFTB-K. A novel interpolation-based algorithm is developed for parallelizing the determination of the Fermi level in the DC method. The performance of the DC-DFTB-K program is assessed using a laboratory computer and the K computer. Numerical tests show the high efficiency of the DC-DFTB-K program, a single-point energy gradient calculation of a one-million-atom system is completed within 60 s using 7290 nodes of the K computer. © 2016 Wiley Periodicals, Inc.

An improved divide-and-conquer algorithm for the banded matrices with narrow bandwidths

In this paper we propose a novel divide-and-conquer (DC) algorithm to compute the SVD of banded matrices, and further accelerate it by using rank-structured matrix techniques, especially the hierarchically semiseparable (HSS) matrix. The DC algorithm for the symmetric banded eigenvalue problem can also be accelerated similarly. For matrices with few deflations, the banded DC algorithms require more flops than the classical DC algorithm, and thus they are suitable for narrowly banded matrices. While, if there exist many deflations, the banded DC algorithms can be faster than the classical ones for matrices with relatively large bandwidths. Numerous experiments have been done to test the proposed algorithms. Some of the tested matrices are from construction and some are from real applications. Comparing with the DC algorithm in Intel MKL, our proposed algorithms can be hundreds times faster for matrices with narrow bandwidths or many deflations.

A Divide-and-Conquer Algorithm for Rigging Elections Problem

A Divide-and-Conquer Algorithm for Rigging Elections Problem

Extended Divide-and-Conquer Algorithm for Uncertainty Analysis of Multibody Systems in Polynomial Chaos Expansion Framework

In this paper, an advanced algorithm is presented to efficiently form and solve the equations of motion of multibody problems involving uncertainty in the system parameters and/or excitations. Uncertainty is introduced to the system through the application of polynomial chaos expansion (PCE). In this scheme, states of the system, nondeterministic parameters, and constraint loads are projected onto the space of specific orthogonal base functions through modal values. Computational complexity of traditional methods of forming and solving the resulting governing equations drastically grows as a cubic function of the size of the nondeterministic system which is significantly larger than the original deterministic multibody problem. In this paper, the divide-and-conquer algorithm (DCA) will be extended to form and solve the nondeterministic governing equations of motion avoiding the construction of the mass and Jacobian matrices of the entire system. In this strategy, stochastic governing equations of motion of each individual body as well as those associated with kinematic constraints at connecting joints are developed in terms of base functions and modal terms. Then using the divide-and-conquer scheme, the entire system is swept in the assembly and disassembly passes to recursively form and solve nondeterministic equations of motion for modal values of spatial accelerations and constraint loads. In serial and parallel implementations, computational complexity of the method is expected to, respectively, increase as a linear and logarithmic function of the size.

Efficient Divide-and-Conquer Classification Based on Parallel Feature-Space Decomposition for Distributed Systems

This study presents a divide-and-conquer (DC) approach based on feature space decomposition for classification. When large-scale datasets are present, typical approaches usually employed truncated kernel methods on the feature space or DC approaches on the sample space. However, this did not guarantee separability between classes, owing to overfitting. To overcome such problems, this work proposes a novel DC approach on feature spaces consisting of three steps. Firstly, we divide the feature space into several subspaces using the decomposition method proposed in this paper. Subsequently, these feature subspaces are sent into individual local classifiers for training. Finally, the outcomes of local classifiers are fused together to generate the final classification results. Experiments on large-scale datasets are carried out for performance evaluation. The results show that the error rates of the proposed DC method decreased comparing with the state-of-the-art fast SVM solvers, e.g., reducing error rates by 10.53% and 7.53% on RCV1 and covtype datasets respectively.

A divide-and-conquer approach to compute collision cross sections in the projection approximation method

The prevalent method to compute collision cross sections of large molecules is the projection approximation (PA) method that involves Monte Carlo (MC) integration of molecular projections on randomly chosen planes. Here we propose a new strategy to compute these projections based on a divide-and-conquer (DC) strategy. It is demonstrated that the DC method is faster and results in more accurate molecular projections than MC integration for large biomolecules using similar integration criteria. A new software tool (CCS) is presented for efficient computation of collision cross sections in the PA method.

New and Extended Applications of the Divide-and-Conquer Algorithm for Multibody Dynamics

This work presents a survey of the current and ongoing research by the authors who use the divide-and-conquer algorithm (DCA) to reduce the computational burden associated with various aspects of multibody dynamics. This work provides a brief discussion of various topics that are extensions of previous DCA-based algorithms or novel uses of this algorithm in the multibody dynamics context. These topics include constraint error stabilization, spline-based modeling of flexible bodies, model fidelity transitions for flexible-body systems, and large deformations of flexible bodies. It is assumed that the reader is familiar with the “Advances in the Application of the DCA to Multibody System Dynamics” text as the notation used in this work is explained therein and provides a summary of how the DCA has been used previously.

Advances in the Application of the Divide-and-Conquer Algorithm to Multibody System Dynamics

This paper summarizes the various recent advancements achieved by utilizing the divide-and-conquer algorithm (DCA) to reduce the computational burden associated with many aspects of modeling, designing, and simulating articulated multibody systems. This basic algorithm provides a framework to realize O(n) computational complexity for serial task scheduling. Furthermore, the framework of this algorithm easily accommodates parallel task scheduling, which results in coarse-grain O(log n) computational complexity. This is a significant increase in efficiency over forming and solving the Newton–Euler equations directly. A survey of the notable previous work accomplished, though not all inclusive, is provided to give a more complete understanding of how this algorithm has been used in this context. These advances include applying the DCA to constrained systems, flexible bodies, sensitivity analysis, contact, and hybridization with other methods. This work reproduces the basic mathematical framework for applying the DCA in each of these applications. The reader is referred to the original work for the details of the discussed methods.

An Accelerated Divide-and-Conquer Algorithm for the Bidiagonal SVD Problem

In this paper, aiming at solving the bidiagonal SVD problem, a classical divide-and-conquer (DC) algorithm is modified, which needs to compute the SVD of broken arrow matrices by solving secular equations. The main cost of DC lies in the updating of singular vectors, which involves two matrix-matrix multiplications. We find that the singular vector matrices of a broken arrow matrix are Cauchy-like matrices and have an off-diagonal low-rank property, so they can be approximated efficiently by hierarchically semiseparable (HSS) matrices. Hereby, by using the HSS techniques, the complexity of computing singular vectors can be reduced significantly. An accelerated DC algorithm is proposed, denoted by ADC. Furthermore, we use a structured low-rank approximation method to construct these HSS approximations. Numerous experiments show ADC is both fast and numerically stable. When dealing with large matrices with few deflations, ADC can be 3x faster than DC in the optimized LAPACK libraries such as Intel MKL without any degradation in accuracy. These techniques can be used to similarly solve the symmetric tridiagonal eigenvalue problem. (A corrected PDF is attached.)