Stochastic Dual Coordinate Ascent Research Articles

Composite Fins Subsonic Flutter Prediction Based on Machine Learning

In the present work, the potential application of machine learning techniques in the flutter prediction of composite materials missile fins is investigated. The flutter velocity data set required for different fin aerodynamic geometries and materials is generated using a hybrid data collection method: from the wind tunnel experiments at flows ranging from 5 to 30 m/s at Re = 300,000 to 500,000, whereas synthetic data is collected using modified NACA flutter boundary model. Once the flutter data are collected, different regression algorithms were investigated, and the results were compared in terms of accuracy (when compared to the experimentally obtained results and the numerical flutter models), training time (minimization), and R-squared values (maximization). The algorithms investigated and their performance analyzed are fast forest regression, SDCA regression (stochastic dual coordinate ascent), and the light GBM (light gradient boosting machine) regression algorithm that belongs to the gradient boosting regression algorithm family. It was found that the light GBM algorithm renders the most accurate results. Based on this research, it can be concluded that artificial intelligence (machine learning) techniques can be successfully deployed in the analysis of complex flutter phenomena.

Comparison of Selected Machine Learning Algorithms in the Analysis of Mental Health Indicators

Machine learning is increasingly being used to solve clinical problems in diagnosis, therapy and care. Aim: the main aim of the study was to investigate how the selected machine learning algorithms deal with the problem of determining a virtual mental health index. Material and Methods: a number of machine learning models based on Stochastic Dual Coordinate Ascent, limited-memory Broyden–Fletcher–Goldfarb–Shanno, Online Gradient Descent, etc., were built based on a clinical dataset and compared based on criteria in the form of learning time, running time during use and regression accuracy. Results: the algorithm with the highest accuracy was Stochastic Dual Coordinate Ascent, but although its performance was high, it had significantly longer training and prediction times. The fastest algorithm looking at learning and prediction time, but slightly less accurate, was the limited-memory Broyden–Fletcher–Goldfarb–Shanno. The same data set was also analyzed automatically using ML.NET. Findings from the study can be used to build larger systems that automate early mental health diagnosis and help differentiate the use of individual algorithms depending on the purpose of the system.

Top-k Partial Label Machine.

To deal with ambiguities in partial label learning (PLL), the existing PLL methods implement disambiguations, by either identifying the ground-truth label or averaging the candidate labels. However, these methods can be easily misled by the false-positive labels in the candidate label set. We find that these ambiguities often originate from the noise caused by highly correlated or overlapping candidate labels, which leads to the difficulty in identifying the ground-truth label on the first attempt. To give the trained models more tolerance, we first propose the top-k partial loss and convex top-k partial hinge loss. Based on the losses, we present a novel top-k partial label machine (TPLM) for partial label classification. An efficient optimization algorithm is proposed based on accelerated proximal stochastic dual coordinate ascent (Prox-SDCA) and linear programming (LP). Moreover, we present a theoretical analysis of the generalization error for TPLM. Comprehensive experiments on both controlled UCI datasets and real-world partial label datasets demonstrate that the proposed method is superior to the state-of-the-art approaches.

On the Convergence of Stochastic Primal-Dual Hybrid Gradient

In this paper, we analyze the recently proposed stochastic primal-dual hybrid gradient (SPDHG) algorithm and provide new theoretical results. In particular, we prove almost sure convergence of the iterates to a solution and linear convergence with standard step sizes, independent of strong convexity constants. Our assumption for linear convergence is metric subregularity, which is satisfied for smooth and strongly convex problems in addition to many nonsmooth and/or nonstrongly convex problems, such as linear programs, Lasso, and support vector machines. In the general convex case, we prove optimal sublinear rates for the ergodic sequence and for randomly selected iterate, without bounded domain assumptions. We also provide numerical evidence showing that SPDHG with standard step sizes shows favorable and robust practical performance against its specialized strongly convex variant SPDHG-$\mu$ and other state-of-the-art algorithms including variance reduction methods and stochastic dual coordinate ascent.

A safe double screening strategy for elastic net support vector machine

Elastic net support vector machine (ENSVM) is an effective and popular classification technique. It has been widely used in many practical applications. However, solving large-scale problems still remains challenging. Inspired by its sparsity, a safe double screening rule (DSR) is proposed for accelerating ENSVM. Its main idea is to reduce the scale of the model by discarding the inactive features and samples simultaneously. In this way, the computational speed can be accelerated. Another superiority of DSR is safety, i.e., the discarded features and samples must be inactive. Its key strategy is to estimate the region containing optimal solution based on the feasible solution and duality gap. Thus, the DSR can be embedded into the process of solving the model until the algorithm converges. In addition, the safe keeping rule is constructed to identify the active features and samples. So, the DSR only needs to work on the remaining set after safe keeping. In this way, the screening process of DSR can be accelerated. Moreover, the Stochastic Dual Coordinate Ascent (SDCA) method is employed as an efficient solver. Numerical experiments on an artificial dataset and eighteen benchmark datasets demonstrate the feasibility and validity of our proposed method.

Hybrid-DCA: A double asynchronous approach for stochastic dual coordinate ascent

In prior works, stochastic dual coordinate ascent (SDCA) has been parallelized in a multi-core environment where the cores communicate through shared memory, or in a multi-processor distributed memory environment where the processors communicate through message passing. In this paper, we propose a hybrid SDCA framework for multi-core clusters, the most common high performance computing environment that consists of multiple nodes each having multiple cores and its own shared memory. We distribute data across nodes where each node solves a local problem in an asynchronous parallel fashion on its cores, and then the local updates are aggregated via an asynchronous across-node update scheme. The proposed double asynchronous method converges to a global solution for L-Lipschitz continuous loss functions, and at a linear convergence rate if a smooth convex loss function is used. Extensive empirical comparison has shown that our algorithm scales better than the best known shared-memory methods and runs faster than previous distributed-memory methods. Big datasets, such as one of 280 GB from the LIBSVM repository, cannot be accommodated on a single node and hence cannot be solved by a parallel algorithm. For such a dataset, our hybrid algorithm takes less than 30 s to achieve a duality gap of 10−5 on 16 nodes each using 12 cores, which is significantly faster than the best known distributed algorithms, such as CoCoA+, that take more than 160 s on 16 nodes.

Simultaneous Safe Feature and Sample Elimination for Sparse Support Vector Regression

Sparse support vector regression (SSVR) is an effective regression technique. It has been successfully applied to many practical problems. However, it remains challenging to handle the large-scale problems. A nice property of SSVR is double sparsity in the sense that most irrelevant features and samples have no effect on the regressor. Inspired by it, we propose a simultaneous safe feature and sample screening rule based on the strong convexity of objective function (FSSR1) to accelerate SSVR, including the linear and nonlinear cases. Most inactive features and samples can be simultaneously discarded before training SSVR. Only one reduced SSVR (RSSVR) needs to be solved. Moreover, to further speed up RSSVR, the screening rule based on duality gap (FSSR2) is applied to continuously discard the remaining inactive variables during the reduced model training process. Combining the rule with the grid-search method, the framework of our final method (FSSR–SSVR) is to alternatively conduct FSSR1 and FSSR2, which leads to substantial savings in both the memory usage and the computational cost. There are two appealing advantages of our FSSR-SSVR: first, it is safe, i.e., the features and samples discarded by FSSR are guaranteed to be inactive; second, it has synergy effect (in other words, the results of the previous feature screening can improve the performance of the next sample screening, and vice versa). Furthermore, the stochastic dual coordinate ascent (SDCA) method is employed as an efficient solver. Experiments on three synthetic datasets and 11 real-world datasets demonstrate the efficiency of our method.

Dual Free Adaptive Minibatch SDCA for Empirical Risk Minimization

In this paper, we develop an adaptive dual free Stochastic Dual Coordinate Ascent (adfSDCA) algorithm for regularized empirical risk minimization problems. This is motivated by the recent work on dual free SDCA of Shalev-Shwartz (2016). The novelty of our approach is that the coordinates to update at each iteration are selected non-uniformly from an adaptive probability distribution, and this extends the previously mentioned work which only allowed for a uniform selection of ``dual coordinates from a fixed probability distribution. We describe an efficient iterative procedure for generating the non-uniform samples, where the scheme selects the coordinate with the greatest potential to decrease the sub-optimality of the current iterate. We also propose a heuristic variant of adfSDCA that is more aggressive than the standard approach. Furthermore, in order to utilize multi-core machines, we consider a mini-batch adfSDCA algorithm and develop complexity results that guarantee the algorithm's convergence. The work is concluded with several numerical experiments to demonstrate the practical benefits of the proposed approach.

Optimizing Top-<inline-formula> <tex-math notation="LaTeX">$k$ </tex-math> </inline-formula> Multiclass SVM via Semismooth Newton Algorithm

Top- performance has recently received increasing attention in large data categories. Advances, like a top- multiclass support vector machine (SVM), have consistently improved the top- accuracy. However, the key ingredient in the state-of-the-art optimization scheme based upon stochastic dual coordinate ascent relies on the sorting method, which yields complexity. In this paper, we leverage the semismoothness of the problem and propose an optimized top- multiclass SVM algorithm, which employs semismooth Newton algorithm for the key building block to improve the training speed. Our method enjoys a local superlinear convergence rate in theory. In practice, experimental results confirm the validity. Our algorithm is four times faster than the existing method in large synthetic problems; Moreover, on real-world data sets it also shows significant improvement in training time.

Acceleration for proximal stochastic dual coordinate ascent algorithm in solving regularised loss minimisation with norm

An accelerated version of the proximal stochastic dual coordinate ascent (SDCA) algorithm in solving regularised loss minimisation with norm is presented, wherein a momentum is introduced and the strong theoretical guarantees of SDCA are shared. Moreover, it is also suitable for various key machine learning optimisation problems including support vector machine (SVM), multiclass SVM, logistic regression, and ridge regression. In particular, the Nestrov's estimate sequence technique to adjust the weight coefficient dynamically and conveniently is adopted. It is applied for training linear SVM from the large training dataset. Experimental results show that the proposed method has a competitive classification performance and faster convergence speed than state-of-the-art algorithms.

EDGE2VEC: Edge Representations for Large-Scale Scalable Hierarchical Learning

In present front-line of Big Data, prediction tasks over the nodes and edges in complex deep architecture needs a careful representation of features by assigning hundreds of thousands, or even millions of labels and samples for information access system, especially for hierarchical extreme multi-label classification. We introduce edge2vec, an edge representations framework for learning discrete and continuous features of edges in deep architecture. In edge2vec, we learn a mapping of edges associated with nodes where random samples are augmented by statistical and semantic representations of words and documents. We argue that infusing semantic representations of features for edges by exploiting word2vec and para2vec is the key to learningricherrepresentationsforexploringtargetnodes or labels in the hierarchy. Moreover, we design and implement a balanced stochastic dual coordinate ascent (DCA)-based support vector machine for speeding up training. We introduce a global decision-based top-down walks instead of random walks to predict the most likelihood labels in the deep architecture. We judge the efficiency of edge2vec over the existing state-of-the-art techniques on extreme multi-label hierarchical as well as flat classification tasks. The empirical results show that edge2vec is very promising and computationally very efficient in fast learning and predicting tasks. In deep learning workbench, edge2vec represents a new direction for statistical and semantic representations of features in task-independent networks.

Group-Based Alternating Direction Method of Multipliers for Distributed Linear Classification.

The alternating direction method of multipliers (ADMM) algorithm has been widely employed for distributed machine learning tasks. However, it suffers from several limitations, e.g., a relative low convergence speed, and an expensive time cost. To this end, in this paper, a novel method, namely the group-based ADMM (GADMM), is proposed for distributed linear classification. In particular, to accelerate the convergence speed and improve global consensus, a group layer is first utilized in GADMM to divide all the slave nodes into several groups. Then, all the local variables (from the slave nodes) are gathered in the group layer to generate different group variables. Finally, by using a weighted average method, the group variables are coordinated to update the global variable (from the master node) until the solution of the global problem is reached. According to the theoretical analysis, we found that: 1) GADMM can mathematically converge at the rate , where is the number of outer iterations and 2) by using the grouping methods, GADMM can improve the convergence speed compared with the distributed ADMM framework without grouping methods. Moreover, we systematically evaluate GADMM on four publicly available LIBSVM datasets. Compared with disADMM and stochastic dual coordinate ascent with alternating direction method of multipliers-ADMM, for distributed classification, GADMM is able to reduce the number of outer iterations, which leads to faster convergence speed and better global consensus. In particular, the statistical significance test has been experimentally conducted and the results validate that GADMM can significantly save up to 30% of the total time cost (with less than 0.6% accuracy loss) compared with disADMM on large-scale datasets, e.g., webspam and epsilon.

Automatic Recognition of Fetal Facial Standard Plane in Ultrasound Image via Fisher Vector.

Acquisition of the standard plane is the prerequisite of biometric measurement and diagnosis during the ultrasound (US) examination. In this paper, a new algorithm is developed for the automatic recognition of the fetal facial standard planes (FFSPs) such as the axial, coronal, and sagittal planes. Specifically, densely sampled root scale invariant feature transform (RootSIFT) features are extracted and then encoded by Fisher vector (FV). The Fisher network with multi-layer design is also developed to extract spatial information to boost the classification performance. Finally, automatic recognition of the FFSPs is implemented by support vector machine (SVM) classifier based on the stochastic dual coordinate ascent (SDCA) algorithm. Experimental results using our dataset demonstrate that the proposed method achieves an accuracy of 93.27% and a mean average precision (mAP) of 99.19% in recognizing different FFSPs. Furthermore, the comparative analyses reveal the superiority of the proposed method based on FV over the traditional methods.

Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization

We introduce a proximal version of the stochastic dual coordinate ascent method and show how to accelerate the method using an inner-outer iteration procedure. We analyze the runtime of the framework and obtain rates that improve state-of-the-art results for various key machine learning optimization problems including SVM, logistic regression, ridge regression, Lasso, and multiclass SVM. Experiments validate our theoretical findings.

Learning Relative Similarity by Stochastic Dual Coordinate Ascent

Learning relative similarity from pairwise instances is an important problem in machine learning and has a wide range of applications. Despite being studied for years, some existing methods solved by Stochastic Gradient Descent (SGD) techniques generally suffer from slow convergence. In this paper, we investigate the application of Stochastic Dual Coordinate Ascent (SDCA) technique to tackle the optimization task of relative similarity learning by extending from vector to matrix parameters. Theoretically, we prove the optimal linear convergence rate for the proposed SDCA algorithm, beating the well-known sublinear convergence rate by the previous best metric learning algorithms. Empirically, we conduct extensive experiments on both standard and large-scale data sets to validate the effectiveness of the proposed algorithm for retrieval tasks.

RankCNN: When learning to rank encounters the pseudo preference feedback

Learning to rank has received great attentions in the field of text retrieval for several years. However, a few researchers introduce the topic into visual reranking due to the special nature of image presentation. In this paper, a novel unsupervised visual reranking is proposed, termed rank via the convolutional neural networks (RankCNN). This approach integrates deep learning with pseudo preference feedback. The optimal set of pseudo preference pairs is first detected from initial list by a modified graph-based method. Ranking is then reduced to pairwise classification in the architecture of CNN. In addition, Accelerated Mini-Batch Stochastic Dual Coordinate Ascent (ASDCA) is introduced to the framework to accelerate the training. The experiments indicate the competitive performance on the LETOR 4.0, the Paris and the Francelandmark dataset.

Stochastic dual coordinate ascent methods for regularized loss

Stochastic Gradient Descent (SGD) has become popular for solving large scale supervised machine learning optimization problems such as SVM, due to their strong theoretical guarantees. While the clo...