Parallel Learning Algorithm Research Articles

A Fuzzy-Model-Based Approach to Optimal Control for Nonlinear Markov Jump Singularly Perturbed Systems: A Novel Integral Reinforcement Learning Scheme

A fuzzy-model-based approach is developed to investigate the reinforcement learning-based optimization for nonlinear Markov jump singularly perturbed systems. As the first attempt, an offline parallel iteration learning algorithm is presented to solve the coupled algebraic Riccati equations with singular perturbation and jumping parameters. Furthermore, based on the integral reinforcement learning approach, a novel online parallel learning algorithm is proposed by employing the slow and fast sampled data simultaneously, where the impacts of stochastic jumping and ill-conditioned numerical problems are avoided. Meanwhile, the convergence of the proposed learning algorithms is proved. Finally, we present a tunnel diode circuit model to demonstrate the efficacy of the proposed methods.

Hybrid Parallel Stochastic Configuration Networks for Industrial Data Analytics

As a class of randomized learner model, stochastic configuration networks (SCNs) have been successfully applied in a few data analytics tasks. Given the industrial big data modeling tasks, however, the original SCNs potentially lead to excessive training time. To this end, this article extends SCNs to a hybrid parallel version, termed hybrid parallel SCNs (HPSCNs). In the hybrid parallel learning algorithm, two SCNs are synchronously constructed. The difference between the two of them is that one employs a point-incremental algorithm, and another one adopts a block-incremental algorithm. Moreover, a data parallel method is established based on a dynamic block strategy to accelerate the establishment of candidate node pool for each SCN. Additionally, this article proposes an adaptive hyperparameter adjustment method, which allows the hyperparameters in the supervisory mechanism to be automatically adjusted along with the learning process. Comparative experiments are first conducted through four large-scale benchmark datasets, followed by the fully discussion on the algorithm parameters. Finally, a practical industrial application case is made, where a grinding particle size soft sensor is developed based on HPSCN, showing the effectiveness of the proposed algorithm.

Game theory-based energy efficiency optimization model for the Internet of Things

This paper conducts a game-theoretic based optimization study on the energy efficiency of the Internet of Things (IoT). For the problem that wireless sensor node information in the IoT requires the selection of a suitable backbone access point for energy efficiency optimization, this paper first establishes a mathematical model for the system-level energy optimization of the sensor node free-choice access point problem and then proposes a game model based on the concept of cooperation and the corresponding utility function, and after theoretical analysis. Among the new connections in the future, more than 30% are suitable for carrying by the cellular network, so the network challenges brought by mobile operators are huge. The paper then studied the current mainstream Internet of Things technology in the development of mobile networks—the​ application principles and key technologies of Narrow Band Internet of Things (NB-IoT), combined with the current wireless network optimization and maintenance work, and studied How does the Internet of Things become the focus of the industry and actively lay out and promote industrial development for operators, and how does NB-IoT move from a concept to small-scale commercial use, become the operator’s leading role in promoting the standard industry, and hope to become an industry leader It is proved that the best access point allocation scheme is the optimal equilibrium point of the proposed game. Then a non-correlated parallel learning algorithm is proposed, according to which the system can converge to the optimal equilibrium point with a very low probability after learning, which is the optimal solution of the proposed system energy efficiency optimization problem, and achieve the global optimal system energy efficiency. Compared with other models, our model has improved efficiency by about 12% and accuracy by about 8%, and it can be applied in practice.

Distributed matrix factorization based on fast optimization for implicit feedback recommendation

In big data scenarios, matrix factorization (MF) is widely used in recommendation systems as it can offer high accuracy and scalability. However, when using MF to process large-scale implicit feedback data, the following two problems arise. One is that it is difficult to effectively obtain negative feedback information, which causes relatively poor recommendation accuracy. The other is that the limited resources of a single machine make the model training inefficient, and in particular, the acquisition of negative feedback information further increases the time complexity of model training. In order to solve the above two problems, we first propose a user-activity and item-popularity weighted matrix factorization (UIWMF) recommendation algorithm, which assigns every missing data different weight based on user activity and item popularity, gets negative feedback information more realistically, and leads to better recommendation accuracy. Meanwhile, in order to reduce the additional computational overhead caused by the weight strategy, we develop a fast optimization strategy to enhance the efficiency. In order to break the resource constraints of a single machine, we propose a distributed UIWMF (DUIWMF) algorithm based on Spark, which adopts an efficient parallel learning algorithm to train the model and utilizes cached in-block and out-block information to effectively reduce the communication overhead in a distributed environment. We conduct experiments on three public datasets, and the experimental results demonstrate that, comparing with the baseline MF methods, DUIWMF model has comparable performance in terms of recommendation accuracy and model training efficiency.

A novel parallel learning algorithm for pattern classification

In today’s data-intensive applications, machine learning constructs algorithms that are capable of learning and making predications on the data. Margin setting algorithm (MSA) is a novel machine learning algorithm for pattern classification. It employs an artificial immune system approach to generates prototype regions as the classification boundaries. However, its computation time limited its applications in real-world application. When the datasets grow in size and algorithm complexity increases, it is necessary to spread the work among multiple cores and processors. To reduce the execution time during classification, a parallel implementation of MSA, called PMSA is proposed for multicore and multiprocessor system. It is the first work to scale up the classification time of MSA using parallel implementation. To evaluate the proposed PMSA algorithm, we used standard image datasets of 512 × 512 pixels and 321 × 481 pixels. Besides, benchmark datasets from University of California, Irvine Machine Learning Repository are also used. They are 768 data samples from dataset Pima Indian Diabetes, 683 data samples from dataset Wisconsin Breast Cancer, 690 data samples from dataset Australian Credit Approval, 178 data samples from dataset Wine and 391 data samples from dataset Svmguide2. The classification performance is compared with another two state-of-the-art classification algorithms: the artificial neural network and the support vector machine. The results show the proposed PMSA gains significant improvements in terms of execution time, with a promising speedup compared to the single-threaded CPU counterpart.

MR-AntMiner: A Novel MapReduce Classification Rule Discovery with Ant Colony Intelligence

Ant colony optimization (ACO) algorithms have been successfully applied to data classification problems that aim at discovering a list of classification rules. However, on the one hand, the ACO algorithm has defects including long search times and convergence issues with non-optimal solutions. On the other hand, given bottlenecks such as memory restrictions, time complexity, or data complexity, it is too hard to solve a problem when its scale becomes too large. One solution for this issue is to design a highly parallelized learning algorithm. The MapReduce programming model has quickly emerged as the most common model for executing simple algorithmic tasks over huge volumes of data, since it is simple, highly abstract, and efficient. Therefore, MapReduce-based ACO has been researched extensively. However, due to its unidirectional communication model and the inherent lack of support for iterative execution, ACO algorithms cannot easily be implemented on MapReduce. In this paper, a novel classification rule discovery algorithm is proposed, namely MR-AntMiner, which can capitalize on the benefits of the MapReduce model. In order to construct quality rules with fewer iterations as well as less communication between different nodes to share the parameters used by each ant, our algorithm splits the training data into some subsets that are randomly mapped to different mappers; then the traditional ACO algorithm is run on each mapper to gain the local best rule set, and the global best rule list is produced in the reducer phase according to a voting mechanism. The performance of our algorithm was studied experimentally on 14 publicly available data sets and further compared to several state-of-the-art classification approaches in terms of accuracy. The experimental results show that the predictive accuracy obtained by our algorithm is statistically higher than that of the compared targets. Furthermore, experimental studies show the feasibility and the good performance of the proposed parallelized MR-AntMiner algorithm.

A meta-feature based unified framework for both cold-start and warm-start explainable recommendations

Recently, recommender systems have received an increasing amount of attention from researchers due to their indispensable role in the more and more popular e-commercial websites. Although a lot of methods have been proposed for warm-start recommendation, cold-start recommendation still remains open as one of the major challenges of recommender systems. The existing approaches often suffer from two defects. The first is the lack of unified framework. The existing researches often deal with the cold-start recommendation and the warm-start recommendation separately, which makes their respective methods hard to integrate into a system and keeps the cold-start users/items away from the existing ones. The second is the poor interpretability. The existing methods often ignore the complicated preferential relationships between users and item features, and can not quantitatively explain the multiple reasons that cause a user chooses an item. In this paper, we aim at the problem of making explainable recommendations for both warm-start and cold-start users/items in a unified framework, of which the challenges are three-fold, the lack of meaningful information, large-scale data, and quantitative explanation. To address these challenges, we propose a novel concept referred to as meta-feature, and a Meta-feature based Explainable Recommendation Framework (MERF). Meta-features are latent features about item features, which can reveal the preferential relationship between users and item features, not just the items. MERF is able to make recommendations for both cold-start and warm-start users/items in a unified framework based on meta-feature. Especially, thanks to meta-feature, MERF can make cold-start recommendations requiring no historical rating records but just the item features. To make a recommendation with a quantitative explanation, we propose a Personalized Feature Preference (PFP) vector to characterize the different importance of item features to a user. MERF makes a recommendation based on an Item Rating Matrix and an Explanation Matrix, which can be estimated by fusing PFP and meta-features. To improve the efficiency of MERF, we also propose a parallel learning algorithm and an incremental updating algorithm for PFP. At last, extensive experiments conducted on real datasets verify the effectiveness and efficiency of the proposed approach.

Using Machine Learning to Address Customer Privacy Concerns: An Application with Click-stream Data

The ever-increasing volume of consumer data provide unprecedented opportunities for firms to predict consumer behavior, target customers, and provide customized service. Recent trends of more restrictive privacy regulations worldwide, however, present great challenges for firms whose business activities rely on consumer data. We address these challenges by applying the recently developed federated learning approach --- a privacy-preserving machine learning approach that uses a parallelized learning algorithm to train a model locally on each individual user's device. We apply this approach to data from an online retailer and train a Gated Recurrent Unit recurrent neural network to predict each consumer's click-stream. We show the firm can predict each consumer's activities with a high level of accuracy without the need to store, access, or analyze consumer data in a centralized location, thereby protecting their sensitive information.

Latent‐lSVM classification of very high‐dimensional and large‐scale multi‐class datasets

SummaryWe propose a new parallel learning algorithm of latent local support vector machines (SVM), called latent‐lSVM for effectively classifying very high‐dimensional and large‐scale multi‐class datasets. The common framework of texts/images classification tasks using the Bag‐Of‐(visual)‐Words model for the data representation leads to hard classification problem with thousands of dimensions and hundreds of classes. Our latent‐lSVM algorithm performs these complex tasks into two main steps. The first one is to use latent Dirichlet allocation for assigning the datapoint (text/image) to some topics (clusters) with the corresponding probabilities. This aims at reducing the number of classes and the number of datapoints in the cluster compared to the full dataset, followed by the second one: to learn in a parallel way nonlinear SVM models to classify data clusters locally. The numerical test results on nine real datasets show that the latent‐lSVM algorithm achieves very high accuracy compared to state‐of‐the‐art algorithms. An example of its effectiveness is given with an accuracy of 70.14% obtained in the classification of Book dataset having 100 000 individuals in 89 821 dimensional input space and 661 classes in 11.2 minutes using a PC Intel(R) Core i7‐4790 CPU, 3.6 GHz, 4 cores.

A parallel C4.5 decision tree algorithm based on MapReduce

SummaryIn the supervised classification, large training data are very common, and decision trees are widely used. However, as some bottlenecks such as memory restrictions, time complexity, or data complexity, many supervised classifiers including classical C4.5 tree cannot directly handle big data. One solution for this problem is to design a highly parallelized learning algorithm. Motivated by this, we propose a parallelized C4.5 decision tree algorithm based on MapReduce (MR‐C4.5‐Tree) with 2 parallelized methods to build the tree nodes. First, an information entropy‐based parallelized attribute selection method (MR‐A‐S) on several subsets for MR‐C4.5‐Tree is proposed to confirm the best splitting attribute and the cut points. Then, a data splitting method (MR‐D‐S) in parallel is presented to partition the training data into subsets. At last, we introduce the MR‐C4.5‐Tree learning algorithm that grows in a top‐down recursive way. Besides, the depth of the constructed decision tree, the number of samples and the maximal class probability in each tree node are used as the termination conditions to avoid the over‐partitioning problem. Experimental studies show the feasibility and the good performance of the proposed parallelized MR‐C4.5‐Tree algorithm.

Feedback linearization and feedforward neural network control with application to twin rotor mechanism

A twin rotor multi-input multi-output system (TRMMS) is a high-order nonlinear system with a significant cross-coupling effect. The control of TRMMSs is considered a markedly challenging topic in the field of robust control. This study proposes a novel feedback linearization and feedforward neural network controller design for a TRMMS with almost disturbance decoupling (ADD) capabilities. The proposed composite controller achieves exponentially global stability and ADD performance without applying any traditional parallel learning algorithms. This study proposes an organization of the feedforward neural network and the weights among the layers to guarantee the stability of the overall system. A number of nonlinear systems, which are too complex to be solved by general ADD studies, are proposed in this study to demonstrate that the proposed methodology can effectively achieve the tracking and ADD performances through Matlab. Moreover, an efficient algorithm is proposed for designing the feedback linearization and feedforward neural network control with ADD and tracking capabilities.

BNFinder2: Faster Bayesian network learning and Bayesian classification

Summary: Bayesian Networks (BNs) are versatile probabilistic models applicable to many different biological phenomena. In biological applications the structure of the network is usually unknown and needs to be inferred from experimental data. BNFinder is a fast software implementation of an exact algorithm for finding the optimal structure of the network given a number of experimental observations. Its second version, presented in this article, represents a major improvement over the previous version. The improvements include (i) a parallelized learning algorithm leading to an order of magnitude speed-ups in BN structure learning time; (ii) inclusion of an additional scoring function based on mutual information criteria; (iii) possibility of choosing the resulting network specificity based on statistical criteria and (iv) a new module for classification by BNs, including cross-validation scheme and classifier quality measurements with receiver operator characteristic scores.Availability and implementation: BNFinder2 is implemented in python and freely available under the GNU general public license at the project Web site https://launchpad.net/bnfinder, together with a user’s manual, introductory tutorial and supplementary methods.Contact: dojer@mimuw.edu.pl or bartek@mimuw.edu.plSupplementary information: Supplementary data are available at Bioinformatics online.

An efficient parallel neural network-based multi-instance learning algorithm

Multiple instance learning (MIL) has been studied actively in recent years. However, it is facing a computational challenge due to the large scale of data volume. Parallel computing is a good way of overcoming the computational challenge. In this paper, we propose a new MIL method based on a MIL back-propagation neural network (MIBP), which is an extension of the standard back-propagation neural network (BPNN) that uses labeled bags of instances as training data. We use parallel computing to speed up the learning process. The proposed method finds a concept point t in the feature space which is close to instances from positive bags and far from instances in negative bags. The description of our method is as follows: First, train MIBP with positive and negative bags. Second, extract t from the trained MIBP. This is achieved by, for each positive bag, presenting all the instances to the trained MIBP and selecting the one with maximal output value. The t is then obtained by averaging all the extracted instances. Finally, a sensitivity analysis of the trained MIBP is performed to obtain feature relevance/weighting information. Parallel computing is performed during the training of the MIBP. We conduct experiments to measure the performance of the obtained t when used for classification purposes and evaluate the parallel computing method. The experimental results on the MUSK data set show that our method has better classification performance and is more computationally efficient than other well-established MIL methods.

Determination of Wireless Networks Parameters through Parallel Hierarchical Support Vector Machines

We consider the problems of 1) estimating the physical locations of nodes in an indoor wireless network, and 2) estimating the channel noise in a MIMO wireless network, since knowing these parameters are important to many tasks of a wireless network such as network management, event detection, location-based service, and routing. A hierarchical support vector machines (H-SVM) scheme is proposed with the following advantages. First, H-SVM offers an efficient evaluation procedure in a distributed manner due to hierarchical structure. Second, H-SVM could determine these parameters based only on simpler network information, e.g., the hop counts, without requiring particular ranging hardware. Third, the exact mean and the variance of the estimation error introduced by H-SVM are derived which are seldom addressed in previous works. Furthermore, we present a parallel learning algorithm to reduce the computation time required for the proposed H-SVM. Thanks for the quicker matrix diagonization technique, our algorithm can reduce the traditional SVM learning complexity from O(n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> ) to O(n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) where n is the training sample size. Finally, the simulation results verify the validity and effectiveness for the proposed H-SVM with parallel learning algorithm.

THE MORPHOLOGY OF HUMAN ACTIONS: FINDING ESSENTIAL ACTUATORS, MOTION PATTERNS, AND THEIR COORDINATION

In this paper, we present the steps required for the construction of a praxicon, a structured lexicon of human actions, through the learning of grammar systems for human actions. The discovery of a Human Activity Language involves learning the syntax of human motion which requires the construction of this praxicon. The morphology inference process assumes that a non-arbitrary symbolic representation of the human movement is given. Thus, to analyze the morphology of a particular action, we are given a symbolic representation for the motion of each actuator associated with several repeated performances of this action. As a formal model, we propose a new Parallel Synchronous Grammar System where each component grammar corresponds to an actuator. We present a novel parallel learning algorithm to induce this grammar system. Our representation explicitly contains the set of joints (degrees of freedom) actually responsible for achieving the goal aimed by the activity, the motion performed by each participating actuator, and the synchronization rules modeling coordination among these actuators. We evaluated our inference approach with synthetic data and real human motion data. The algorithm manages to induce the correct grammar system even when the input contains noise. Therefore, our approach was successful in both representational and learning aspects, and may serve as a tool to parse movement, learn patterns, and to generate actions.

A MULTI-DIMENSION PREDICTOR BASED ON PDRNN FOR ENGINE ROOM MONITORING

This paper presents a new multi-dimension predictive model based on the diagonal recurrent neural networks (PDRNN) with a parallel learning algorithm. This method can be used to predict not only values, but also some points in the multidimension space. And also its applications in marine engine room will be discussed in the paper. Some analysis results show the significant improvement to multi parameters simultaneous prediction using the PDRNN algorithm in marine engine room.

A scalable parallel algorithm for training a hierarchical mixture of neural experts

Efficient parallel learning algorithms are proposed for training a powerful modular neural network, the hierarchical mixture of experts (HME). Parallelizations are based on the concept of modular parallelism, i.e. parallel execution of network modules. From modeling the speedup as a function of the number of processors and the number of training examples, several improvements are derived, such as pipelining the training examples by packets. Compared to experimental measurements, theoretical models are accurate. For regular topologies, an analysis of the models shows that the parallel algorithms are highly scalable when the size of the experts grows from linear units to multi-layer perceptrons (MLPs). These results are confirmed experimentally, achieving near-linear speedups for HME-MLP. Although this work can be viewed as a case study in the parallelization of HME neural networks, both algorithms and theoretical models can be expanded to different learning rules or less regular tree architectures.

Neurofuzzy mixture of experts network parallel learning and model construction algorithms

A connection between a fuzzy neural network model with the mixture of experts network (MEN) modelling approach is established. Based on this linkage, two new neuro-fuzzy MEN construction algorithms are proposed to overcome the curse of dimensionality that is inherent in the majority of associative memory networks and/or other rule based systems. The first construction algorithm employs a function selection manager module in an MEN system. The second construction algorithm is based on a new parallel learning algorithm in which each model rule is trained independently, for which the parameter convergence property of the new learning method is established. As with the first approach, an expert selection criterion is utilised in this algorithm. These two construction methods are equivalent in their effectiveness in overcoming the curse of dimensionality by reducing the dimensionality of the regression vector, but the latter has the additional computational advantage of parallel processing. The proposed algorithms are analysed for effectiveness followed by numerical examples to illustrate their efficacy for some difficult data based modelling problems.

PARALLEL NEURAL LEARNING FOR CONTROL PROBLEMS ON A BUS-BASED ARCHITECTURE

In [6], we distributed training instances over a single-channel broadcast communication model to speed up execution of the back-propagation learning algorithm for classification problems. In this paper, we extend this concept to control problems, where the output is not necessarily 0 or 1, but ranges over an interval. We first propose a modified back-propagation learning algorithm that incrementally decreases the error threshold by half in order to process training instances with large weight changes as quickly as possible. This modified back-propagation learning algorithm is then parallelized using the single-channel broadcast communication model to n processors, where n is the number of training instances. Finally, the parallel back-propagation learning algorithm is modified for execution on a bounded number of processors to cope with real-world conditions.

A Nearly Optimal Back-Propagation Learning Algorithm on a Bus-Based Architecture

The back-propagation model is applied mainly to multiple layers of neurons with feed-forward connections. It applies the generalized delta learning rule to derive appropriate weights on connections from a set of training instances. In the past, a parallel back-propagation learning algorithm on a bus-based architecture with an infinite number of processors was proposed to speed up the learning of weights. In this paper, the parallel back-propagation learning algorithm is modified, so that it is applicable to any number of processors. The given training instances are equally partitioned and put on these processors. When learning starts, each processor searches its own training subset to update the weight matrix and to broadcast the new result. Both analytical and experimental results show that the average speed-up can reach nearly O(r) by r processors if r is much less than the number of training instances.