Twin Extreme Learning Machine Research Articles

Twin extreme learning machine based on heteroskedastic Gaussian noise model and its application in short-term wind-speed forecasting

Extreme learning machine (ELM) has received increasingly more attention because of its high efficiency and ease of implementation. However, the existing ELM algorithms generally suffer from the drawbacks of noise sensitivity and poor robustness. Therefore, we combine the advantages of twin hyperplanes with the fast speed of ELM, and then introduce the characteristics of heteroscedastic Gaussian noise. In this paper, a new regressor is proposed, which is called twin extreme learning machine based on heteroskedastic Gaussian noise (TELM-HGN). In addition, the augmented Lagrange multiplier method is introduced to optimize and solve the presented model. Finally, a significant number of experiments were conducted on different data-sets including real wind-speed data, Boston housing price dataset and stock dataset. Experimental results show that the proposed algorithms not only inherits most of the merits of the original ELM, but also has more stable and reliable generalization performance and more accurate prediction results. These applications demonstrate the correctness and effectiveness of the proposed model.

Distance Metric Optimization-Driven Neural Network Learning Framework for Pattern Classification

As a novel neural network learning framework, Twin Extreme Learning Machine (TELM) has received extensive attention and research in the field of machine learning. However, TELM is affected by noise or outliers in practical applications so that its generalization performance is reduced compared to robust learning algorithms. In this paper, we propose two novel distance metric optimization-driven robust twin extreme learning machine learning frameworks for pattern classification, namely, CWTELM and FCWTELM. By introducing the robust Welsch loss function and capped L2,p-distance metric, our methods reduce the effect of outliers and improve the generalization performance of the model compared to TELM. In addition, two efficient iterative algorithms are designed to solve the challenges brought by the non-convex optimization problems CWTELM and FCWTELM, and we theoretically guarantee their convergence, local optimality, and computational complexity. Then, the proposed algorithms are compared with five other classical algorithms under different noise and different datasets, and the statistical detection analysis is implemented. Finally, we conclude that our algorithm has excellent robustness and classification performance.

Quantum algorithm for twin extreme learning machine

Twin extreme learning machine (TELM) is a powerful learning algorithm, which aims at learning two nonparallel hyperplanes for data classification. However, classical TELM algorithm becomes computationally expensive when it involves big data sets. In this paper, we devise a quantum TELM algorithm to address this issue. Specifically, we first utilize the quantum amplitude estimation algorithm to prepare the desired input states and then call the quantum linear systems of equations, which adopts block-encoding technique, to obtain the model parameters in the training process. Then we invoke the swap test to estimate the distances from a new data point to the two hyperplanes and then make a classification in the prediction stage. The final complexity analysis demonstrates that our algorithm has an exponential speedup under certain conditions over the classical counterpart.

Robust Fisher-Regularized Twin Extreme Learning Machine with Capped L1-Norm for Classification

Twin extreme learning machine (TELM) is a classical and high-efficiency classifier. However, it neglects the statistical knowledge hidden inside the data. In this paper, in order to make full use of statistical information from sample data, we first come up with a Fisher-regularized twin extreme learning machine (FTELM) by applying Fisher regularization into TELM learning framework. This strategy not only inherits the advantages of TELM, but also minimizes the within-class divergence of samples. Further, in an effort to further boost the anti-noise ability of FTELM method, we propose a new capped L1-norm FTELM (CL1-FTELM) by introducing capped L1-norm in FTELM to dwindle the influence of abnormal points, and CL1-FTELM improves the robust performance of our FTELM. Then, for the proposed FTELM method, we utilize an efficient successive overrelaxation algorithm to solve the corresponding optimization problem. For the proposed CL1-FTELM, an iterative method is designed to solve the corresponding optimization based on re-weighted technique. Meanwhile, the convergence and local optimality of CL1-FTELM are proved theoretically. Finally, numerical experiments on manual and UCI datasets show that the proposed methods achieve better classification effects than the state-of-the-art methods in most cases, which demonstrates the effectiveness and stability of the proposed methods.

A generalized adaptive robust distance metric driven smooth regularization learning framework for pattern recognition

In this work, we propose a novel generalized adaptive robust distance metric called Lδ(u). Compared with other distance metrics, Lδ(u) has some desirable salient properties, such as symmetry, boundedness, robustness, nonconvexity, and adaptivity, with both first-order and higher-order moments from samples. On the other hand, Lδ(u) can pick different robust distance metrics for different learning tasks during the learning process by the adaptive parameter δ. Furthermore, we apply Lδ(u) to twin extreme learning machine (TELM) and develop an smooth regularized TELM learning framework for supervised and semi-supervised classification. By introducing the structural risk minimization (SRM) principle and smoothing techniques, the learning framework perfectly overcomes the computational burden associated with the matrix inversion operation required during TELM solving, while also significantly improving performance. More importantly, the proposed learning framework not only improves the robustness of TELM, but also can effectively use the geometric information of the marginal distribution embedded in the unlabeled samples to construct a more reasonable classifier. Finally, the globally convergent and quadratic convergent fast Newton-Armijo algorithm and DC (difference of convex functions) programming algorithm (DCA) are designed to solve the proposed methods. Experimental results demonstrate the effectiveness and robustness of our methods.

Time efficient variants of Twin Extreme Learning Machine

Twin Extreme Learning Machine models can obtain better generalization ability than the standard Extreme Learning Machine model. But, they require to solve a pair of quadratic programming problems for this. It makes them more complex and computationally expensive than the standard Extreme Learning Machine model. In this paper, we propose two novel time-efficient formulations of the Twin Extreme Learning Machine, which only require the solution of systems of linear equations for obtaining the final classifier. In this sense, they can combine the benefits of the Twin Support Vector Machine and standard Extreme Learning Machine in the true sense. We term our first formulation as ‘Least Squared Twin Extreme Learning Machine’. It minimizes the L2-norm of error variables in its optimization problem. Our second formulation ‘Weighted Linear loss Twin Extreme Learning Machine’ uses the weighted linear loss function for calculating the empirical error, which makes it insensitive towards outliers. Numerical results obtained with multiple benchmark datasets show that proposed formulations are time efficient with better generalization ability. Further, we have used the proposed formulations in the detection of phishing websites and shown that they are much more effective in the detection of phishing websites than other Extreme Learning Machine models.

Research on Twin Extreme Learning Fault Diagnosis Method Based on Multi-Scale Weighted Permutation Entropy

Due to the complicated engineering operation of the check valve in a high−pressure diaphragm pump, its vibration signal tends to show non−stationary and non−linear characteristics. These leads to difficulty extracting fault features and, hence, a low accuracy for fault diagnosis. It is difficult to extract fault features accurately and reliably using the traditional MPE method, and the ELM model has a low accuracy rate in fault classification. Multi−scale weighted permutation entropy (MWPE) is based on extracting multi−scale fault features and arrangement pattern features, and due to the combination of extracting a sequence of amplitude features, fault features are significantly enhanced, which overcomes the deficiency of the single−scale permutation entropy characterizing the complexity of vibration signals. It establishes the check valve fault diagnosis model from the twin extreme learning machine (TELM). The TELM fault diagnosis model established, based on MWPE, aims to find a pair of non−parallel classification hyperplanes in the equipment state space to improve the model’s applicability. Experiments show that the proposed method effectively extracts the characteristics of the vibration signal, and the fault diagnosis model effectively identifies the fault state of the check valve with an accuracy rate of 97.222%.

Lagrangian Regularized Twin Extreme Learning Machine for Supervised and Semi-Supervised Classification

Twin extreme learning machine (TELM) is a phenomenon of symmetry that improves the performance of the traditional extreme learning machine classification algorithm (ELM). Although TELM has been widely researched and applied in the field of machine learning, the need to solve two quadratic programming problems (QPPs) for TELM has greatly limited its development. In this paper, we propose a novel TELM framework called Lagrangian regularized twin extreme learning machine (LRTELM). One significant advantage of our LRTELM over TELM is that the structural risk minimization principle is implemented by introducing the regularization term. Meanwhile, we consider the square of the l2-norm of the vector of slack variables instead of the usual l1-norm in order to make the objective functions strongly convex. Furthermore, a simple and fast iterative algorithm is designed for solving LRTELM, which only needs to iteratively solve a pair of linear equations in order to avoid solving two QPPs. Last, we extend LRTELM to semi-supervised learning by introducing manifold regularization to improve the performance of LRTELM when insufficient labeled samples are available, as well as to obtain a Lagrangian semi-supervised regularized twin extreme learning machine (Lap-LRTELM). Experimental results on most datasets show that the proposed LRTELM and Lap-LRTELM are competitive in terms of accuracy and efficiency compared to the state-of-the-art algorithms.

Regularized based implicit Lagrangian twin extreme learning machine in primal for pattern classification

In this paper, we suggest a novel approach termed as regularized based implicit Lagrangian twin extreme learning machine in primal as a pair of unconstrained convex minimization problem (RILTELM) where regularization term is added to follow the structural risk minimization principle. Here, we consider 2-norm of the slack vector of variables to make the problem strongly convex which results in a unique solution. Since it has non-smooth plus functions in their objective function, so we find an approximate solution by replacing the non-smooth plus function with smooth approximation function because to find an approximation solution in primal space is always superior to its dual. Due to non-smooth plus function, we solve the problem by either smooth approximation approach or generalized derivative approach. In addition, a functional iterative scheme is also suggested to find the optimal solution. Hence, no external optimization toolbox is required unlike in twin extreme learning machine (TELM) and twin support vector machine (TWSVM). The numerical experiments are demonstrated on artificial and real-world datasets and compared with TWSVM, ELM, TELM and LSTELM to establish the efficacy and applicability of proposed RILTELM.

A systematic review on the developments and evaluation of twin learning algorithms

There are fruitful advancements in Machine Learning Algorithms as per the demand of the digital world. The degree of performance of Twin Algorithms should have an upper hand over its parents. The improvements in the running time of Twin Extreme Learning Machine (TELM) and Twin Support Vector Machine (TWSVM) are commendable and have given confidence to the researchers. This paper contributes a detailed study on the developments of twin learning algorithms that are happened on top of the TWSVM, TELM and Twin Random Vector Functional Link (TRVFL). The single-hidden-layer learning paradigms and the twin models that have come after Extreme Learning Machine (ELM) tried to address a series of pitfalls of their parents. The iterative versions of single-hidden-layer models and their robust performances will catalyst growth in artificial intelligence. In this paper, some of the works in twin learning algorithms that are either technically sound or better in the performances are taken for the study. The working principle and performance of the algorithms are detailed with the help of published works and findings.

Robust supervised and semi-supervised twin extreme learning machines for pattern classification

In this paper, we first propose a novel robust loss function called adaptive capped Lθε-loss. The Lθε-loss has some interesting properties, such as robustness, non-convexity, and boundedness. During the learning process, for different problems, we can choose different loss functions through adaptive parameter θ. Then, a new robust twin extreme learning machine (RTELM) framework is presented by applying Lθε-loss and capped L1-norm distance metric. Compared with the twin extreme learning machine (TELM), RTELM overcomes the disadvantages of L2-norm distance metric and hinge loss, especially for the problem with outliers, while inherits the advantages of TELM. Further, we present a new Laplacian RTELM (Lap-RTELM for short) by introducing manifold regularization terms into RTELM. Intuitively, the Lap-RTELM can effectively utilize geometric information embedded in unlabeled samples and merge them as manifold regularization terms to learn a more reasonable classifier for semi-supervised classification (SSC) problems. Finally, two effective iterative algorithms are designed to solve the challenges brought by the non-convex optimization problems RTELM and Lap-RTELM, and theoretically guarantee the convergence, local optimality, and computational complexity of algorithms. Experiments on multiple datasets show that the proposed RTELM and Lap-RTELM are competitive with existing methods.

Capped L1-norm distance metric-based fast robust twin extreme learning machine

In this paper, we propose a new 0.00,0.00,1.00 fast robust twin extreme learning machine (FRTELM) based on the least squares sense and capped L1-norm distance metric. FRTELM first replaced the inequality constraints in TELM with equality constraints, and then introduced the capped L1-norm distance metric to replace the L2-norm distance metric in TELM. FRTELM not only retains the advantages of TELM, but also overcomes the shortcomings of TELM exaggeration of outliers based on squared L2-norm distance metrics. This improvement improves the robustness and learning efficiency of TELM in solving outlier problems. An efficient optimization algorithm is exploited to solve the nonconvex and nonsmooth challenging problem. In theory, we analyze and discuss the complexity of the algorithm in detail, and prove the convergence and local optimality of the algorithm. Extensive experiments conducted across multiple datasets demonstrates that the proposed method is competitive with state-of-the-art methods in terms of robustness and feasibility.

Supervised and semi-supervised twin parametric-margin regularized extreme learning machine

Twin extreme learning machine (TELM) has attracted considerable attention and achieved great success in the machine learning field. However, its performance will be severely affected when outliers exist in the dataset since TELM does not consider heteroscedasticity in practical applications. To improve the performance of TELM, a novel learning framework called twin parametric-margin extreme learning machine (TPMELM) was proposed. Further, to enhance the classification performance of our TPMELM in a semi-supervised learning setting, a Laplacian TPMELM (Lap-TPMELM) was developed by introducing manifold regularization into TPMELM. Using the geometric information of the marginal distribution embedded in unlabeled samples, Lap-TPMELM can effectively construct a more reasonable classifier. The TPMELM and Lap-TPMELM are suitable for many situations, especially when the data has heteroscedastic error structure. Moreover, the TPMELM and Lap-TPMELM are helpful in clarifying theoretical interpretation of parameters which control the bounds on proportions of support vectors and boundary errors. An efficient technique (successive over-relaxation, SOR) is applied in TPMELM and Lap-TPMELM, respectively. Experimental results show the effectiveness and reliability of the proposed methods.

Unconstrained convex minimization based implicit Lagrangian twin extreme learning machine for classification (ULTELMC)

The recently proposed twin extreme learning machine (TELM) requires solving two quadratic programming problems (QPPs) in order to find two non-parallel hypersurfaces in the feature that brings in the additional requirement of external optimization toolbox such as MOSEK. In this paper, we propose implicit Lagrangian TELM for classification via unconstrained convex minimization problem (ULTELMC) and further suggest iterative convergent schemes which eliminates the requirement of external optimization toolbox generally required in solving the quadratic programming problems (QPPs) of TELM. The solutions to the dual variables of the proposed ULTELMC are obtained using iterative schemes containing ‘plus’ function which is not differentiable. To overcome this shortcoming, the generalized derivative approach and smooth approximation approaches are suggested. Further, to test the performance of the proposed approaches, classification performances are compared with support vector machine (SVM), twin support vector machine (TWSVM), extreme learning machine (ELM), twin extreme learning machine (TELM) and Lagrangian extreme learning machine (LELM). Moreover, non-requirement to solve QPPs makes the iterative schemes find the solution faster as compared to the reported methods that finds the solution in dual space. Computational times required in finding the solutions are also presented for comparison.

An Application and Performance Evaluation of Twin Extreme Learning Machine Classifier for Intrusion Detection

Network along with Security is most significant in the digitalized environment. It is necessary to secure data from hackers and intruders. A strategy involved in protection of information from hackers will be termed as Intrusion Detection System (IDS).By taking into nature of attack or the usual conduct of user, investigation along with forecasting activities of the clients will be performed by mentioned system.Variousstrategies are utilized for the intrusion detection system. For the purpose of identification of hacking activity, utilization of machine learning based approach might be considered as novel strategy.In this paper, for identification of the hacking activity will be carried out by Twin Extreme Learning Machines (TELM).Employing the concept of Twin Support Vector Machine with the fundamental structure of Extreme Learning Machine is considered in the establishment of Twin Extreme Learning Machine (TELM).Also, its performance and accuracy are compared with the other intrusion detection techniques.

Unconstrained convex minimization based implicit Lagrangian twin random vector Functional-link networks for binary classification (ULTRVFLC)

In the spirit of twin support vector machine (TWSVM) and based on random vector Functional-link (RVFL) network, we propose implicit Lagrangian twin random vector Functional-link network via unconstrained minimization problem for better generalization performance. Unlike RVFL, which is a least-squares method, inequality constraints are employed to the proposed optimization problem to allow tolerating certain amount of error for handling overfitting problem. In this paper, the proposed constrained optimization problem is first reformulated to an unconstrained minimization problem and then is solved by iterative convergent schemes to eliminate the requirement to solving quadratic programming problems (QPPs) of its corresponding dual formulation. The ‘plus’ function present in the reformulated dual formulations of the proposed schemes is not differentiable which is handled by applying the generalized derivative approach or smooth approximation techniques and further solved by applying Newton–Armijo stepsize method. To establish the efficacy of the proposed algorithms, numerical experiments are carried out and comparison of results are carried out with support vector machine (SVM), twin support vector machine (TWSVM), kernel ridge regression (KRR), random vector Functional-link (RVFL) net, extreme learning machine (ELM), constrained extreme learning machine (CoELM) and twin extreme learning machine (TELM). The classification results are presented in terms of accuracy and further, statistical comparison of the algorithms’ performance is carried out to verify the significance of the proposed methods.

Sparse Twin Extreme Learning Machine With $\varepsilon$ -Insensitive Zone Pinball Loss

Twin extreme learning machine (TELM) based on the hinge-loss function shows great potential for pattern classification. However, the hinge loss is related to the shortest distance between sets and the corresponding classifier is hence sensitive to noise and unstable for resampling. In contrast, the $\varepsilon $ -insensitive zone pinball loss is related to the quantile distance therefore is not sensitive to noise, and the resulting solution is sparse. To improve the performance of TELM, we propose a novel TELM learning framework by introducing $\varepsilon $ -insensitive zone pinball loss function into TELM. Compared to TELM with hinge loss, the proposed SPTELM has the same computational complexity and is insensitive to noise, resampling stability and maintaining the sparsity of the solution. Further, we theoretically analyzed the sparsity, noise insensitivity and time complexity of SPTELM. Experimental results on multiple datasets demonstrate the noise insensitive, retains sparsity of the proposed method.

Laplacian twin extreme learning machine for semi-supervised classification

Abstract Twin extreme learning machine (TELM) is an efficient and effective method for pattern classification, based on widely known extreme learning machine (ELM). However, TELM is mainly used to deal with supervised learning problems. In this paper, we extend TELM to handle semi-supervised learning problems and propose a novel Laplacian twin extreme learning machine (LapTELM), which simultaneously trains two related and paired semi-supervised ELMs with two nonparallel separating planes for the final classification. The proposed method exploits the geometry structure property of the unlabeled samples and incorporates it as a manifold regularization term. This allows LapTELM to reap the benefits of fully exploring the plentiful unlabeled samples while retaining the learning ability and efficiency of TELM. Moreover, the paper shows that semi-supervised and supervised TELM can form an unified learning framework. Compared with several mainstream semi-supervised learning methods, the experimental results on the synthetic and several real-world data sets verify the effectiveness and efficiency of LapTELM.

Twin extreme learning machines for pattern classification

Extreme learning machine (ELM) is an efficient and effective learning algorithm for pattern classification. For binary classification problem, traditional ELM learns only one hyperplane to separate different classes in the feature space. In this paper, we propose a novel twin extreme learning machine (TELM) to simultaneously train two ELMs with two nonparallel classification hyperplanes. Specifically, TELM first utilizes the random feature mapping mechanism to construct the feature space, and then two nonparallel separating hyperplanes are learned for the final classification. For each hyperplane, TELM jointly minimizes its distance to one class and requires it to be far away from the other class. TELM incorporates the idea of twin support vector machine (TSVM) into the basic framework of ELM, thus TELM could have the advantages of the both algorithms. Moreover, compared to TSVM, TELM has fewer optimization constraint variables but with better classification performance. We also introduce a successive over-relaxation technique to speed up the training of our algorithm. Comprehensive experimental results on a large number of datasets verify the effectiveness and efficiency of TELM.

Prediction Interval Construction for Byproduct Gas Flow Forecasting Using Optimized Twin Extreme Learning Machine

Prediction of byproduct gas flow is of great significance to gas system scheduling in iron and steel plants. To quantify the associated prediction uncertainty, a two-step approach based on optimized twin extreme learning machine (ELM) is proposed to construct prediction intervals (PIs). In the first step, the connection weights of the twin ELM are pretrained using a pair of symmetric weighted objective functions. In the second step, output weights of the twin ELM are further optimized by particle swarm optimization (PSO). The objective function is designed to comprehensively evaluate PIs based on their coverage probability, width, and deviation. The capability of the proposed method is validated using four benchmark datasets and two real-world byproduct gas datasets. The results demonstrate that the proposed approach constructs higher quality prediction intervals than the other three conventional methods.