Orthogonal Least-squares Algorithm Research Articles

A Brand-New Simple, Fast, and Effective Residual-Based Method for Radial Basis Function Neural Networks Training

The radial basis function (RBF) neural network is a type of universal approximator, and has been widely used in various fields. Improving the training speed and compactness of RBF networks are critical for promoting their applications. In the present study, we propose a simple, fast, and effective RBF networks training method, which is based on the residual extreme points and their neighborhoods (thus called the REN method for short in this paper). The REN method calculates RBF centers and widths through a two-level iterative process, and realizes two main functionalities, namely 1) adding multiple centers within one pass through the whole data set, and 2) calculating RBF widths specifically for each center. The use of this algorithm does not need any parameter adjustments, and the models for approximation or classification can be obtained by only one run. The performance of the proposed REN algorithm is compared with the classic and powerful orthogonal least squares (OLS) algorithm. By reaching the same accuracies, the REN algorithm trains RBF networks 50 and 320 times faster, in the chirp (0˜50 Hz, 2 s, 1 kHz, 2001 samples) and two-dimensional peaks (2401 samples) signal approximation tasks respectively, than the OLS algorithm does, and the number of centers obtained by the REN algorithm is reduced by half. When incorporating the same number of centers, the REN algorithm achieves accuracies up to 3 orders of magnitude higher than the best results obtained by the OLS algorithm. In the classification task of a real discrete breast cancer data, both methods result in accuracies comparable to many existent methods, but the REN algorithm has the advantages of fast training speeds and no requirements for parameter adjustments. The REN algorithm proposed in this study may potentially be used for tasks with large scale of data or applications that require high model performances.

Spatial Construction for Modeling of Unknown Distributed Parameter Systems

Many industrial processes are distributed parameter systems that require complete spatial information for decisions and control. For modeling unknown distributed parameter systems (DPSs), a spatial construction method is proposed to preserve the spatial information between sensing locations. With the help of the spatial construction method, continuous spatial basis functions (SBFs) can be constructed to capture the spatial information lost in the time-space separation. The corresponding temporal dynamics can be identified using the generalized radial basis function network with the orthogonal least-squares (OLS) algorithm. After the time-space synthesis, the constructed spatiotemporal model can provide continuous modeling in the spatial domain with satisfactory performance. Convergence analysis proves that the proposed method can guarantee bounded errors. Finally, the experiments on a linear thermal process and a nonlinear catalytic process validate the effectiveness of the proposed method under limited sensors and its robustness when one of the sensors fails.

Temperature extraction method for infrared image of high voltage power equipment

An infrared temperature prediction method for power equipment is proposed based on the radial basis function (RBF) network optimized by quantum genetic algorithm (QGA) and orthogonal least squares algorithm (OLSA). The modified compound algorithm was used to optimize parameters of the RBF network. A temperature prediction model was established through the fitting of pixels and temperatures of the infrared image of an equipment. After image matching, the infrared temperature at a position can be directly obtained from the visible image. Meanwhile, we can also directly read temperature values of different positions from the infrared image and identify the corresponding positions in the visible image. Experimental results indicate that the algorithm proposed has a better prediction performance than the RBF network optimized by OLSA alone and by adaptive genetic algorithm (AGA) and OLSA. It improves the generalization capacity of RBF network, resulting a more stable input and a higher prediction accuracy. The algorithm proposed facilitates temperature analysis and condition-based maintenance for substations.

Active Users Detection Based on Orthogonal Least Squares

The Internet of Things (IoT) is the network of numerous physical users that are able to gather and share electronic information with each other. One of the main challenges of IoT is to detect the active users. We first employ the orthogonal least squares (OLS) algorithm to detect the active users. We then prove that under some conditions on the noise vector $\mathbf {v}$ , OLS can successfully detect all the $K$ active users from the linear model $\mathbf {y} = \mathbf {H} \mathbf {x}+\mathbf {v}$ in at most $K$ iterations provided that $\mathbf {H}$ satisfies the restricted isometry property with $\delta _{K + 1} . Numerical tests are also performed to show the efficiency and effectiveness of the OLS algorithm.

Perturbation Analysis of Orthogonal Least Squares

AbstractThe Orthogonal Least Squares (OLS) algorithm is an efficient sparse recovery algorithm that has received much attention in recent years. On one hand, this paper considers that the OLS algorithm recovers the supports of sparse signals in the noisy case. We show that the OLS algorithm exactly recovers the support of $K$-sparse signal $\boldsymbol{x}$ from $\boldsymbol{y}=\boldsymbol{\unicode[STIX]{x1D6F7}}\boldsymbol{x}+\boldsymbol{e}$ in $K$ iterations, provided that the sensing matrix $\boldsymbol{\unicode[STIX]{x1D6F7}}$ satisfies the restricted isometry property (RIP) with restricted isometry constant (RIC) $\unicode[STIX]{x1D6FF}_{K+1}<1/\sqrt{K+1}$, and the minimum magnitude of the nonzero elements of $\boldsymbol{x}$ satisfies some constraint. On the other hand, this paper demonstrates that the OLS algorithm exactly recovers the support of the best $K$-term approximation of an almost sparse signal $\boldsymbol{x}$ in the general perturbations case, which means both $\boldsymbol{y}$ and $\boldsymbol{\unicode[STIX]{x1D6F7}}$ are perturbed. We show that the support of the best $K$-term approximation of $\boldsymbol{x}$ can be recovered under reasonable conditions based on the restricted isometry property (RIP).

An Improved Algorithm based on MOLS for CS

The purpose of sparse recovery based on compressed sensing is to reconstruct sparse signals from linear compressed measurements. Greedy algorithm is often used to solve the inverse problem of underdetermined equations. Both Orthogonal Matching Pursuit(OMP) and Orthogonal Least Squares(OLS) greedy algorithms have been widely applied. Unlike the OMP algorithm, the first task of the OLS algorithm is to find the support set dropping the residual fastest. Multiple Orthogonal Least Squares(MOLS) algorithm adds the idea of multiple support set selection to the OLS algorithm, which greatly reduces the time complexity of the OLS algorithm, but needs to take the sparsity K as a priori condition. Based on this, a novel sparse recovery algorithm called Changing Stage Orthogonal Least Squares(CSOLS) is proposed in this paper. Compared with the MOLS algorithm, the most innovative feature of the CSOLS algorithm is the signal reconstruction ability without a priori information sparsity, finishing sparse recovery by conditionally broadening the search step. Compared with the MOLS algorithm and the traditional greedy algorithms with regard to the Frequency of Exact Reconstruction(FER) under the different sparsity and measurements, the CSOLS algorithm shows terrific recovery performance.

Accelerated orthogonal least-squares for large-scale sparse reconstruction

We study the problem of inferring a sparse vector from random linear combinations of its components. We propose the Accelerated Orthogonal Least-Squares (AOLS) algorithm that improves performance of the well-known Orthogonal Least-Squares (OLS) algorithm while requiring significantly lower computational costs. While OLS greedily selects columns of the coefficient matrix that correspond to non-zero components of the sparse vector, AOLS employs a novel computationally efficient procedure that speeds up the search by anticipating future selections via choosing L columns in each step, where L is an adjustable hyper-parameter. We analyze the performance of AOLS and establish lower bounds on the probability of exact recovery for both noiseless and noisy random linear measurements. In the noiseless scenario, it is shown that when the coefficients are samples from a Gaussian distribution, AOLS with high probability recovers a k-sparse m-dimensional sparse vector using O(klog⁡mk+L−1) measurements. Similar result is established for the bounded-noise scenario where an additional condition on the smallest nonzero element of the unknown vector is required. The asymptotic sampling complexity of AOLS is lower than the asymptotic sampling complexity of the existing sparse reconstruction algorithms. In simulations, AOLS is compared to state-of-the-art sparse recovery techniques and shown to provide better performance in terms of accuracy, running time, or both. Finally, we consider an application of AOLS to clustering high-dimensional data lying on the union of low-dimensional subspaces and demonstrate its superiority over existing methods.

Recovery of Sparse Signals Using Multiple Orthogonal Least Squares

Sparse recovery aims to reconstruct sparse signals from compressed linear measurements. In this paper, we propose a sparse recovery algorithm called multiple orthogonal least squares (MOLS), which extends the well-known orthogonal least squares (OLS) algorithm by allowing multiple $L$ indices to be selected per iteration. Owing to its ability to catch multiple support indices in each selection, MOLS often converges in fewer iterations and hence improves the computational efficiency over the conventional OLS algorithm. Theoretical analysis shows that MOLS ( $L > 1$ ) performs exact recovery of $K$ -sparse signals ( $K > 1$ ) in at most $K$ iterations, provided that the sensing matrix obeys the restricted isometry property with isometry constant $\delta _{LK} When $L = 1,$ MOLS reduces to the conventional OLS algorithm and our analysis shows that exact recovery is guaranteed under $\delta_{K +1} . This condition is nearly optimal with respect to $\delta _{K+1}$ in the sense that, even with a small relaxation (e.g., $\delta_{K + 1} = 1 / \sqrt{K}$ ), exact recovery with OLS may not be guaranteed. The recovery performance of MOLS in the noisy scenario is also studied. It is shown that stable recovery of sparse signals can be achieved with the MOLS algorithm when the signal-to-noise ratio scales linearly with the sparsity level of input signals.

Improved adaptive neural network control for humanoid robot hand in workspace

In order to improve the control performance of humanoid robot hand in workspace, an adaptive control method based on improved neural network was proposed. rival-penalized competitive learning and recursive orthogonal least-squares algorithms were applied to reinforce the learning capability of Gaussian radial basis function neural network and realize the real-time of neural network. Moreover, an improved neural network model for humanoid robot hand was established with Ge-Lee matrix and its operator, and a controller was designed. Finally, an example of humanoid robot hand finger was provided. The results showed that the proposed control method could effectively control the unknown nonlinear dynamic properties and load disturbances of the finger with a much smaller tracking errors.

A hybrid learning method composed by the orthogonal least-squares and the back-propagation learning algorithms for interval A2-C1 type-1 non-singleton type-2 TSK fuzzy logic systems

The purpose of this paper is to present a hybrid learning method for interval A2-C1 type-1 non-singleton type-2 TSK fuzzy logic system that uses the recursive orthogonal least-squares algorithm to tune the type-1 consequent parameters, and the back-propagation algorithm to tune the interval type-2 antecedent parameters. Based on the combination of these two training algorithms the new hybrid learning method changes the interval type-2 fuzzy model parameters adaptively and minimizes the proposed error function as the new type-1 non-singleton input---output data pairs are processed. Its antecedent sets are interval type-2 fuzzy sets, its consequent sets are type-1 fuzzy sets, and its inputs are type-1 non-singleton fuzzy numbers with uncertain standard deviations. Comparison with the non-hybrid interval A2-C1 type-1 non-singleton type-2 Takagi---Sugeno---Kang fuzzy logic system that only uses the back-propagation algorithm for both antecedent and consequent parameter's adaptation demonstrates that the proposed hybrid algorithm is a well-performing nonlinear adaptation that enables the interval type-2 fuzzy model to optimally match the nonlinear behavior of the process. The application of the interval type-2 fuzzy logic as adaptable predictor using the proposed hybrid learning method was constructed for the modeling and prediction of the transfer bar surface temperature in an industrial hot strip mill facility. Experimental results demonstrated that this method improves the temperature prediction performance of the interval A2-C1 type-1 non-singleton type-2 Takagi---Sugeno---Kang fuzzy logic system.

Estimation of the state of charge for a LFP battery using a hybrid method that combines a RBF neural network, an OLS algorithm and AGA

A method to accurately estimate the State-of-Charge (SOC) for LiFePO4 (LFB) batteries is urgently required, to address the issues associated with the increased use of LFP batteries for portable devices. This paper proposes a hybrid method that combines a radial basis function (RBF) neural network, an orthogonal least-squares (OLS) algorithm and an adaptive genetic algorithm (AGA) to estimate the SOC in discharging condition. The OLS algorithm determines the optimal number of nodes in the hidden layer of the RBF neural network. With an optimal RBF neural network structure, the AGA is then used to tune the parameters of the RBF neural network, including the centers and widths of RBF and the connection weights. The trained RBF neural network is then used to estimate the SOC of a LFP battery. In order to demonstrate the effectiveness of the proposed estimation method, the method is tested using LFP batteries under several different discharging conditions. The effectiveness of the proposed method is compared with the Coulomb integration method and a back propagation (BP) neural network. The results show that the proposed method outperforms the other methods.

System identification methods for metal rubber devices

Metal rubber (MR) devices, a new wire mesh material, have been extensively used in recent years due to several unique properties especially in adverse environments. Although many practical studies have been completed, the related theoretical research on metal rubber is still in its infancy. In this paper, a semi-constitutive dynamic model that involves nonlinear elastic stiffness, nonlinear viscous damping and bilinear hysteresis Coulomb damping is adopted to model MR devices. The model is first approximated by representing the bilinear hysteresis damping as Chebyshev polynomials of the first kind and then generalised by taking into account the effects of noises. A very efficient systematic procedure based on the orthogonal least squares (OLS) algorithm, the adjustable prediction error sum of squares (APRESS) criterion and the nonlinear model validity tests is proposed for model structure detection and parameter estimation of MR devices for the first time. The OLS algorithm provides a powerful tool to effectively select the significant model terms step by step, one at a time, by orthogonalising the associated terms and maximising the error reduction ratio, in a forward stepwise manner. The APRESS statistic regularises the OLS algorithm to facilitate the determination of the optimal number of model terms that should be included into the model. And whether the final identified dynamic model is adequate and acceptable is determined by the model validity tests. Because of the orthogonal property of the OLS algorithm, the selection of the dynamic model terms and noise model terms are totally decoupled and the approach also leads to a parsimonious model. Numerical ill-conditioning problems which can arise in the conventional least squares algorithm can be avoided as well. The methods of choosing the sampling interval for nonlinear systems are also incorporated into the approach. Finally by utilising the response of a cylindrical MR specimen, it is shown how the model structure can be detected in a practical application.

An RBF Neural Network Combined with OLS Algorithm and Genetic Algorithm for Short-Term Wind Power Forecasting

An accurate forecasting method for power generation of the wind energy conversion system (WECS) is urgently needed under the relevant issues associated with the high penetration of wind power in the electricity system. This paper proposes a hybrid method that combines orthogonal least squares (OLS) algorithm and genetic algorithm (GA) to construct the radial basis function (RBF) neural network for short-term wind power forecasting. The RBF neural network is composed of three-layer structures, which contain the input, hidden, and output layers. The OLS algorithm is used to determine the optimal number of nodes in a hidden layer of RBF neural network. With an appropriate RBF neural network structure, the GA is then used to tune the parameters in the network, including the centers and widths of RBF and the connection weights in second stage. To demonstrate the effectiveness of the proposed method, the method is tested on the practical information of wind power generation of a WECS installed in Taichung coast of Taiwan. Comparisons of forecasting performance are made to the persistence method and back propagation neural network. The good agreements between the realistic values and forecasting values are obtained; the test results show the proposed forecasting method is accurate and reliable.

The automatic model selection and variable kernel width for RBF neural networks

The Orthogonal Least Squares (OLS) algorithm has been extensively used in basis selection for RBF networks, but it is unable to perform model selection automatically because the tolerance ρ must be specified manually. This introduces noise and it is difficult to implement in the parametric complexity of real-time system. Therefore, a generic criterion that detects the optimum number of its basis functions is proposed. In this paper, not only the Bayesian Information Criterion (BIC) method, used for fitness calculation, is incorporated into the basis function selection process of the OLS algorithm for assigning its appropriate number, but also a new method is developed to optimize the widths of the Gaussian functions in order to improve the generalization performance. The augmented algorithm is employed to the Radial Basis Function Neural Networks (RBFNN) for known and unknown noise nonlinear dynamic systems and its performance is compared with the standard OLS; experimental results show that both the efficacy of BIC for fitness calculation and the importance of proper choice of basis function widths are significant.

A hybrid particle swarm optimization and its application in neural networks

In this paper, a novel particle swarm optimization model for radial basis function neural networks (RBFNN) using hybrid algorithms to solve classification problems is proposed. In the model, linearly decreased inertia weight of each particle (ALPSO) can be automatically calculated according to fitness value. The proposed ALPSO algorithm was compared with various well-known PSO algorithms on benchmark test functions with and without rotation. Besides, a modified fisher ratio class separability measure (MFRCSM) was used to select the initial hidden centers of radial basis function neural networks, and then orthogonal least square algorithm (OLSA) combined with the proposed ALPSO was employed to further optimize the structure of the RBFNN including the weights and controlling parameters. The proposed optimization model integrating MFRCSM, OLSA and ALPSO (MOA-RBFNN) is validated by testing various benchmark classification problems. The experimental results show that the proposed optimization method outperforms the conventional methods and approaches proposed in recent literature.

Real Time Economic and Emission Dispatch using RBF Network with OLS and MPSO Algorithms

paper proposes a new approach to real time economic and emission dispatch by using orthogonal least-squares (OLS) and modified particle swarm optimization (MPSO) algorithms to construct the radial basis function (RBF) network. The objectives considered are fuel cost and NOx/CO2 emissions. The RBF network is composed of input, hidden, and output layers. The OLS algorithm provides a simple and efficient means for fitting radial basis function networks. The MPSO algorithm is implemented to tune the parameters in the network, including the dilation and translation of RBF centers and the weights between the hidden and output layer. The proposed approach has been tested on the IEEE 30-bus six-generator system. Testing results indicate that the proposed approach can make a quick response and yield accurate Real time economic and emission solutions.

Optimisation of Ethylbenzene Conversion and Selectivity of p-diethylbenzene Using Differential Evolution Technique via OLS Modelling

This paper delves into the application of a robust optimisation tool, differential evolution (DE), to the disproportionation of ethylbenzene (EB) to produce the commercially important para-diethylbenzene (p-DEB). Through the application of this technique, EB conversion and selectivity of the p-DEB product are maximised against the three input variables of temperature, pressure and liquid hourly space velocity (LHSV), using a model. The catalytic process is represented by a multi-input multi-output (MIMO) orthogonal least-square (OLS) algorithm, using historic data. The optimised solutions are experimentally verified and show significant improvement in the p-DEB conversion and selectivity.

Radial Basis Function Network for Ore Grade Estimation

This paper highlights the performance of a radial basis function (RBF) network for ore grade estimation in an offshore placer gold deposit. Several pertinent issues including RBF model construction, data division for model training, calibration and validation, and efficacy of the RBF network over the kriging and the multilayer perceptron models have been addressed in this study. For the construction of the RBF model, an orthogonal least-square algorithm (OLS) was used. The efficacy of this algorithm was testified against the random selection algorithm. It was found that OLS algorithm performed substantially better than the random selection algorithm. The model was trained using training data set, calibrated using calibration data set, and finally validated on the validation data set. However, for accurate performance measurement of the model, these three data sets should have similar statistical properties. To achieve the statistical similarity properties, an approach utilizing data segmentation and genetic algorithm was applied. A comparative evaluation of the RBF model against the kriging and the multilayer perceptron was then performed. It was seen that the RBF model produced estimates with the R2 (coefficient of determination) value of 0.39 as against of 0.19 for the kriging and of 0.18 for the multilayer perceptron.

Identification of multiscale spatio-temporal dynamical systems using a wavelet multiresolution analysis

In this article, a new algorithm for the multiscale identification of spatio-temporal dynamical systems is derived. It is shown that the input and output observations can be represented in a multiscale manner based on a wavelet multiresolution analysis. The system dynamics at some specific scale of interest can then be identified using an orthogonal forward least-squares algorithm. This model can then be converted between different scales to produce predictions of the system outputs at different scales. The method can be applied to both multiscale and conventional spatio-temporal dynamical systems. For multiscale systems, the method can generate a parsimonious and effective model at a coarser scale while considering the effects from finer scales. Additionally, the proposed method can be used to improve the performance of the identification when the measurements are noisy. Numerical examples are provided to demonstrate the application of the proposed new approach.

Modelling customer satisfaction for product development using genetic programming

Product development involves several processes in which product planning is the first one. Several tasks normally are required to be conducted in the product-planning process and one of them is to determine settings of design attributes for products. Facing with fierce competition in marketplaces, companies try to determine the settings such that the best customer satisfaction of products could be obtained. To achieve this, models that relate customer satisfaction to design attributes need to be developed first. Previous research has adopted various modelling techniques to develop the models, but those models are not able to address interaction terms or higher-order terms in relating customer satisfaction to design attributes, or they are the black-box type models. In this paper, a method based on genetic programming (GP) is presented to generate models for relating customer satisfaction to design attributes. The GP is first used to construct branches of a tree representing structures of a model where interaction terms and higher-order terms can be addressed. Then an orthogonal least-squares algorithm is used to determine the coefficients of the model. The models thus developed are explicit and consist of interaction terms and higher-order terms in relating customer satisfaction to design attributes. A case study of a digital camera design is used to illustrate the proposed method.