Regressor Vector Research Articles

Highly efficient hierarchical online nonlinear regression using second order methods

We introduce highly efficient online nonlinear regression algorithms that are suitable for real life applications. We process the data in a truly online manner such that no storage is needed, i.e., the data is discarded after being used. For nonlinear modeling we use a hierarchical piecewise linear approach based on the notion of decision trees where the space of the regressor vectors is adaptively partitioned based on the performance. As the first time in the literature, we learn both the piecewise linear partitioning of the regressor space as well as the linear models in each region using highly effective second order methods, i.e., Newton–Raphson Methods. Hence, we avoid the well known over fitting issues by using piecewise linear models, however, since both the region boundaries as well as the linear models in each region are trained using the second order methods, we achieve substantial performance compared to the state of the art. We demonstrate our gains over the well known benchmark data sets and provide performance results in an individual sequence manner guaranteed to hold without any statistical assumptions. Hence, the introduced algorithms address computational complexity issues widely encountered in real life applications while providing superior guaranteed performance in a strong deterministic sense.

Conditional heavy-tail behavior with applications to precipitation and river flow extremes

This article deals with the right-tail behavior of a response distribution $$F_Y$$ conditional on a regressor vector $${\mathbf {X}}={\mathbf {x}}$$ restricted to the heavy-tailed case of Pareto-type conditional distributions $$F_Y(y| {\mathbf {x}})=P(Y\le y| {\mathbf {X}}={\mathbf {x}})$$ , with heaviness of the right tail characterized by the conditional extreme value index $$\gamma ({\mathbf {x}})>0$$ . We particularly focus on testing the hypothesis $${\mathscr {H}}_{0,tail}: \gamma ({\mathbf {x}})=\gamma _0$$ of constant tail behavior for some $$\gamma _0>0$$ and all possible $${\mathbf {x}}$$ . When considering $${\mathbf {x}}$$ as a time index, the term trend analysis is commonly used. In the recent past several such trend analyses in extreme value data have been published, mostly focusing on time-varying modeling of location or scale parameters of the response distribution. In many such environmental studies a simple test against trend based on Kendall’s tau statistic is applied. This test is powerful when the center of the conditional distribution $$F_Y(y|{\mathbf {x}})$$ changes monotonically in $${\mathbf {x}}$$ , for instance, in a simple location model $$\mu ({\mathbf {x}})=\mu _0+x\cdot \mu _1$$ , $${\mathbf {x}}=(1,x)'$$ , but the test is rather insensitive against monotonic tail behavior, say, $$\gamma ({\mathbf {x}})=\eta _0+x\cdot \eta _1$$ . This has to be considered, since for many environmental applications the main interest is on the tail rather than the center of a distribution. Our work is motivated by this problem and it is our goal to demonstrate the opportunities and the limits of detecting and estimating non-constant conditional heavy-tail behavior with regard to applications from hydrology. We present and compare four different procedures by simulations and illustrate our findings on real data from hydrology: weekly maxima of hourly precipitation from France and monthly maximal river flows from Germany.

Differential games, partial-state stabilization, and model reference adaptive control

In this paper, we develop a unified framework to find state-feedback control laws that solve two-player zero-sum differential games over the infinite time horizon and guarantee partial-state asymptotic stability of the closed-loop system. Partial-state asymptotic stability is guaranteed by means of a Lyapunov function that is positive definite and decrescent with respect to part of the system state. The existence of a saddle point for the system׳s performance measure is guaranteed by the fact that this Lyapunov function satisfies a partial differential equation that corresponds to a steady-state form of the Hamilton–Jacobi–Isaacs equation. In the second part of this paper, we show how our differential game framework can be applied not only to solve pursuit-evasion and robust optimal control problems, but also to assess the effectiveness of a model reference adaptive control law. Specifically, the model reference adaptive control architecture is designed to guarantee satisfactory trajectory tracking for uncertain nonlinear dynamical systems, whose matched nonlinearities are captured by the regressor vector. By modeling matched and unmatched nonlinearities, which are not captured by the regressor vector, as the pursuer׳s and evader׳s control inputs in a differential game, we provide an explicit characterization of the system׳s uncertainties that do not disrupt the trajectory tracking capabilities of the adaptive controller. Two numerical examples illustrate the applicability of our results.

Highly efficient nonlinear regression for big data with lexicographical splitting

This paper considers the problem of online piecewise linear regression for big data applications. We introduce an algorithm, which sequentially achieves the performance of the best piecewise linear (affine) model with optimal partition of the space of the regressor vectors in an individual sequence manner. To this end, our algorithm constructs a class of $$2^D$$ sequential piecewise linear models over a set of partitions of the regressor space and efficiently combines them in the mixture-of-experts setting. We show that the algorithm is highly efficient with computational complexity of only $$O(mD^2)$$ , where m is the dimension of the regressor vectors. This efficient computational complexity is achieved by efficiently representing all of the $$2^D$$ models using a “lexicographical splitting graph.” We analyze the performance of our algorithm without any statistical assumptions, i.e., our results are guaranteed to hold. Furthermore, we demonstrate the effectiveness of our algorithm over the well-known data sets in the machine learning literature with computational complexity fraction of the state of the art.

Universal Nonlinear Regression on High Dimensional Data Using Adaptive Hierarchical Trees

We study online sequential regression with nonlinearity and time varying statistical distribution when the regressors lie in a high dimensional space. We escape the curse of dimensionality by tracking the subspace of the underlying manifold using a hierarchical tree structure. We use the projections of the original high dimensional regressor space onto the underlying manifold as the modified regressor vectors for modeling of the nonlinear system. By using the proposed algorithm, we reduce the computational complexity to the order of the depth of the tree and the memory requirement to only linear in the intrinsic dimension of the manifold. The proposed techniques are specifically applicable to high dimensional streaming data analysis in a time varying environment. We demonstrate the significant performance gains in terms of mean square error over the other state of the art techniques through simulated as well as real data.

Feasible Parameter Set Approximation for Linear Models with Bounded Uncertain Regressors

Nonconvex feasible parameter sets are encountered in set membership identification whenever the regressor vector is affected by bounded uncertainty. This occurs for example when considering standard output error models, or when the available measurements are provided by binary or quantized sensors. In this paper, a unifying framework is proposed to deal with several identification problems involving a nonconvex feasible parameter set and a procedure is proposed for approximating the minimum volume orthotope containing the feasible set. The procedure exploits different relaxations for autoregressive and input parameters, based on the solution of a sequence of linear programming problems. The proposed technique is shown to provide tight bounds in some special cases. Moreover, it is extended to cope with bounds not aligned with the parameter coordinates, in order to obtain polytopic approximations of the feasible set. A number of numerical tests on randomly generated models and data sets demonstrates the accuracy of the computed set approximations.

Constrained Optimal Input Signal Design for Data-Centric Estimation Methods

This technical note examines the design of constrained input signals for data-centric estimation methods which systematically generate a local function approximation from a database of regressors at a current operating point. The proposed method addresses the optimal distribution of regressor vectors under constraints for a linear time-invariant (LTI) system. The resulting nonconvex optimization problems are solved using semidefinite relaxation methods. Numerical examples illustrate the benefits and usefulness of the proposed input signal design formulations.

Support kernels regression for NARMA system identification (SKR<SUB align="right">NARMA)

This paper deals with the identification of nonlinear systems using multi-kernel approach. First, we have improved the support vector regression (SVR) method in order to identify nonlinear complex system. Our idea consists of dividing the regressor vector in several blocks, and, for each one a kernel function is used. This blockwise SVR approach is called support kernel regression (SKR). Furthermore, we have proposed two methods, SKR(rbf-lin) and SKR(rbf-rbf). Second, the SKR approach is improved to deal with the problem of NARMA system identification. Therefore, a new method called support kernel regression for NARMA model (SKRNARMA) is suggested. The basic idea is to consider the terms of auto-correlation and cross-correlation of the nonlinearity of input output discrete time processes, and for every term a kernel function is used. An example of MIMO system is presented for qualitative comparison with the classical SVR approach based on a single kernel function. The results reveal the accuracy and the robustness of the obtained model based on our proposed (SKRNARMA)-based approach.

Closed-Loop Identification Applied to a DC Servomechanism: Controller Gains Analysis

Usually, when parameter identification is applied, there are some gains related to the identification algorithm whose value must be carefully adjusted in order to obtain a good performance of the algorithm. However, when performing closed loop identification, there are some other constants that in general are not taken into account for the identification algorithm: the controller gains, which may appear inside the identification algorithm, specifically in the regressor vector, which is very important for the parameter convergence according to the persistence of excitation condition. Therefore, the effect of these gains on the estimated parameters should be analyzed so that better estimates can be obtained. This paper addresses the behavior of the parameter estimates for a closed-loop identification methodology applied to a DC servomechanism with a bounded perturbation signal and a PD controller. It is shown that, with this perturbation, the parameter estimates converge to a region whose size can be modified not only by varying the identification algorithm gains but also by modifying the P and D controller gains in a suitable way.

Estimating Smooth Structural Change in Cointegration Models

This paper studies nonlinear cointegration models in which the structural coefficients may evolve smoothly over time. These time-varying coefficient functions are well-suited to many practical applications and can be estimated conveniently by nonparametric kernel methods. It is shown that the usual asymptotic methods of kernel estimation completely break down in this setting when the functional coefficients are multivariate. The reason for this breakdown is a kernel-induced degeneracy in the weighted signal matrix associated with the nonstationary regressors, a new phenomenon in the kernel regression literature. Some new techniques are developed to address the degeneracy and resolve the asymptotics, using a path-dependent local coordinate transformation to re-orient coordinates and accommodate the degeneracy. The resulting asymptotic theory is fundamentally different from the existing kernel literature, giving two different limit distributions with different convergence rates in the different directions (or combinations) of the (functional) parameter space. Both rates are faster than the usual rate for nonlinear models with smoothly changing coefficients and local stationarity. Hence two types of super-consistency apply in nonparametric kernel estimation of time-varying coefficient cointegration models. The higher rate of convergence lies in the direction of the nonstationary regressor vector at the local coordinate point. The lower rate lies in the degenerate directions but is still super-consistent for nonparametric estimators. In addition, local linear methods are used to reduce asymptotic bias and a fully modified kernel regression method is proposed to deal with the general endogenous nonstationary regressor case. Simulations are conducted to explore the finite sample properties of the methods and a practical application is given to examine time varying empirical relationships involving consumption, disposable income, investment and real interest rates.

NARX modelling of a lithium iron phosphate battery used for electrified vehicle simulation

Non-linear autoregressive exogenous (NARX) black-box modelling methodology is presented to model a lithium iron phosphate battery for system-level electrified vehicle simulation. The NARX model regressor vector is carefully chosen for dynamically representing the battery voltage and its dependence on state of charge (SOC) and charging/discharging current. Three types of non-linearity estimators, i.e., wavelet network, one-layer sigmoid network, and binary tree partition, are investigated and compared. The prediction error minimisation by means of the advanced adaptive Gaussian-Newton search algorithm is applied to implement the model parameterisation. The impact of the number of basis function units on the model accuracy and complexity is also studied. A preferred NARX model is determined, according to a comprehensive evaluation of model accuracies in two different datasets and complexity. A comparison between the preferred NARX model and a conventionally statically non-linear black-box battery model is made.

Estimating Smooth Structural Change in Cointegration Models

This paper studies nonlinear cointegration models in which the structural coefficients may evolve smoothly over time. These time-varying coefficient functions are well-suited to many practical applications and can be estimated conveniently by nonparametric kernel methods. It is shown that the usual asymptotic methods of kernel estimation completely break down in this setting when the functional coefficients are multivariate. The reason for this breakdown is a kernel-induced degeneracy in the weighted signal matrix associated with the nonstationary regressors, a new phenomenon in the kernel regression literature. Some new techniques are developed to address the degeneracy and resolve the asymptotics, using a path-dependent local coordinate transformation to re-orient coordinates and accommodate the degeneracy. The resulting asymptotic theory is fundamentally different from the existing kernel literature, giving two different limit distributions with different convergence rates in the different directions (or combinations) of the (functional) parameter space. Both rates are faster than the usual (?nh) rate for nonlinear models with smoothly changing coefficients and local stationarity. Hence two types of super-consistency apply in nonparametric kernel estimation of time-varying coefficient cointegration models. The higher rate of convergence (n?h) lies in the direction of the nonstationary regressor vector at the local coordinate point. The lower rate (nh) lies in the degenerate directions but is still super-consistent for nonparametric estimators. In addition, local linear methods are used to reduce asymptotic bias and a fully modified kernel regression method is proposed to deal with the general endogenous nonstationary regressor case. Simulations are conducted to explore the finite sample properties of the methods and a practical application is given to examine time varying empirical relationships involving consumption, disposable income, investment and real interest rates.

General Framework for Latent Variable Model Inversion for the Design and Manufacturing of New Products

Latent variable regression model (LVRM) inversion is a useful tool to support the development of new products and their manufacturing conditions. The objective of the model inversion exercise is that of finding the best combination of regressors (e.g., raw material properties, process parameters) that are needed to obtain a desired response (e.g., product quality) from the model. Each of the published applications where model inversion has been applied utilizes a tailored approach to achieve the inversion, given the specific objectives and needs. These approaches range from the direct inversion of the LVRM to the formulation of an objective function that is optimized using nonlinear programming. In this paper we present a framework that aims to give a holistic view of the optimization formulations that can arise from the need to invert an LVRM. The different sets of equations that become relevant (either as a term within the objective function or as a constraint) are discussed, and an example of these scenarios is also provided. Additional to the formulation of the different scenarios and their objective functions, this work proposes a new metric (the P2 statistic) to cross-validate the ability of the model to reconstruct the regressor vector (analogous to the Q2 statistic aimed at the predictability of the response). This new metric comes from the need to not only predict the response from the regressor, but to also reconstruct the regressors from the scores values. In this context, a discussion is provided on the effect of uncertainty in the reconstruction of the regressor (the actual design) as these values are normally given upstream as targets to the supplier of materials, or as set points to the process.

Adaptive feedforward compensation algorithms for AVC systems in the presence of a feedback controller

In Jacobson, Johnson, Mc Cormick, and Sethares (2001) and Landau, Alma, and Airimiţoaie (2011) adaptation algorithms taking into account the “positive” feedback coupling arising in most of the active noise and vibration control systems have been proposed and analyzed. The stability of the system requires satisfaction of a positive real condition through an appropriate filtering of the regressor vector. It is shown in this note that the presence in addition of a feedback controller on one hand strongly influences the positive real conditions for stability and the structure of the filter to be used in the algorithm and on the other hand improves significantly the performance of the system. Experimental results obtained on an active vibration control (AVC) system clearly illustrate the benefit of using a hybrid adaptive feedforward + feedback approach.

A robust adaptive control architecture for disturbance rejection and uncertainty suppression with L ∞ transient and steady‐state performance guarantees

SUMMARYIn this paper, a new adaptive control architecture for linear and nonlinear uncertain dynamical systems is developed to address the problem of high‐gain adaptive control. Specifically, the proposed framework involves a new and novel controller architecture involving a modification term in the update law that minimizes an error criterion involving the distance between the weighted regressor vector and the weighted system error states. This modification term allows for fast adaptation without hindering system robustness. In particular, we show that the governing tracking closed‐loop system error equation approximates a Hurwitz linear time‐invariant dynamical system with input–output signals. This key feature of our framework allows for robust stability analysis of the proposed adaptive control law using system theory. We further show that by properly choosing the design parameters in the modification term, we can guarantee a desired bandwidth of the adaptive controller, guaranteed transient closed‐loop performance, and an a priori characterization of the size of the ultimate bound of the closed‐loop system trajectories. Several illustrative numerical examples are provided to demonstrate the efficacy of the proposed approach. Copyright © 2012 John Wiley & Sons, Ltd.

Identification and Estimation of Average Partial Effects in "Irregular" Correlated Random Coefficient Panel Data Models

In this paper we study identification and estimation of a correlated random coefficients (CRC) panel data model. The outcome of interest varies linearly with a vector of endogenous regressors. The coefficients on these regressors are heterogenous across units and may covary with them. We consider the average partial effect (APE) of a small change in the regressor vector on the outcome (cf. Chamberlain (1984), Wooldridge (2005a)). Chamberlain (1992) calculated the semiparametric efficiency bound for the APE in our model and proposed a √N-consistent estimator. Nonsingularity of the APE's information bound, and hence the appropriateness of Chamberlain's (1992) estimator, requires (i) the time dimension of the panel (T) to strictly exceed the number of random coefficients (p) and (ii) strong conditions on the time series properties of the regressor vector. We demonstrate irregular identification of the APE when T = p and for more persistent regressor processes. Our approach exploits the different identifying content of the subpopulations of stayers—or units whose regressor values change little across periods—and movers—or units whose regressor values change substantially across periods. We propose a feasible estimator based on our identification result and characterize its large sample properties. While irregularity precludes our estimator from attaining parametric rates of convergence, its limiting distribution is normal and inference is straightforward to conduct. Standard software may be used to compute point estimates and standard errors. We use our methods to estimate the average elasticity of calorie consumption with respect to total outlay for a sample of poor Nicaraguan households.

Least squares support kernel machines (LS-SKM) for identification

This paper presents a novel approach for non-linear systems identification called ‘least squares support kernel machines (LS-SKM)’. Instead of using a least squares support vector machines (LS-SVM) with a single kernel function, the proposed approach combines several kernels in order to take advantage of their performances and also reflects the fact that practical learning problems often involve multiple, heterogeneous data sources. The idea is to divide the regressor vector in several regressor vectors, and, for each vector a kernel function is used. The choice of kernel function and the corresponding parameters is an important task which is related to the non-linear system degrees. A constrained particle swarm optimisation (CPSO) is used to give solution for the determination of optimised kernel parameters. Two examples are presented for qualitative comparison with the classical LS-SVM. The results reveal the accuracy and the robustness of the obtained model based on our proposed hybrid method.

Determinant Efficiencies in Ill‐Conditioned Models

The canonical correlations between subsets of OLS estimators are identified with design linkage parameters between their regressors. Known collinearity indices are extended to encompass angles between each regressor vector and remaining vectors. One such angle quantifies the collinearity of regressors with the intercept, of concern in the corruption of all estimates due to ill‐conditioning. Matrix identities factorize a determinant in terms of principal subdeterminants and the canonical Vector Alienation Coefficients between subset estimators—by duality, the Alienation Coefficients between subsets of regressors. These identities figure in the study of D and Ds as determinant efficiencies for estimators and their subsets, specifically, Ds‐efficiencies for the constant, linear, pure quadratic, and interactive coefficients in eight known small second‐order designs. Studies on D‐ and Ds‐efficiencies confirm that designs are seldom efficient for both. Determinant identities demonstrate the propensity for Ds‐inefficient subsets to be masked through near collinearities in overall D‐efficient designs.

Curse of dimensionality and related issues in nonparametric functional regression

Recently, some nonparametric regression ideas have been extended to the case of functional regression. Within that framework, the main concern arises from the infinite dimensional nature of the explanatory objects. Specifically, in the classical multivariate regression context, it is well-known that any nonparametric method is affected by the so-called “curse of dimensionality”, caused by the sparsity of data in high-dimensional spaces, resulting in a decrease in fastest achievable rates of convergence of regression function estimators toward their target curve as the dimension of the regressor vector increases. Therefore, it is not surprising to find dramatically bad theoretical properties for the nonparametric functional regression estimators, leading many authors to condemn the methodology. Nevertheless, a closer look at the meaning of the functional data under study and on the conclusions that the statistician would like to draw from it allows to consider the problem from another point-of-view, and to justify the use of slightly modified estimators. In most cases, it can be entirely legitimate to measure the proximity between two elements of the infinite dimensional functional space via a semi-metric, which could prevent those estimators suffering from what we will call the “curse of infinite dimensionality”.

On‐line parameter estimation for a class of time‐varying continuous systems with bounded disturbances

AbstractIn this paper, we proposed an on‐line parameter estimation algorithm for a class of time‐varying continuous systems with bounded disturbance. In this method, a novel polynomial approximator with a bounded regressor vector is constructed and utilized to approximate the time‐varying parameters. The direct least‐squares algorithm is employed to acquire the on‐line estimates, so that several useful properties of the direct estimation, such as fast convergence and robustness to the bounded disturbance, are reflected in our method. We have proved that the estimation error of this method is bounded. Furthermore, the bound on the Euclidean norm of the estimation error is derived. The simulation results demonstrate that this method can provide accurate estimates of time‐varying parameters even under the influence of bounded disturbance. Copyright © 2010 John Wiley & Sons, Ltd.