Kernel-based Regularization Method Research Articles

On kernel design for regularized non-causal system identification

Through one decade’s development, the kernel-based regularization method (KRM) has become a complement to the classical maximum likelihood/prediction error method and an emerging new system identification paradigm. One recent example is its application in the non-causal system identification, and the key issue lies in the design and analysis of kernels for non-causal systems. In this paper, we develop systematic ways to deal with this issue. In particular, we first introduce the guidelines for kernel design and then extend the system theoretic framework to design the so-called non-causal simulation-induced (NCSI) kernel, and we also study its structural properties, including stability and semiseparability. Finally, we consider some special cases of the NCSI kernel and show their advantage over the existing kernels through numerical simulations.

Kernel-based regularized iterative learning control of repetitive linear time-varying systems

For data-driven iterative learning control (ILC) methods, both the model estimation and controller design problems are converted to parameter estimation problems for some chosen model structures. It is well-known that if the model order is not chosen carefully, models with either large variance or large bias would be resulted, which is one of the obstacles to further improve the modeling and tracking performances of data-driven ILC in practice. An emerging trend in the system identification community to deal with this issue is using regularization instead of the statistical tests, e.g., AIC, BIC, and one of the representatives is the so-called kernel-based regularization method (KRM). In this paper, we integrate KRM into data-driven ILC to handle a class of repetitive linear time-varying systems, and moreover, we show that the proposed method has ultimately bounded tracking error in the iteration domain. The numerical simulation results show that in contrast with the least squares method and some existing data-driven ILC methods, the proposed one can give faster convergence speed, better accuracy and robustness in terms of the tracking performance.

Clustering of Linear Time-Invariant Systems: A Bayesian Nonparametric Method

In this paper, we consider the clustering of linear time-invariant (LTI) systems according to the similarity of dynamics. There are two main difficulties for clustering of LTI systems: one is to determine the labels of LTI systems and the number of clusters, and another one is to build models of LTI systems. To address these two difficulties simultaneously, a Bayesian nonparametric method based on Dirichlet process mixture models (DPMM) and kernel-based regularization method is proposed. To be specific, the Dirichlet process mixtures of Bayesian linear regression models (DPM-BLRM) is proposed to determine the label of each LTI system according to the probability that the system belongs to each cluster, and adapt the number of clusters to the complexity of data. In addition, DPM-BLRM incorporates the prior knowledge of LTI systems to deal with the possible large variance of model parameters estimation when the collected data is short or has low signal-to-noise ratio, where the carrier of prior knowledge is kernel matrix. Then Markov chain Monte Carlo (MCMC) method is used for Bayesian inference for the DPM-BLRM. The simulation results illustrate the efficiency of the proposed method.

Kernel-based Regularized Iterative Learning Control of Repetitive Linear Time-varying Systems

Abstract The selections for the model orders and the number of controller parameters have not been discussed for many data-driven iterative learning control (ILC) methods. If they are not chosen carefully, the estimated model and designed controller will lead to either large variance or large bias. In this paper we try to use the kernel-based regularization method (KRM) to handle the model estimation problem and the controller design problem for unknown repetitive linear time-varying systems. In particular, we have used the diagonal correlated kernel and the marginal likelihood maximization method for the two problems. Numerical simulation results show that smaller mean square errors for each time instant are obtained by using the proposed ILC method in comparison with an existing data-driven ILC approach.

A new multiple kernel-based regularization method for identification of delay linear dynamic systems

Model parameter estimation, model order selection, variable selection and time delay estimation are four important issues that receive increasing interests in dynamic system identification. However, previous work did not solve the four issues simultaneously. Motivated by the multiple kernel-based regularization method (MKRM), this paper proposes a new multiple kernel-based regularization method for joint model parameter estimation, model order selection, variable selection and time delay estimation for delay linear dynamic systems, referred to as the MKRM-D. Then, an efficient iterative reweighted algorithm is derived to solve the resulting difference of convex functions programming (DCP) problem. In addition, by exploiting the structure of the objective function in each iteration of this algorithm, the alternating direction method of multipliers (ADMM) is employed to decompose the centralized problem into a series of independent subproblems with lower variable dimension, which can be solved in a parallel and distributed manner. The performance of the proposed method is demonstrated by numerical experiments using both synthetic and real data.

On input design for regularized LTI system identification: Power-constrained input

Input design is an important issue for classical system identification methods but has not been investigated for the kernel-based regularization method (KRM) until very recently. In this paper, we consider the input design problem of KRMs for LTI system identification. Different from the recent result, we adopt a Bayesian perspective and in particular make use of scalar measures (e.g., the A-optimality, D-optimality, and E-optimality) of the Bayesian mean square error matrix as the design criteria subject to power-constraint on the input. Instead of solving the optimization problem directly, we propose a two-step procedure. In the first step, by making suitable assumptions on the unknown input, we construct a quadratic map (transformation) of the input such that the transformed input design problems are convex, and the global minima of the transformed input design problem can thus be found efficiently by applying well-developed convex optimization software packages. In the second step, we derive the characterization of the optimal input based on the global minima found in the first step by solving the inverse image of the quadratic map. In addition, we derive analytic results for some special types of kernels, which provide insights on the input design and also its dependence on the kernel structure.

On asymptotic properties of hyperparameter estimators for kernel-based regularization methods

The kernel-based regularization method has two core issues: kernel design and hyperparameter estimation. In this paper, we focus on the second issue and study the properties of several hyperparameter estimators including the empirical Bayes (EB) estimator, two Stein’s unbiased risk estimators (SURE) (one related to impulse response reconstruction and the other related to output prediction) and their corresponding Oracle counterparts, with an emphasis on the asymptotic properties of these hyperparameter estimators. To this goal, we first derive and then rewrite the first order optimality conditions of these hyperparameter estimators, leading to several insights on these hyperparameter estimators. Then we show that as the number of data goes to infinity, the two SUREs converge to the best hyperparameter minimizing the corresponding mean square error, respectively, while the more widely used EB estimator converges to another best hyperparameter minimizing the expectation of the EB estimation criterion. This indicates that the two SUREs are asymptotically optimal in the corresponding MSE senses but the EB estimator is not. Surprisingly, the convergence rate of two SUREs is slower than that of the EB estimator, and moreover, unlike the two SUREs, the EB estimator is independent of the convergence rate of ΦTΦ∕N to its limit, where Φ is the regression matrix and N is the number of data. A Monte Carlo simulation is provided to demonstrate the theoretical results.

On kernel design for regularized LTI system identification

There are two key issues for the kernel-based regularization method: one is how to design a suitable kernel to embed in the kernel the prior knowledge of the LTI system to be identified, and the other one is how to tune the kernel such that the resulting regularized impulse response estimator can achieve a good bias–variance tradeoff. In this paper, we focus on the issue of kernel design. Depending on the type of the prior knowledge, we propose two methods to design kernels: one is from a machine learning perspective and the other one is from a system theory perspective. We also provide analysis results for both methods, which not only enhances our understanding for the existing kernels but also directs the design of new kernels.

The Cauchy problem of coupled elliptic sine–Gordon equations with noise: Analysis of a general kernel-based regularization and reliable tools of computing

Developments in numerical methods for problems governed by nonlinear partial differential equations underpin simulations with sound arguments in diverse areas of science and engineering. In this paper, we explore the regularization method for the coupled elliptic sine–Gordon equations along with Cauchy data. The system of equations originates from the static case of the coupled hyperbolic sine–Gordon equations modeling the coupled Josephson junctions in superconductivity, and so far it addresses the Josephson π-junctions. In general, the Cauchy problem is not well-posed, and herein the Hadamard-instability occurs drastically. Generalizing the kernel-based regularization method, we propose a stable approximate solution. Confirmed by the error estimate, this solution strongly converges to the exact solution in L2-norm. The main concern of this paper is also with the way to compute the regularized solution formed by an alike integral equation. We employ the proposed techniques that successfully approximated the highly oscillatory integral, and apply the Picard-like iteration to organize an efficient and reliable tool of computations. The results are viewed as the improvement as well as the generalization of many previous works. The paper is also accompanied by a numerical example that demonstrates the potential of this idea.

On kernel structures for regularized system identification (I): a machine learning perspective

A center issue for system identification is how to get a model estimate that achieves a good balance between the data fit and the model complexity. For the recently introduced kernel-based regularization method for linear system identification, the problem becomes first how to design a suitable kernel structure and second how to determine a right kernel among the kernel structure. In this paper and its companion one, we will focus on the issue of kernel structure design. Depending on the type of the prior knowledge, we provide two different ways: from a machine learning perspective or from a system theory perspective. We will focus on the first perspective here. In particular, we show that both the stable spline kernel and the diagonal correlated kernel belong to the class of the so-called exponentially convex locally stationary (ECLS) kernels. This finding motivates to construct ECLS or LS kernels for this regularization method in different ways, e.g., based on carefully designed state space models.

System Identification Via Sparse Multiple Kernel-Based Regularization Using Sequential Convex Optimization Techniques

Model estimation and structure detection with short data records are two issues that receive increasing interests in System Identification. In this paper, a multiple kernel-based regularization method is proposed to handle those issues. Multiple kernels are conic combinations of fixed kernels suitable for impulse response estimation, and equip the kernel-based regularization method with three features. First, multiple kernels can better capture complicated dynamics than single kernels. Second, the estimation of their weights by maximizing the marginal likelihood favors sparse optimal weights, which enables this method to tackle various structure detection problems, e.g., the sparse dynamic network identification and the segmentation of linear systems. Third, the marginal likelihood maximization problem is a difference of convex programming problem. It is thus possible to find a locally optimal solution efficiently by using a majorization minimization algorithm and an interior point method where the cost of a single interior-point iteration grows linearly in the number of fixed kernels. Monte Carlo simulations show that the locally optimal solutions lead to good performance for randomly generated starting points.

Constructive state space model induced kernels for regularized system identification

There are two key issues for the kernel-based regularization method: the kernel structure design and the hyper-parameter estimation. In this contribution, we introduce a new family of kernel structures based on state space models. It has more flexible and more general structure, and includes some of stable spline kernels and diagonal correlated kernels as special cases. We also tested a different method for the hyper-parameter estimation by maximizing a profile marginal likelihood and examined three methods dealing with the initialization. Monte Carlo simulations show that the tested kernels are on the average a bit better than the tuned correlated kernel and the profile marginal likelihood maximization and the pre-windowing method work well for hyper-parameter estimation and initialization.