Smoothing spline estimation of generalised varying-coefficient mixed model

Abstract

The generalised varying-coefficient model with longitudinal data faces a challenge that the data are correlated, as multiple observations are measured from each individual. In this article we consider the generalised varying-coefficient mixed model (GVCMM) which uses a varying-coefficient model to fit mean functions, while accounting for overdispersion and correlation by adding random effects. Smoothing splines are used to estimate the smooth but arbitrary nonparametric coefficient functions. The usually intractable integration involved in evaluating the quasi-likelihood function is approximated by the Laplace method. This suggests that the GVCMM can be approximately represented by a generalised linear mixed model. Hence, the smoothing parameters and the variance components can be estimated by using the restricted maximum log-likelihood (REML) approach, where the smoothing parameters are treated as an extra variance component vector. We illustrate the performance of the proposed method through some simulation and an application to a real data set.