Backfitting Algorithm Research Articles

A Sequential Stein's Method for Faster Training of Additive Index Models.

The additive index models (AIMs) can be viewed as a kind of artificial neural networks based on nonparametric activation or so-called ridge functions. Recently, they are shown to achieve enhanced explainability after incorporating various interpretability constraints. However, the training of AIMs by either the backfitting algorithm or the joint stochastic optimization is known to be very slow for especially high dimensional inputs. In this article, we propose a novel sequential approach based on the celebrated Stein's lemma. The proposed SeqStein method can successfully decouple the training of AIMs into two separable steps, namely, the following: 1) Stein's estimation of the projection indices and 2) nonparametric estimation of ridge functions using the smoothing splines. We show through numerical experiments that the SeqStein algorithm is not only more efficient for training AIMs, but also inclined to produce more interpretable models that have smooth ridge functions with sparse and nearly orthogonal projection indices.

Generalized Additive Spatial Smoothing (GASS): A Multiscale Regression Framework for Modeling Neighborhood Effects Across Spatial Supports

A new technique called generalized additive spatial smoothing (GASS) is introduced for modeling neighborhood effects within a regression framework. GASS has a number of desirable features, namely that it provides a data-driven mechanism for endogenously selecting neighborhoods based on a spatial scale hyperparameter. By allowing different scale hyperparameters to be selected for different relationships in the model, the technique is inherently multiscale and allows neighborhoods to vary by relationship. In addition, GASS includes a measure of uncertainty associated with each scale hyperparameter. These characteristics make it attractive for modeling phenomena where proximity might be an important aspect of a process, especially when a clear definition of proximity is not immediately available. Through multiscale data-driven spatial smoothing, GASS conducts a form of change of support and therefore also facilitates the incorporation of data from diverse sources. Finally, the technique is flexible and can be adapted and expanded with relative ease because it builds on generalized additive modeling. After providing an overview of the methodology, including a modified backfitting algorithm for calibration, a simulation experiment is described and an empirical example modeling bike-share usage is presented. The simulation results indicate that GASS can generally produce reliable results pertaining to both the regression coefficients and scale hyperparameters, and the results from the empirical example demonstrate that the GASS approach provides a better model fit and captures relationships that might otherwise be obfuscated. Overall, these results highlight the potential of the GASS framework and the importance of measuring multiscale neighborhood effects.

Local Influence for the Thin-Plate Spline Generalized Linear Model

Thin-Plate Spline Generalized Linear Models (TPS-GLMs) are an extension of Semiparametric Generalized Linear Models (SGLMs), because they allow a smoothing spline to be extended to two or more dimensions. This class of models allows modeling a set of data in which it is desired to incorporate the non-linear joint effects of some covariates to explain the variability of a certain variable of interest. In the spatial context, these models are quite useful, since they allow the effects of locations to be included, both in trend and dispersion, using a smooth surface. In this work, we extend the local influence technique for the TPS-GLM model in order to evaluate the sensitivity of the maximum penalized likelihood estimators against small perturbations in the model and data. We fit our model through a joint iterative process based on Fisher Scoring and weighted backfitting algorithms. In addition, we obtained the normal curvature for the case-weight perturbation and response variable additive perturbation schemes, in order to detect influential observations on the model fit. Finally, two data sets from different areas (agronomy and environment) were used to illustrate the methodology proposed here.

Semiparametric Analysis of Covariance Model for a Crossover Design with Carryover Effects

We postulate a semiparametric mixed analysis of the covariance model for a crossover design with carryover effects. The model is estimated via a hybrid of restricted maximum likelihood estimation and smoothing splines regression embedded into the backfitting algorithm. To mitigate heterogeneity in experimental settings common in clinical trials, responses are adjusted using a nonparametric function of the covariates. A bootstrap-based test for random treatment effect using the adjusted responses is also proposed. Simulation studies indicate that the bootstrap-based test for variance components is correctly sized. The test is also powerful and is relatively robust to the magnitude of variance in the alternative hypothesis. Furthermore, the test is advantageous over classical analysis of covariance in the presence of design and measurement issues in experiments.

Generalized Bayesian Additive Regression Trees Models: Beyond Conditional Conjugacy

Bayesian additive regression trees have seen increased interest in recent years due to their ability to combine machine learning techniques with principled uncertainty quantification. The Bayesian backfitting algorithm used to fit BART models, however, limits their application to a small class of models for which conditional conjugacy exists. In this article, we greatly expand the domain of applicability of BART to arbitrary generalized BART models by introducing a very simple, tuning-parameter-free, reversible jump Markov chain Monte Carlo algorithm. Our algorithm requires only that the user be able to compute the likelihood and (optionally) its gradient and Fisher information. The potential applications are very broad; we consider examples in survival analysis, structured heteroskedastic regression, and gamma shape regression.

A robust estimation based on penalised regularisation for the varying-coefficient additive model

To effectively handle functional data and longitudinal data, we propose a robust estimation approach based on penalised regularisation with the framework of the varying-coefficient additive model. Our method utilises an iterative backfitting algorithm that leverages splines to estimate the component functions. The proposed algorithm showcases both stability and effectiveness in model fitting, particularly when dealing with scenarios involving high-dimensional covariates and high levels of noise. We establish a comprehensive theoretical framework for the proposed method. Furthermore, we investigate the asymptotic normality of the estimators and construct asymptotic confidence bands to quantify the uncertainty of the estimates. To validate the performance of our method, we conduct extensive simulation studies and compare the results with those obtained using alternative approaches. Additionally, we apply our proposed method to a real dataset focussed on young high school dropouts in the US, demonstrating its practical applicability.

Modeling Environmental Pollution Using Varying-Coefficients Quantile Regression Models under Log-Symmetric Distributions

Many phenomena can be described by random variables that follow asymmetrical distributions. In the context of regression, when the response variable Y follows such a distribution, it is preferable to estimate the response variable for predictor values using the conditional median. Quantile regression models can be employed for this purpose. However, traditional models do not incorporate a distributional assumption for the response variable. To introduce a distributional assumption while preserving model flexibility, we propose new varying-coefficients quantile regression models based on the family of log-symmetric distributions. We achieve this by reparametrizing the distribution of the response variable using quantiles. Parameter estimation is performed using a maximum likelihood penalized method, and a back-fitting algorithm is developed. Additionally, we propose diagnostic techniques to identify potentially influential local observations and leverage points. Finally, we apply and illustrate the methodology using real pollution data from Padre Las Casas city, one of the most polluted cities in Latin America and the Caribbean according to the World Air Quality Index Ranking.

Modified Local Linear Estimators in Partially Linear Additive Models with Right-Censored Data Based on Different Censorship Solution Techniques.

This paper introduces a modified local linear estimator (LLR) for partially linear additive models (PLAM) when the response variable is subject to random right-censoring. In the case of modeling right-censored data, PLAM offers a more flexible and realistic approach to the estimation procedure by involving multiple parametric and nonparametric components. This differs from the widely used partially linear models that feature a univariate nonparametric function. The LLR method is employed to estimate unknown smooth functions using a modified backfitting algorithm, delivering a non-iterative solution for the right-censored PLAM. To address the censorship issue, three approaches are employed: synthetic data transformation (ST), Kaplan-Meier weights (KMW), and the kNN imputation technique (kNNI). Asymptotic properties of the modified backfitting estimators are detailed for both ST and KMW solutions. The advantages and disadvantages of these methods are discussed both theoretically and practically. Comprehensive simulation studies and real-world data examples are conducted to assess the performance of the introduced estimators. The results indicate that LLR performs well with both KMW and kNNI in the majority of scenarios, along with a real data example.

On the local modeling of count data: multiscale geographically weighted Poisson regression

A recent addition to the suite of techniques for local statistical modeling is the implementation of the multiscale geographically weighted regression (MGWR), a multiscale extension to geographically weighted regression (GWR). Using a back-fitting algorithm, MGWR relaxes the restrictive assumption in GWR that all processes being modeled operate at the same spatial scale and allows the estimation of a unique indicator of scale, the bandwidth, for each process. However, the current MGWR framework is limited to use with continuous data making it unsuitable for modeling data that do not typically exhibit a Gaussian distribution. This study expands the application of the MGWR framework to scenarios involving discrete response outcomes (count data following a Poisson’s distribution). Use of this new MGWR Poisson regression (MGWPR) model is demonstrated with a simulated data set and then with COVID-19 case counts within New York City at the zip code level. The results from the simulated data underscore the superiority of the MGWPR model in effectively capturing spatial processes that influence count data patterns, particularly those operating across diverse spatial scales. For empirical data, the results reveal significant spatial variations in relationships between socio-ecological factors and COVID-19 cases – variations often missed by traditional ‘global’ models.

Nonparametric and Semiparametric Quantile Regression via a New MM Algorithm

Quantile regression is a popular method with a wide range of scientific applications, but the computation for quantile regression is challenging. Hunter and Lange proposed an MM algorithm for solving optimization problems in parametric quantile regression models. For nonparametric and semiparametric quantile regression, their algorithm can be applied to estimate unknown quantile functions in a pointwise manner. However, the resulting estimates may suffer from drawbacks like nonsmooth with discontinuous points and unstable at extreme quantile levels. To remedy the above issues, we propose a new MM algorithm and show that it yields continuous, smoother, and faster estimated quantile functions. We systematically study the new MM algorithm using the local linear quantile regression model. We prove that the proposed algorithm preserves the monotone descent property in an asymptotic sense. We then extend it to some popular nonparametric and semiparametric quantile regression models. For semiparametric models, we propose new efficient backfitting algorithms based on the new MM algorithm. Compared to traditional backfitting algorithms, the new procedures can significantly reduce computational costs for fully iterative backfitting. The performance of the proposed algorithms is demonstrated via extensive simulation studies and a real data example. Supplementary materials for this article are available online.

A survey of smoothing techniques based on a backfitting algorithm in estimation of semiparametric additive models

AbstractThis paper aims to present an overview of Semiparametric additive models. An estimation of the finite‐parameters of semiparametric regression models that involve additive nonparametric components is explained, including their historical background. In addition, three different smoothing techniques are considered in order to show the working procedures of the estimators and to explore their statistical properties: smoothing splines, kernel smoothing and local linear regression. These methods are compared with respect to both their theoretical and practical behaviors. A simulation study and a real data example are carried out to reveal the performances of the three methods. Accordingly, the advantages and disadvantages of each method regarding semiparametric additive models are presented based on their comparative scores using determined evaluation metrics for loss of information.This article is categorized under: Statistical Learning and Exploratory Methods of the Data Sciences > Modeling Methods Statistical and Graphical Methods of Data Analysis > Multivariate Analysis Statistical Models > Semiparametric Models

Modeling Air Pollution Using Partially Varying Coefficient Models with Heavy Tails

The increase in air pollution levels in recent decades around the world has caused a negative impact on human health. A recent investigation by the World Health Organization indicates that nine out of ten people on the planet breathe air containing high levels of pollutants and seven million people die each year from this cause. This problem is present in several cities in South America due to dangerous levels of particulate matter present in the air, particularly in the winter period, making it a public health problem. Santiago in Chile and Lima in Peru are among the ten cities with the highest levels of air pollution in South America. The location, climate, and anthropogenic conditions of these cities generate critical episodes of air pollution, especially in the coldest months. In this context, we developed a semiparametric model to predict particulate matter levels as a function of meteorological variables. For this, we discuss estimation and diagnostic procedures using a Student’s t-based partially varying coefficient model. Parameter estimation is performed through the penalized maximum likelihood method using smoothing splines. To obtain the parameter estimates, we present a weighted back-fitting algorithm implemented in R-project and Matlab software. In addition, we developed local influence techniques that allowed us to evaluate the potential influence of certain observations in the model using four different perturbation schemes. Finally, we applied the developed model to real data on air pollution and meteorological variables in Santiago and Lima.

Sparse Spatial Autoregressive and Spatio-temporal Models for COVID-19 Incidence in the Philippines

Philippine COVID-19 data have so many gaps resulting from lack of mass testing, late reporting of test results, and unreported cases – leading to too much noise and sparsity. A sparse spatial autoregressive model linking COVID-19 incidence and mortality rates to the healthcare system, demographic and economic indicators, disease prevalence, vaccination, urbanity, and environmental factors is proposed. The model allows for irregular spatial units accounting for temporal dependencies within a neighborhood was estimated using the Cochranne-Orcutt procedure embedded into the backfitting algorithm. Daily COVID-19 cases and deaths across provinces and cities in the National Capital Region (NCR) from 01 April 2020–15 September 2021 show a significant association of COVID-19 prevalence rate with the number of health workers, revenue of the local government unit (LGU), and prevalence of tuberculosis (TB). On the other hand, COVID-19 mortality rates are associated with the number of health workers, number of licensed COVID-19 testing laboratories, number of cities in an LGU, revenue of the LGU, prevalence rate of cancer, and prevalence rate of TB. The models emphasize the importance of resources available in the LGU that can boost the capabilities of the health care system. Pre-existing health conditions (co-morbidities) in the communities also determine the prevalence and mortality rates of COVID-19 in the Philippines.

Statistical supervised learning with engineering data: a case study of low frequency noise measured on semiconductor devices

Our practical motivation is the analysis of potential correlations between spectral noise current and threshold voltage from common on-wafer MOSFETs. The usual strategy leads to the use of standard techniques based on Normal linear regression easily accessible in all statistical software (both free or commercial). However, these statistical methods are not appropriate because the assumptions they lie on are not met. More sophisticated methods are required. A new strategy based on the most novel nonparametric techniques which are data-driven and thus free from questionable parametric assumptions is proposed. A backfitting algorithm accounting for random effects and nonparametric regression is designed and implemented. The nature of the correlation between threshold voltage and noise is examined by conducting a statistical test, which is based on a novel technique that summarizes in a color map all the relevant information of the data. The way the results are presented in the plot makes it easy for a non-expert in data analysis to understand what is underlying. The good performance of the method is proven through simulations and it is applied to a data case in a field where these modern statistical techniques are novel and result very efficient.

Backfitting for large scale crossed random effects regressions

Regression models with crossed random effect errors can be very expensive to compute. The cost of both generalized least squares and Gibbs sampling can easily grow as N3/2 (or worse) for N observations. Papaspiliopoulos, Roberts and Zanella (Biometrika 107 (2020) 25–40) present a collapsed Gibbs sampler that costs O(N), but under an extremely stringent sampling model. We propose a backfitting algorithm to compute a generalized least squares estimate and prove that it costs O(N). A critical part of the proof is in ensuring that the number of iterations required is O(1), which follows from keeping a certain matrix norm below 1−δ for some δ>0. Our conditions are greatly relaxed compared to those for the collapsed Gibbs sampler, though still strict. Empirically, the backfitting algorithm has a norm below 1−δ under conditions that are less strict than those in our assumptions. We illustrate the new algorithm on a ratings data set from Stitch Fix.

Scalable logistic regression with crossed random effects

The cost of both generalized least squares (GLS) and Gibbs sampling in a crossed random effects model can easily grow faster than N3∕2 for N observations. Ghosh et al. (2022) develop a backfitting algorithm that reduces the cost to O(N). Here we extend that method to a generalized linear mixed model for logistic regression. We use backfitting within an iteratively reweighted penalized least squares algorithm. The specific approach is a version of penalized quasi-likelihood due to Schall (1991). A straightforward version of Schall’s algorithm would also cost more than N3∕2 because it requires the trace of the inverse of a large matrix. We approximate that quantity at cost O(N) and prove that this substitution makes an asymptotically negligible difference. Our backfitting algorithm also collapses the fixed effect with one random effect at a time in a way that is analogous to the collapsed Gibbs sampler of Papaspiliopoulos et al. (2020). We use a symmetric operator that facilitates efficient covariance computation. We illustrate our method on a real dataset from Stitch Fix. By properly accounting for crossed random effects we show that a naive logistic regression could underestimate sampling variances by several hundred fold.

A Robust Segmented Mixed Effect Regression Model for Baseline Electricity Consumption Forecasting

Renewable energy production has been surging around the world in recent years. To mitigate the increasing uncertainty and intermittency of the renewable generation, proactive demand response algorithms and programs are proposed and developed to further improve the utilization of load flexibility and increase the efficiency of power system operation. One of the biggest challenges to efficient control and operation of demand response resources is how to forecast the baseline electricity consumption and estimate the load impact from demand response resources accurately. In this paper, we propose a mixed effect segmented regression model and a new robust estimate for forecasting the baseline electricity consumption in Southern California, USA, by combining the ideas of random effect regression model, segmented regression model, and the least trimmed squares estimate. Since the log-likelihood of the considered model is not differentiable at breakpoints, we propose a new backfitting algorithm to estimate the unknown parameters. The estimation performance of the new estimation procedure has been demonstrated with both simulation studies and the real data application for the electric load baseline forecasting in Southern California.

Additive regression for non-Euclidean responses and predictors

Additive regression is studied in a very general setting where both the response and predictors are allowed to be non-Euclidean. The response takes values in a general separable Hilbert space, whereas the predictors take values in general semimetric spaces, which covers a very wide range of nonstandard response variables and predictors. A general framework of estimating additive models is presented for semimetric space-valued predictors. In particular, full details of implementation and the corresponding theory are given for predictors taking values in Hilbert spaces and/or Riemannian manifolds. The existence of the estimators, convergence of a backfitting algorithm, rates of convergence and asymptotic distributions of the estimators are discussed. The finite sample performance of the estimators is investigated by means of two simulation studies. Finally, three data sets covering several types of non-Euclidean data are analyzed to illustrate the usefulness of the proposed general approach.

Modeling conditional reference regions: Application to glycemic markers.

Many clinical decisions are taken based on the results of continuous diagnostic tests. Usually, only the results of one single test is taken into consideration, the interpretation of which requires a reference range for the healthy population. However, the use of two different tests, can be necessary in the diagnosis of certain diseases. This obliges a bivariate reference region be available for their interpretation. It should also be remembered that reference regions may depend on patient variables (eg, age and sex) independent of the suspected disease. However, few proposals have been made regarding the statistical modeling of such reference regions, and those put forward have always assumed a Gaussian distribution, which can be rather restrictive. The present work describes a new statistical method that allows such reference regions to be estimated with no insistence on the results being normally distributed. The proposed method is based on a bivariate location-scale model that provides probabilistic regions covering a specific percentage of the bivariate data, dependent on certain covariates. The reference region is estimated nonparametrically and the nonlinear effects of continuous covariates via polynomial kernel smoothers in additive models. The bivariate model is estimated using a backfitting algorithm, and the optimal smoothing parameters of the kernel smoothers selected by cross-validation. The model performed satisfactorily in simulation studies under the assumption of non-Gaussian conditions. Finally, the proposed methodology was found to be useful in estimating a reference region for two continuous diagnostic tests for diabetes (fasting plasma glucose and glycated hemoglobin), taking into account the age of the patient.

Hierarchical Total Variations and Doubly Penalized ANOVA Modeling for Multivariate Nonparametric Regression

For multivariate nonparametric regression, functional analysis of variance (ANOVA) modeling aims to capture the relationship between a response and covariates by decomposing the unknown function into various components, representing main effects, two-way interactions, etc. Such an approach has been pursued explicitly in smoothing spline ANOVA modeling and implicitly in various greedy methods such as MARS. We develop a new method for functional ANOVA modeling, based on doubly penalized estimation using total-variation and empirical-norm penalties, to achieve sparse selection of component functions and their basis functions. For this purpose, we formulate a new class of hierarchical total variations, which measures total variations at different levels including main effects and multi-way interactions, possibly after some order of differentiation. Furthermore, we derive suitable basis functions for multivariate splines such that the hierarchical total variation can be represented as a regular Lasso penalty, and hence we extend a previous backfitting algorithm to handle doubly penalized estimation for ANOVA modeling. We present extensive numerical experiments on simulations and real data to compare our method with existing methods including MARS, tree boosting, and random forest. The results are very encouraging and demonstrate notable gains from our method in prediction or classification accuracy and simplicity of the fitted functions. Supplementary materials for this article are available online.