Slice inverse regression with score functions

Dmitry Babichev,Francis Bach

doi:10.1214/18-ejs1428

Abstract

We consider non-linear regression problems where we assume that the response depends non-linearly on a linear projection of the covariates. We propose score function extensions to sliced inverse regression problems, both for the first-order and second-order score functions. We show that they provably improve estimation in the population case over the non-sliced versions and we study finite sample estimators and their consistency given the exact score functions. We also propose to learn the score function as well, in two steps, i.e., first learning the score function and then learning the effective dimension reduction space, or directly, by solving a convex optimization problem regularized by the nuclear norm. We illustrate our results on a series of experiments.

Highlights

Non-linear regression and related problems such as non-linear classification are core important tasks in machine learning and statistics
We propose to extend slice inverse regression” (SIR) by the use of score functions to go beyond elliptically symmetric distributions, and we show that the new method combining SIR and score functions is formally better than the plain average derivative method” (ADE) method
MAVE SADE with known score SADE with unknown score estimation of this unknown k-dimensional space, which is often called the effective dimension reduction or e.d.r. space

Summary

Introduction

Non-linear regression and related problems such as non-linear classification are core important tasks in machine learning and statistics. We consider a random vector x ∈ Rd, a random response y ∈ R, and a regression model of the form y = f (x) + ε, (1.1). Which we want to estimate from n independent and identically distributed (i.i.d.) observations (xi, yi), i = 1, . Our goal is to estimate the function f from these data. A traditional key difficulty in this general regression problem is the lack of parametric assumptions regarding the functional form of f , leading to a problem of non-parametric regression. This is often tackled by searching implicitly or explicitly a function f within an infinite-dimensional vector space. The number n of observations for any level of precision is exponential in dimension

Objectives

Methods

Conclusion