Abstract

We introduce a flexible optimization model for maximum likelihood-type estimation (M-estimation) that encompasses and generalizes a large class of existing statistical models, including Huber’s concomitant M-estimator, Owen’s Huber/Berhu concomitant estimator, the scaled lasso, support vector machine regression, and penalized estimation with structured sparsity. The model, termed perspective M-estimation, leverages the observation that convex M-estimators with concomitant scale as well as various regularizers are instances of perspective functions, a construction that extends a convex function to a jointly convex one in terms of an additional scale variable. These nonsmooth functions are shown to be amenable to proximal analysis, which leads to principled and provably convergent optimization algorithms via proximal splitting. We derive novel proximity operators for several perspective functions of interest via a geometrical approach based on duality. We then devise a new proximal splitting algorithm to solve the proposed M-estimation problem and establish the convergence of both the scale and regression iterates it produces to a solution. Numerical experiments on synthetic and real-world data illustrate the broad applicability of the proposed framework.

Highlights

  • High-dimensional regression methods play a pivotal role in modern data analysis

  • A large body of statistical work has focused on estimating regression coefficients under various structural assumptions, such as sparsity of the regression vector [36]

  • A more fundamental objective in statistical inference is the estimation of both location and scale of the statistical model from the data

Read more

Summary

Introduction

High-dimensional regression methods play a pivotal role in modern data analysis. A large body of statistical work has focused on estimating regression coefficients under various structural assumptions, such as sparsity of the regression vector [36]. The objective function, which we refer to as the homoscedastic Huber M-estimator function, is jointly convex in both b and scalar σ, and amenable to global optimization Under suitable assumptions, this estimator can identify outliers o and can estimate a scale that is proportional to the diagonal entries of C in the homoscedastic case, In [2], it was proposed that joint convex optimization of regression vector and standard deviation may be advantageous in sparse linear regression. Using geometrical insights revealed by the dual problem, we derive new proximity operators for several perspective functions, including the generalized scaled lasso, the generalized Huber, the abstract Vapnik, and the generalized Berhu function This enables the development of a unifying algorithmic framework for global optimization of the proposed model using modern splitting techniques. Numerical experiments on synthetic and real-world data illustrate the applicability of the framework

Proximity operators of perspective functions
Notation and background on convex analysis
Perspective functions
Examples
Optimization model and examples
Algorithm
Numerical experiments
Numerical illustrations on low-dimensional data
Numerical illustrations for correlated designs and outliers
Findings
Robust regression for gene expression data
Full Text
Published version (Free)

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call