Robust multitask learning in high dimensions under memory constraint

Abstract

AbstractWe investigate multitask learning in the context of multivariate linear regression with high dimensional covariates and heavy‐tailed noise, while under the constraint of limited memory. To tackle the computational complexity arising from the non‐smoothness of the quantile loss, we reformulate it as an equivalent least squares loss, which yields robust solutions even in the presence of heavy‐tailed noise. We incorporate a group lasso penalty into the quantile loss to produce sparse solutions, and an accelerated proximal sub‐gradient descent algorithm to speed up the computation while ensuring explicit forms for penalized solutions at each iteration. The proposed algorithm is general and can be applied to similar optimization problems. Moreover, we introduce a communication‐efficient distributed algorithm that guarantees optimal convergence rates after finite communication rounds in cases where computing resources such as memory are insufficient. We also study the theoretical properties of the resultant estimate and relax the widely used model selection consistency assumption on the initial estimate. We demonstrate the effectiveness of our proposal through extensive numerical studies.