The advance of underdetermined linear regression optimization based on implicit bias

Abstract

The gradient descent method has the characteristics of easy realization and simple structure. The traditional gradient descent method has many advantages, especially in solving convex optimization problems. In recent years, some researchers have noticed that the gradient descent algorithm is helpful to solve the problem of underdetermined linear regression optimization. Therefore, in order to explore the specific relationship between gradient descent and under-determined linear regression optimization, this paper focuses on a case with a unique finite root loss function and discusses in detail the working principles of the natural-gradient-descent, mirror-descent, and gradient-descent in this case, and proves the auxiliary effect of gradient descent on under-determined linear regression optimization theoretically. How to use the characteristics of the gradient descent algorithm to design a suitable model, and how do we use the idea of the gradient descent algorithm to find an answer for problem of under determined linear regression optimization in the practical application will be the future research work to continue to advance.