Line search and gradient method for solving constrained optimization problems

Abstract

The problem of optimizing functionals with linear or orthogonal constraints arises in many applications in engineering and applied sciences. In this paper, a unified framework involving constrained optimization using gradient descent in conjunction with exact or approximate line search is developed. In this framework, the optimality conditions are enforced at each step while optimizing along the direction of the gradient of the Lagrangian of the problem. Among many applications, this paper proposes learning algorithms which extract principal and minor components, reduced rank Wiener filter, and the first few minimum or maximum singular vectors of rectangular matrices. The main attraction of these algorithms is that they are matrix inverse free and thus are computationally efficient for large dimensional problems.