2 - Classical Optimization Techniques

Abstract

The classical techniques of optimization include methods of maxima and minima in differential calculus for solving continuous optimization problems. The theory of maxima and minima is universally applied in science and engineering. In statistics, such techniques are needed in estimation. Optimality criteria such as those of least squares, maximum likelihood, and minimum chi-square utilize classical techniques of maxima and minima. In other areas of statistics, such as in design of experiments, survey sampling, testing hypotheses, and regression analysis, extensive use of these techniques is made. This chapter describes the necessary and sufficient conditions for an optimum of a function defined on the Euclidean space Rn. The constrained optimization problem is considered later. One way to solve the constrained optimization problem is to convert it to an unconstrained problem through elimination of constraints, if possible. However, in many cases, the constraints may not be explicitly solvable to allow simple elimination, so other methods, such as the method of Lagrange multipliers, are used.