AbstractWe study properties and algorithms of a minimization problem of the maximum generalized eigenvalue of symmetric-matrix-valued affine functions, which is nonsmooth and quasiconvex, and has application to eigenfrequency optimization of truss structures. We derive an explicit formula of the Clarke subdifferential of the maximum generalized eigenvalue and prove the maximum generalized eigenvalue is a pseudoconvex function, which is a subclass of a quasiconvex function, under suitable assumptions. Then, we consider smoothing methods to solve the problem. We introduce a smooth approximation of the maximum generalized eigenvalue and prove the convergence rate of the smoothing projected gradient method to a global optimal solution in the considered problem. Also, some heuristic techniques to reduce the computational costs, acceleration and inexact smoothing, are proposed and evaluated by numerical experiments.