Polygonal shape transformation processes observed in liposomes [H. Hotani, J. Mol. Biol. 178, 113 (1984)] have been analyzed on the basis of the Helfrich spontaneous curvature model. First, a mathematical solution for the biconcave axisymmetric vesicles [H. Naito, M. Okuda, and Z. Ou-Yang, Phys. Rev. E 47, 2304 (1993)] is derived, and then the second variation of the shape energy of the slightly deformed vesicles from the biconcave shape is calculated. After the minimization of the shape energy, it is found that the biconcave vesicle is transformed into elliptical, triangular, square, pentagonal, or other polygonal shapes above the threshold osmotic pressure difference. At a constant value of \ensuremath{\Delta}V/${\mathit{V}}_{0}$, the osmotic pressure is found to be a monotonic increasing function of m, except for m=2, where ${\mathit{V}}_{0}$ is the initial volume of the biconcave vesicle, \ensuremath{\Delta}V is the change in the volume of the deformed vesicle, and m denotes mth polygonal deformation. It is shown that the experimental results of the polygonal shape transformation in liposomes can be well explained by the present theoretical predictions. \textcopyright{} 1996 The American Physical Society.