A figure-of-merit (FOM) of an auto-stereoscopic display system is introduced and adopted to characterize the system performance. This FOM takes into account of the ratio of the signal to the noise arising from the crosstalk from the adjacent channels as well as the brightness uniformity of viewing areas; hence, it is directly related to the glasses-free 3D viewing comfort. With a steadily improving FOM as a target, the genetic algorithm is applied to optimize the optical system, giving rise to substantially improved characteristics of an auto-stereoscopic display system. The numerical simulation is verified with an experiment of a multi-view auto-stereoscopic display unit. It is shown that the system can provide a high fidelity of the display effect with the crosstalk ratio being reduced from around 5% to nearly 1%, which is a very low value obtainable for an auto-stereoscopic system.