The oblique incident phase-shifting interferometric measurement method is an effective technique for assessing gear tooth form deviation. In the measurement process, the tooth flank incident angle is a crucial factor determining the quality of interferograms. Addressing the challenge in traditional optical path design, where the selection of the tooth flank incident angle encounters a trade-off among quality features such as measurable tooth area (MTA), interference fringe density (IFD) and interferogram compression ratio (ICR), this paper proposes an optimized method for the tooth flank incident angle. Firstly, a comprehensive evaluation model is established to calculate and analyze the MTA, IFD and ICR of the helical gear. Secondly, by integrating BP neural network and Multi-Objective Particle Swarm Optimization (MOPSO) algorithm, a tooth flank incident angle optimization method is devised to simultaneously optimize the measurement light incident angle γ and the rotation angle of the gear axis β. Finally, simulation analyses and physical experiments are employed to demonstrate the feasibility of the proposed method, compared to before optimization, the pseudocorrelation (PSD) value has increased by an average of 34%, and the residual points have decreased by an average of 80%, confirming its capability to improve quality interferogram and measurement accuracy.