In this paper, vibro-impact-induced gear whine radiated by full hybrid transmission equipped with a compound type power-split planetary gear train is investigated. For accurate simulations of the planetary gear set vibrations, an integrated dynamic model of the transmission is established using professional software MASTA, on which compliance effects of gear pairs, bearings, shafts, housing and planetary carrier are counted for. Numerical analyses are carried out for dynamic responses of the hybrid transmission. Bench tests are also conducted for the present model validation. Computational effort is made to recognize parametrically resonate patterns that cause gear whine in experiment. An optimal tooth modification design is presented for the real planetary gear set. It is shown by test results using the optimized planetary gear sets that gear whine responses of the gearboxes are considerably improved.