Chatter is a very detrimental phenomenon occurring in milling process especially in thin-wall milling, which has been a limitation to achieve high productivity and good surface quality. The prediction of milling stability for chatter avoidance plays a key role in high performance milling. However, compared with the case of milling a part with a flexible cutter, the stability analysis and experimental validation for thin-wall milling are seldom provided. In this paper, an approach to obtaining the 3D stability lobe in thin-wall milling is proposed considering both the helix angle effect of cutter and the dynamic behavior of thin-walled part. A systematic cutting force model, which is capable of incorporating the helix angle effect and run-out effect of cutter, is first built. After the effective stiffness of the thin-walled part along the entire tool position is acquired by combining FEM with an impact experiment, the model of dynamic milling system with respect to the mode shape of thin-walled part is then developed. Based on the model, an extended high-order time domain method is subsequently utilized to generate the 3D stability lobe diagram considering helix angle effect. Finally, the proposed method is validated by the results of dedicated experiments.