Self-excited chatter vibrations are the most detrimental for the quality and safety of the machining operations. To reduce the undesired vibrations, the dynamic behaviors of the machine tool-workpiece system and the chatter stability models need to be studied. In this paper, gear hobbing is studied using analytical and experimental approaches. Firstly, based on the chatter stability theory in relation to milling, analytical model of chatter stability in gear hobbing process is derived. Secondly, a new approach is presented to identify dynamic modal parameters using operational modal analysis (OMA). Using natural excitation technique (NExT) and auto regressive moving average (ARMA) model time series analysis method. At last, the prediction capability is verified using machining experiments, and the results show that the stability diagrams is of great potential to supply guidance for the processing parameters development.