Cubic boron nitride (CBN), with a hardness second only to diamond and an excellent chemical stability, is used to replace the role of diamond and machine precision/ultra-precision components in ferrous materials. However, the wear mechanisms of CBN tools and their wear-induced effects on machining system vibration seriously affect the machined surface quality. Therefore, this research focus on the wear characteristics of CBN tools in machining of NAK80. Based on the wear characteristics, the change in surface morphology and finish of the machined surface with tool wear was investigated. Meanwhile, the transformation of chip morphology at different wear stages was observed. Also, the effect of tool wear on the machining system vibration is analyzed by Butterworth filtering and Fourier transforming the cutting force signal. Research results show that the wear mechanisms of CBN cutting NAK80 are abrasive wear, oxidative wear and diffusive wear. Further, when the width of the flank wear of the CBN tool reaches 53 μm it causes multimodal vibration of the machining system, resulting in chattering patterns on the machined surface. As the width of the flank wear increased from 56.64 μm to 70.80 μm, the vibration frequency increased by 13.3 %. However, by reducing the feed rate from 5 μm/rev to 2 μm/rev, the vibration frequency can be reduced by 66.8 %. This study provides theoretical help in the research of wear of CBN cutting NAK80.