The formation of serrated chips is the typical characteristic of high-speed machining. In this paper, a new prediction method based on Williams’ model was presented to give the critical cutting condition for serrated chips formation. The serrated chips formation mechanism under different cooling conditions was analyzed by finite element method. Results indicated that the cutting forces abruptly reduces when the serrated chip forms, the chips serrated degree (Gs) under various cooling conditions (i.e., dry, minimum quantity lubrication (MQL), supercritical CO2 (scCO2), and scCO2-MQL) was described by the following order: scCO2(Gs)>scCO2-MQL(Gs)>MQL(Gs)>dry(Gs). The adiabatic shear owing to the cutting heat and crack initiated from the chip free surface under scCO2 cooling promoted the serrated chips formation.