Tool wear and breakage is one of the biggest obstacles for developing the unattended CNC machining. Especially when cutting the difficult-to-machining materials, the tool wear will be more rapid and the tool breakage becomes more frequent, which makes an effective tool wear and breakage monitoring very urgent. In the paper, a physical model-based tool wear and breakage monitoring method is proposed. Firstly, a physical model of milling force with the influence of cutter runout and tool wear is established. The spindle box vibration and cutting torque of spindle drive system induced by milling force have been clarified theoretically. Then, through the measurements of milling force, spindle box vibration and driving current, a tool wear monitoring method by extracting the comprehensive feature from the seven-channel specific cutting force coefficients (SCFCs) has been presented. In the method, a multi-parameter decoupling identification procedure for the cutter runout, tooth wear loss and respective SCFCs has been clarified. The force homogenization effect of multi-tooth with tool wear is found. Moreover, an efficient tool breakage monitoring method is further put forward, which incorporates the amplitude ratios of multi-channel data to form an indicator for judging the occurrence of tool damage. The generation mechanism of the sudden distortion on signal waveform resulting from tool breakage is also explained. Finally, long-time milling experiments with multiple groups of machining parameters have been carried out on the general and heavy-cutting machine tools to verify the validity of the proposed method. The verification results indicate that the tool wear and breakage monitoring method can well estimate the cutting status of cutter. The study can provide a useful research basis for the potential industrial application of tool wear and breakage monitoring technology.