Surface defects which are caused by diamond fly-cutting can degrade the optical characteristics of KDP crystals by inducing the laser damage and resultantly shortening the service life when working in the condition of extremely high-power laser irradiations. The ball-end milling repair can totally remove the defects and correspondingly enhance the ability of the KDP crystals to resist laser damage. However, because of the existence of pull-milling, push-milling, down-milling, and up-milling in the complex machining trajectory, it is of great challenge for soft-brittle KDP crystals to achieve full plastic domain milling of repair points for obtaining ultra-smooth repaired surface. In this study, theoretical models of the maximum uncut chip thickness (UCT) considering the defect are constructed in the various milling modes. The cutting force and surface morphology affected by milling modes on the defective surface are simulated using a three-dimensional (3D) finite element model. Combined with the simulations, micro-milling tests are employed to confirm the influence of milling modes on the material removal mechanisms for defective KDP crystals by cutting force, surface morphology, surface roughness and specific cutting energy (SCE). It is found that defects could reduce the maximum UCT to distinct degrees in the various milling modes. Defects have a greater influence on the maximum UCT in the push-milling mode than that in the pull-milling mode, while the influence is the same in the up-milling and down-milling modes. Interestingly, the ductile removal mode occurs on the both defect-free and defective surfaces in the pull-milling mode when the feed per tooth (fz) is 0.5 µm/z. In contrast, the brittle-ductile transition caused by surface defects occurs in the other three milling modes. Furthermore, defects would lead to severer size effect in the all milling modes. When the fz is 0.1 µm/z, for the defective surface, the SCE is larger and the ploughing effect is more significant in the push-milling mode compared to the pull-milling mode. More importantly, as a result of the difficulty of chip discharge, the SCE is greater in the up-milling mode than in the down-milling mode on the defective surface. This work reveals the influence mechanism of milling modes on the material removal process for defective KDP crystals and offers a theoretical foundation and technical value for improving the repaired surface quality of functional KDP crystals served as the optical devices.