Despite various vibration-assisted cutting techniques have been utilized to increase the machining performances of brittle materials, the highly dynamic variations of cutting process quantities lead to the complicated brittle-ductile transition (BDT) mechanism and the formation of undesired vibration marks on the machined surfaces. In this paper, a novel ultra-precision vibration-assisted cutting process, named trapezoidal modulation diamond cutting (TMDC), is firstly proposed for ductile-regime machining of brittle materials with significantly increased BDT cutting depth. By imposing a dedicate trapezoidal locus to the diamond tool, the unique invariable uncut chip thickness and cutting states were achieved for realizing stable vibration-assisted cutting without the formations of vibration marks. Systematic cutting experiments of KDP crystals were carried out to comprehensively investigate the influences of different process parameters on the machining performances of TMDC process. In addition, the underlying mechanisms of machining performance improvements have been discussed under the different combinations of process parameters, based on which the guidelines for optimal process parameter selection are given for increasing the BDT cutting depths. The outcomes of this study contribute to not only improving the ductile machining efficiency and machining quality of KDP crystals, but also help to deepen the understandings of BDT mechanism during vibration-assisted diamond cutting of common brittle materials.