In addition to the dynamic characteristics of the machine tool, cutting conditions significantly influences the vibration of the cutting system and mid-frequency waviness of the workpiece surface in ultra-precision machining (UPM). In this work, the effects of cutting conditions on machining vibration and surface waviness were investigated by high-speed micro-planning experiments. The origin of vibration and waviness were elucidated by frequency-domain processing of the vibration signal and surface waviness analysis. The dynamic characteristics of machine tool are determined using modal test and analysis. The results revealed that the cutting force has different contributions to the workpiece vibration and tool vibration. In the time domain, distinct differences can be found in stability and attenuation of various waviness components. The divergent effects of cutting force components in different vibration forms are confirmed through a novel cutting force model of micro-planing. The investigation can further reveal the formation mechanisms of cutting system vibration and surface waviness during the micro-cutting process, which can provide practical guidance for suppressing mid-frequency waviness on machined surfaces.