The smoothness tool path is essential in five-axis high speed machining. However, the smoothness of the tool orientation in the part coordinate system cannot assure the smooth motions of the rotary axes of a five-axis machine tool because the conversion of tool posture from the part coordinate system to the machine coordinate system is nonlinear. To generate a tool path with smooth rotary motions of the five-axis machine tool, the tool path in this paper is represented with three spline curves: the cutter location curve and the two trajectory curves of the rotary axes of the machine tool, so that the rotary axes’ motions can be simultaneously smoothed. For a flank milling tool path, the geometric deviations between the envelope surface of the tool movement and the design surface should be respected while smoothing the tool path. The point-to-surface distance is adopted to calculate the geometric deviations, and then a smoothness optimization model is developed to smooth the rotary axes’ motions using the weighted least squares method. A numerical example of five-axis flank milling of a S-shape surface is given to confirm the validity and efficiency of the proposed approach.