In order to solve the problem of deviation between actual and theoretical machining paths due to the presence of rotation axis in five-axis machining, an interpolation algorithm based on the optimization of swing cutter trajectory and the method of corresponding nonlinear error compensation are proposed. Taking A-C dual rotary table five-axis machine tool as an example, the forward and reverse kinematic model of the machine tool is established according to the kinematic chain of the machine tool. Based on the linear interpolation of rotary axis, the generation mechanism of nonlinear error is analyzed, the modeling methods of cutter center point, and cutter axis vector trajectory are proposed respectively, and the parameterized model of swing cutter trajectory is formed. The formula for the nonlinear error is obtained from the two-dimensional cutter center point trajectory. According to the established model of swing cutter trajectory, the synchronous optimization method of cutter center point trajectory and cutter axis vector trajectory is proposed, and the nonlinear error compensation mechanism is established. First, pre-interpolation is performed on the given cutter location data to obtain a model of the swing cutter trajectory for each interpolated segment. Then, the magnitude of the nonlinear error is calculated based on the parameters of the actual interpolation points during formal interpolation, and the nonlinear error is compensated for the interpolation points where the error exceeds \([\varepsilon ]\). In the VERICUT simulation, the maximum machining error was reduced from 50 to 5 μm by this paper method. In actual machining, the surface roughness of the free-form surface was reduced from 10.5 μm before compensation to 1.8 μm. The experimental results show that the proposed method can effectively reduce the impact of nonlinear errors on processing, and is of high practical value for improving the accuracy of cutter position and the quality of complex free-form machining in five-axis machining.