Focusing on the inverse synthetic aperture radar (ISAR) imaging of targets with complex motion, this paper proposes a modified version of the Fourier transform, called non-uniform rotation transform, to achieve cross-range compression. After translational motion compensation, the target’s complex motion is converted into non-uniform rotation. We define two variables—relative angular acceleration (RAA) and relative angular jerk (RAJ)—to describe the rotational nonuniformity. With the estimated RAA and RAJ, rotational nonuniformity compensation is carried out in the non-uniform rotation transform matrix, after which, a focused ISAR image can be obtained. Moreover, considering the possible deviation of RAA and RAJ, we design an optimization scheme to obtain the optimal RAA and RAJ according to the optimal quality of the ISAR image. Consequently, the ISAR imaging of targets with complex motion is converted into a parameter optimization problem in which a stable and clear ISAR image is guaranteed. Compared to precedent imaging methods, the new method achieves better imaging results with a reasonable computational cost. Experimental results verify the effectiveness and advantages of the proposed algorithm.