Adaptive digital self-interference cancellation (ADSIC) is a significant method to suppress self-interference and improve the performance of the linear frequency modulated continuous wave (LFMCW) radar. Due to efficient implementation structure, the conventional method based on least mean square (LMS) is widely used, but its performance is not sufficient for LFMCW radar. To achieve a better self-interference cancellation (SIC) result and more optimal radar performance, we present an ADSIC method based on fractional order LMS (FOLMS), which utilizes the multi-path cancellation structure and adaptively updates the weight coefficients of the cancellation system. First, we derive the iterative expression of the weight coefficients by using the fractional order derivative and short-term memory principle. Then, to solve the problem that it is difficult to select the parameters of the proposed method due to the non-stationary characteristics of radar transmitted signals, we construct the performance evaluation model of LFMCW radar, and analyze the relationship between the mean square deviation and the parameters of FOLMS. Finally, the theoretical analysis and simulation results show that the proposed method has a better SIC performance than the conventional methods.