The heterogeneous mechanical properties of individual phases in SiCp/Al result in significant varying size effect (SE) existed in its deformation process. And modulating the varying SE is crucial for improving the machined surface finish of SiCp/Al in mechanical machining. In this study, we elucidate the mechanisms governing the varying SE in conventional cutting and ultrasonic elliptical vibration-assisted cutting (UEVC) of SiCp/Al using diamond tools by theoretical analysis, numerical simulations and experiments. Specifically, the impact of cutting tool-workpiece contact states in UEVC on the fundamental cutting behavior of SiCp/Al, in terms of microscopic deformation modes and their correlations with machining force variation, machined surface morphology and chip profile, is evaluated. Subsequently, ultrasonic elliptical vibration-assisted diamond turning of SiCp/Al is performed to achieve a surface roughness of 20 nm accompanied with significantly suppressed tool wear. Finally, the compensation effect on the varying SE in SiCp/Al cutting by ultrasonic elliptical vibration of cutting tool is discovered, which leads to decreased brittle fracture and improved surface integrity. The research findings provide a theoretical support for understanding the machining mechanisms of UEVC of SiCp/Al, as well as proposing new perspectives for improving the machinability of SiCp/Al composite accompanied by suppressed varying SE.