Roller burnishing can be applied for surface quality improvement without chip generation in machining processes. A rotary roller burnishing assisted with ultrasonic vibration machining technology has been developed. The present work aims to investigate mechanism of deformation for the machined surface layer material under different vibration frequencies. Firstly, an analysis model and a 2D finite element simulation model of rotary roller burnishing process with and without ultrasonic vibration were established. Secondly, the area of material happened plastic flow and the displacements of material at the free surface were analyzed. Thirdly, von Mises stresses of being deformed material under different frequencies were presented. The residual stress on the free surface is discussed and the simulation result has a good agreement with experimental result. The micro-hardness at the machined surface is measured. The result shows that ultrasonic vibration can decrease the flow stress of material deformation. Finally, the effect of ultrasonic softening and ultrasonic enhancement with the variable vibration frequencies are discussed. The material deformation experiences a process from ultrasonic softening to ultrasonic enhancement in rotary ultrasonic roller burnishing compared to the material deformation in conventional rotary roller burnishing.