Fretting fatigue is a common type of problem in the engineering fields. Due to its multiaxial characteristics, it leads to a shorter overall fatigue life compared to plain fatigue. Fretting fatigue is caused by the micro slip at the contact surface, which is always accompanied by interface wear. This paper presents a refined elastic numerical simulation method combining interfacial wear with continuum damage mechanics (CDM) and fracture mechanics to predict fatigue life. The crack initiation life is determined by damage evolution law and the initiation position is determined by the total dissipated energy. Crack propagation life and path can be predicted using stress intensity factor. The stress redistribution and geometric change of contact surface caused by wear are considered in the whole fatigue process. Results indicate that the fretting fatigue simulation method incorporating wear has higher accuracy compared with the experimental results in the literature. It is noteworthy that wear retards nucleation damage process and crack propagation. As stress amplitude decreases, fatigue life increases, and delay effect of wear becomes more obvious. Additionally, wear causes the crack initiation position to move out, but crack initiation region is still near the slipping zone and propagation path does not change.