This study presents modeling results on fatigue wear of elastomers. A contact problem solution has been derived for the sliding of a system of asperities over a viscoelastic half-space. The mechanical properties of the viscoelastic half-space are described by relations between stresses and strains given by the Volterra integral operator. The contact problem is solved by the boundary element method using an iterative procedure. Stresses in the subsurface layers of the viscoelastic material are analyzed. The damage function of the surface layer is calculated using a reduced stress criterion, the parameters of which are determined on the basis of available experimental data. The wear process is studied under the assumption that the accumulated damage can be summed up. Within the applied frictional interaction model, the wear process presents the delamination of material surface layers of finite thickness at discrete points in time and continuous surface wear by fatigue mechanism. A model calculation of contact fatigue damage accumulation has shown that the time to the first material delamination (incubation period) depends on the sliding velocity and the viscoelastic properties of the material. By analyzing the dependence of the wear rate on the input parameters of the problem, it was investigated how the sliding velocity affects the time of fatigue damage initiation and the run-in and steady-state wear rates in materials with different rheological properties. Model calculations revealed that the wear rate of material surface layers after the incubation period increases smoothly and then stabilizes. The presence of the steady-state wear rate agrees well with experimental data. The developed method for studying fatigue damage accumulation in the surface layers of viscoelastic materials in frictional interaction can also be applied on the macrolevel to determine possible crack initiation sites.