This paper proposes a method that combines the finite element method with an improved energy wear model to establish a three-dimensional (3D) ball-plane contact structures fretting wear model. The correctness of the model was verified by Hertz contact theory and fretting wear experiments. Then, the fretting wear dynamic behaviors of 3D ball-plane contact structures were investigated in detail. The results show that when the contact structure is in the partial slip regime (PSR), the cross-sectional wear scar profile changes from the “W” shape to the “U” shape along the fretting direction. When the contact structure is in the gross slip regime (GSR), the cross-sectional wear scar profile always maintains a “U” shape along the fretting direction. In addition, it was also found in the study that when the contact structure is at GSR under a low normal load or PSR under a high normal load, it will help reduce wear. This study can provide significant theoretical guidance for reducing and protecting fretting wear in contact structures.