Thermosets-polymeric materials play a key role in the modern plastics and polymers composites, which are increasingly used in the areas subject to dynamic loading condition. The materials will experience an amount of micro-damage under such circumstances, while the damage softening behaviors cannot be well captured by viscoelastic or elasto-viscoplastic constitutive models. In this paper, a viscoelastic fractional damage model depended on strain rate is formulated to depict the dynamic responses of thermosetting polymers. A dynamic damage evolution law related to strain and strain rate is developed by means of the fractional calculus. The relationship between fractional order and damage evolution for materials is theoretically investigated. It is observed that the fractional order can be able to characterize the intensity of damage evolution, and the variation of order function is in line with the evolution with damage of material properties. The model applicability is verified by the compressive loading experimental data under different strain rates and temperatures of thermosets. The influences of temperatures on the proposed model and the damage evolution are analyzed by the fitting results. Furthermore, a corresponding thermo-viscoelastic model is developed by considering the temperature effect, of which the predictive ability is tested and the comparisons with the existing model are also presented. The results indicate the proposed model can better capture the stress-strain responses over a wider range of strain.