Refractory tungsten-based medium- and high-entropy alloys form a promising class of strong, heat- and radiation-resistant materials for nuclear applications. Here, we systematically explore the radiation tolerance of Mo–Nb–Ta–V–W alloys using molecular dynamics simulations and a machine-learned interatomic potential. Going from pure W to W-based equiatomic binaries, ternaries, quaternary, and the quinary high-entropy alloy, we simulate the radiation damage accumulation up to 0.4–0.8 dpa through cumulative collision cascades. We find that the radiation damage and evolution are controlled by a combination of cascade-induced clustering, the balance in migration of vacancies and interstitials, and defect cluster binding energies. These mechanisms are strongly influenced by atom size, among which vanadium as the smallest atom is decisive. In pure W and alloys without V, the microstructure is characterised by large and growing interstitial dislocation loops. In stark contrast, in alloys containing V, vacancy dislocation loops are formed directly in collision cascades and the balanced migration rate of vacancies and interstitials promotes recombination and prevents growth of interstitial loops. The atom size differences also lead to clear segregation, with small atoms (V) in interstitial clusters and larger atoms (Ta, Nb) segregating around vacancy clusters. Furthermore, the small cluster sizes, formation of vacancy loops, segregation, and balanced migration in V-containing alloys lead to low swelling and an exceptionally efficient defect annealing compared to pure W and V-free alloys. Our results highlight WTaV as a promising low-activation material, where almost 90% of defects are annealed at 2000 K during 5 ns compared to <20% in pure W.