Vanadium alloys are regarded as promising candidate structural materials for the advanced blanket concept in fusion reactors due to their low activation, good high-temperature strength and, in particular, their compatibility with liquid lithium. In the present work, six kinds of V–5Cr–5Ti alloys under heavy cold work with deformation amounts of 40%, 60% and 80%, and/or subsequent annealing were investigated. Irradiation damage of 0.1, 0.3 and 0.5 dpa was introduced in both specimens using 352.8 MeV Fe ions at 100 °C. Electron backscattered diffraction and transmission electron microscopy (TEM) were used to investigate pre-irradiation microstructures such as grains, dislocations, precipitates and bubbles. X-ray diffraction was used to evaluate the pre-existing dislocation density and TEM was used to image the irradiation defects. The change in hardness was evaluated using micro-hardness tests. Before irradiation, the hardness increased with the increasing deformation amount but decreased after subsequent annealing. Dislocation cells turning into sub-grains with low-angle boundaries were observed, while the deformation amount reached 80% in cold-worked specimens. After irradiation, hardening was observed in all specimens and at all irradiation doses, and a power-law relation was observed in dose-dependent hardening. The effect of the initial microstructure on irradiation hardening was discussed in terms of the sink strength while ignoring grains and precipitates due to their large size. Pre-existing bubbles could effectively reduce irradiation hardening compared with previous results. Meanwhile, with the increasing sink strength of dislocations, hardening decreased in a different manner in cold-worked and annealed specimens. The irradiation defects in some specimens were investigated to clarify the inherent mechanism in the relationship between the initial microstructures and irradiation hardening.