We, for the first time, propose a phase-field model to simulate the evolution of void ensembles under irradiation. The model takes into account one-dimensional migration of self-interstitial atoms (1-D SIA), vacancy diffusion, the generation and reaction between SIA and vacancies as well as the nucleation of voids. A one-dimensional random walker model (based on the theory of first-passage processes) is applied to describe the fast 1-D SIA while the Cahn–Hilliard equation is used to describe the slow three dimensional diffusion of vacancies. The coupling of these two methods greatly improves the computational efficiency for a system with strong inhomogeneity and anisotropy of diffusion. The formation of void lattices is simulated with the resultant model. It is found that a void lattice forms when the mobility of the 1-D SIA is four orders of magnitude larger than that of the vacancy mobility. A high generation rate of interstitials during displacement cascades delays the formation of a void lattice.