The knowledge of the defect and impurity generation rates, as well as the defect spatial distribution, is the corner stone for the understanding of the evolution of material properties under irradiation. This knowledge is also an essential element for comprehensive experimental simulations of the behavior of irradiated materials. In this article the interaction of neutron and proton irradiation with metals is discussed with respect to displacement damage production. Charged particle irradiation is also briefly illustrated. After discussion of the primary interaction of projectiles (neutrons, charged particles in general, and protons in particular) with target atoms/nuclei, we describe the interaction of a recoil atom with other target atoms resulting in the slowing down of the projectile, displacement damage, impurity atom production due to nuclear reactions, and the creation of atomic displacement cascades. Then the further evolution of defect structure is discussed. The next section, devoted to subcascade formation, is divided into two parts. The first experimental evidence of subcascade formation under neutron and charged particle irradiation is presented. Then the models of subcascade formation are described. Finally we review the models for the calculation of displacement damage and show how these models can be applied to displacement damage calculation under neutron irradiation with a demonstration of a real application of the methods discussed to several nuclear facilities. To cite this article: P. Vladimirov, S. Bouffard, C. R. Physique 9 (2008).