In recent years, modeling studies on the interactions between defect clusters have been extensively conducted to predict the behavior of stainless steels in reactors. However, comparable experimental results are desired to validate existing results and provide guidelines for future modeling. The size of defect clusters is expected to be a key factor influencing cluster interactions. Thus, in this work, the effects of growing clusters on their surrounding microstructure were quantitatively examined by in situ transmission electron microscopy during ion irradiation. To this end, a high-purity 316 L stainless steel model alloy was irradiated by 2 MeV Fe2+ to 0.2 dpa at 400 °C and 300 °C separately. At 300 °C, the number density of defect clusters monotonically increased with increasing fluence, whereas at 400 °C, the number density decreased almost immediately after the rapid nucleation regime. This decrease could be explained by the distinct growth of some clusters, which suppressed the nucleation of clusters around them as well as the lifetime of neighboring tiny clusters. Large interstitial-type clusters approximately 6–11 nm in size could be absorbed by neighboring interstitial-type clusters of similar sizes via interstitial crowdion diffusion. The absorption would not occur until both clusters grew large enough to permit a <110> diffusion path between them.