From a theoretical perspective, perfect Green's function recovery in diffusive systems is based on cross-correlation of time-series measured at distinct locations arising from background fluctuations from an infinite set of uncorrelated sources, either naturally occurring or engineered. Clearly such a situation is impossible in practice, and a relevant question to ask, then, is how does an imperfect set of noise sources affect the quality of the resulting empirical Green's function (EGF)? We narrow down this broad question by exploring the effect of source location and make no distinction between whether the noise sources are natural or man made. Following the theory of EGF recovery, the only requirement is that the sources are uncorrelated and endowed with the same (or nearly so) frequency spectrum and amplitude. As such, our intuition suggests that noise sources proximal to the observation points are likely to contribute more to the Green's function estimate than distal ones. However, in what manner and over what spatial extent our intuition is less clear. Thus, in this short note we specifically ask the question, ‘Where are the noise sources that contribute most to the Green's function estimate in heterogeneous, lossy systems?’ We call such a region the volume of relevance (VoR). Our analysis builds upon recent work on 1-D homogeneous systems by examining the effect of heterogeneity, dimensionality and receiver location in both one and two dimensions. Following the strategy of previous work in the field, the analysis is conducted out of mathematical convenience in the frequency domain although we stress that the sources need not be monochromatic. We find that for receivers located symmetrically across an interface between regions of contrasting diffusivity, the VoR rapidly shifts from one side of the interface to the other, and back again, as receiver separation increases. For the case where the receiver pair is located on the interface itself, the shifting is less rapid, and for moderate-to-high diffusivity contrasts, the VoR remains entirely on the more diffusive side over receiver separations up to two to three skin depths based on the high-diffusivity value. Finally, because diffusivity plays a role analogous to resistivity in electromagnetic induction problems, our results suggest that the VoR for the latter is dominated by the air region when the receivers are located on the Earth's surface—a finding that demonstrates the minimal impact of subsurface noise sources for EGF estimation from surface-based electromagnetic geophysical experiments.