In some compounds, exchange repulsion is orientation dependent. However, in contrast to quantum chemical methods that treat exchange explicitly, empirical models assume exchange to be spherically symmetric, yielding an average description only. Here we quantify the anisotropy of exchange and dispersion energy for hydrogen halides and water by probing these compounds with a helium atom using the symmetry-adapted perturbation theory (SAPT). The exchange interaction is reduced by up to 33% due to the σ-hole in hydrogen iodide, depending on the location of the probe. We demonstrate how this anisotropy can be modeled in empirical force fields either using an angle-dependent potential or by introducing virtual sites, reducing the error in the empirical model by a factor of 5 compared to isotropic atoms. Lone-pairs on water, positioned close to perpendicular to the plane of the molecule, on a line with the oxygen atom, and, surprisingly, σ-holes on water both modulate the exchange interaction strongly. Both lone-pairs and σ-holes can be modeled by virtual sites, leading to an 80% reduced error.