The orbital Hall effect (OHE) designates the generation of a charge-neutral flow of orbital angular momentum transverse to an initial charge current. Recent theoretical investigations suggest that transition metals display sizable OHE, encouraging experimental search along this direction. Nonetheless, most of these theories assume that the orbital moment originates from the region immediately surrounding the atom core, adopting the so-called {\it atomic center approximation}. In periodic crystals though, the contribution of the interstitial regions is crucial and can lead to a severe misestimation of the OHE. By applying the "modern theory" of orbital magnetization to the OHE, we assess the relative importance of intra-atomic and inter-atomic contributions in selected materials from first principles. We find that whereas the OHE is mostly of intra-atomic origin for wide band-gap semiconductors (e.g., MoS$_2$), the inter-atomic contribution becomes crucial in narrow band-gap semiconductors (SnTe, PbTe) and transition metals (Pt, V etc.). These predictions invalidate the atomic center approximation adopted in some of the previous works and open perspectives for the realization of efficient sources of orbital currents.