We study the origin of the positron excess observed in the local cosmic-ray spectrum at high energies, and relate it to the cosmic rays and gamma-ray emission across the entire Galaxy. In particular, we explore the hypothesis of a single, dominant source accountable for primary electron-positron pairs. Since we are agnostic about the physical nature of the underlying source population, we consider four models that are representative of young pulsars, old stars (as a tracer of millisecond pulsars), and annihilating dark matter particles. In the dark matter hypothesis, we consider both a cored and a cuspy model for the halo in the Milky Way. Then, we compare the associated gamma-ray sky maps with Fermi-LAT data. We find that the emission arising from pulsar wind nebulae is fairly concentrated near the mid plane, and therefore additional cosmic-ray sources must be invoked to explain the emission at the center of the Galaxy. If the local positron excess were mainly due to millisecond pulsars, inverse Compton scattering by the particles injected would naturally account for a non-negligible fraction of the central gamma-ray emission. The same process would lead to a tension for a standard NFW dark matter profile, exceeding the gamma-ray data by almost a factor of 2 in some regions of the Galaxy. Although the results for an isothermal, cored profile are in agreement with these data, the cross section needed in both cases is around 2 orders of magnitude above the thermal cross-section, disfavouring the dark matter interpretation.