Abstract A population of non-stellar black holes (≳100 M⊙) has been long predicted to wander the Milky Way. We aim to characterize this population by using the L-Galaxies semi-analytical model applied on top of the high resolution Millennium-II merger trees. Our results predict ∼10 wandering black holes with masses ∼2 × 103 M⊙ in a typical z = 0 Milky Way galaxy, accounting for $\sim 2\%$ of the total non-stellar black hole mass budget of the galaxy. We find that the locations of these wanderers correlate with their formation scenario. While the ones concentrated at ≲1 kpc from the galactic nucleus on the disc come from past galactic mergers, the ones formed as a consequence of ejections due to gravitational recoils or the disruption of satellite galaxies are typically located at ≳100 kpc. Such small and large distances might explain the absence of strong observational evidence for wandering black holes in the Milky Way. Our results also indicate that $\sim 67\%$ of the wandering population is conformed by the leftovers of black hole seeds that had little to no growth since their formation. We find that wandering black holes that are leftover seeds become wanderers at an earlier time with respect to grown seeds, and also come from more metal-poor galaxies. Finally, we show that the number of wandering black holes in a Milky Way-type galaxy depends on the seeding efficiency.