The Milky Way centre exhibits an intense flux in the gamma and X-ray bands, whose origin is partly ascribed to the possible presence of a large population of millisecond pulsars (MSPs) and cataclysmic variables (CVs), respectively. However, the number of sources required to generate such an excess is much larger than what is expected from in situ star formation and evolution, opening a series of questions about the formation history of the Galactic nucleus. In this paper we make use of direct $N$-body simulations to investigate whether these sources could have been brought to the Galactic centre by a population of star clusters that underwent orbital decay and formed the Galactic nuclear star cluster (NSC). Our results suggest that the gamma ray emission is compatible with a population of MSPs that were mass segregated in their parent clusters, while the X-ray emission is consistent with a population of CVs born via dynamical interactions in dense star clusters. Combining observations with our modelling, we explore how the observed $\gamma$ ray flux can be related to different NSC formation scenarios. Finally, we show that the high-energy emission coming from the galactic central regions can be used to detect black holes heavier than $10^5\Ms$ in nearby dwarf galaxies.