In this work we study the inverse problem related to the emission of Hawking radiation. We first show how the knowledge of greybody factors of different angular contributions $l$ can be used to constrain the width of the corresponding black hole perturbation potentials. Afterwards we provide a framework to recover the greybody factors from the actual energy emission spectrum, which has to be treated as sum over all multipole numbers. The underlying method for the reconstruction of the potential widths is based on the inversion of the Gamow formula, a parabolic expansion and the P\"oschl-Teller potential. We define a `normalized' energy emission spectrum that turns out to be very beneficial for the numerical fitting process, as well as for an improved qualitative understanding of how much information of the black hole potentials are actually imprinted in the spectrum. The connection to recent studies on the inverse problem using the quasi-normal spectra of ultra compact stars and exotic compact objects is discussed as well. In the appendix we show that the spectrum can be approximated surprisingly well and simply with a parabolic expansion of the peak of the classical black hole scattering potentials.