Immersing a mobile impurity into a many-body quantum system represents a theoretically intriguing and experimentally effective way of probing its properties. In this work, we study the polaron spectral function in various environments, within the framework of Fermi-Hubbard models. Inspired by possible realizations in cold atoms and semiconductor heterostructures, we consider different configurations for the background Fermi gas, including charge density waves, multiple Fermi seas and pair superfluids. While our calculations are performed using an exact-diagonalization approach, hence limiting our analysis to systems of few interacting Fermi particles, we identify robust spectral features supported by theoretical results. Our work provides a benchmark for computations based on mean-field approaches and reveal surprising features of polaron spectra, inspiring new theoretical investigations.