The two most favored explanations of the Fermi Galactic Center gamma-ray excess (GCE) are millisecond pulsars and self annihilation of the smooth dark matter halo of the galaxy. In order to distinguish between these possibilities, we would like to optimally use all information in the available data, including photon direction and energy information.To date, analyses of the GCE have generally treated directional and energy information separately, or have ignored one or the other completely.Here, we develop a method for analyzing the GCE that relies on simulation-basedinference with neural posterior models to jointly analyze photon directional and spectral information while correctly accounting for the spatial and energy resolution of the telescope, here assumed to be the Fermi Large Area Telescope (LAT).Our results also have implications for analyses of the diffuse gamma-ray background, which we discuss.