We compare the decay of plasmons and "conventional" hot electrons within the same series of gold/metal oxide interfaces. We found an accelerated decay of hot electrons at gold-metal oxide interfaces with decreasing band gap of the oxide material. The decay is accelerated by the increased phase space for electron scattering caused by additional interfacial states. Since plasmons decay faster within the same series of gold-metal oxide interfaces, we propose plasmons are able to decay into the same interfacial states as hot electrons. The similarity of plasmon damping to conventional hot electron decay implies that many classical surface analysis techniques and theoretical concepts are transferable to plasmonic systems. Our results support the mechanism of direct decay of plasmons into interfacial hot electron pairs but the utility of these interfacial states for charge transfer reactions remains to be investigated.