Spin-to-charge conversion at metal/oxide interfaces with stacked structures of permalloy/(Cu or Ag)/${\mathrm{Bi}}_{2}{\mathrm{O}}_{3}$ were systematically investigated by using the spin-pumping technique and cross-sectional transmission electron microscopy. Although the transport measurement reproduced the results in previous studies, the interfacial structure of (Cu or Ag)/${\mathrm{Bi}}_{2}{\mathrm{O}}_{3}$ was found to change depending on the crystalline orientation of the Cu or Ag underlayer. While the Ag/${\mathrm{Bi}}_{2}{\mathrm{O}}_{3}$ stacks with Ag(111) had steep interfaces, the formation of a nanometer-scale Cu-O layer can be found in the Cu/${\mathrm{Bi}}_{2}{\mathrm{O}}_{3}$ interface, which should be the main origin of the sign inversion of the conversion coefficients between Cu/${\mathrm{Bi}}_{2}{\mathrm{O}}_{3}$ and Ag/${\mathrm{Bi}}_{2}{\mathrm{O}}_{3}$. This study stresses the importance of nanostructure identification for discussing spin-to-charge conversions at metal/oxide interfaces.