The detection of gravitational waves from the inspiral of a neutron star or stellar-mass black hole into an intermediate-mass black hole (IMBH) promises an entirely new look at strong-field gravitational physics. Gravitational waves from these intermediate-mass-ratio inspirals (IMRIs), systems with mass ratios from ~10:1 to ~100:1, may be detectable at rates of up to a few tens per year by Advanced LIGO/Virgo and will encode a signature of the central body's spacetime. Direct observation of the spacetime will allow us to use the "no-hair" theorem of general relativity to determine if the IMBH is a Kerr black hole (or some more exotic object, e.g. a boson star). Using modified post-Newtonian (pN) waveforms, we explore the prospects for constraining the central body's mass-quadrupole moment in the advanced-detector era. We use the Fisher information matrix to estimate the accuracy with which the parameters of the central body can be measured. We find that for favorable mass and spin combinations, the quadrupole moment of a non-Kerr central body can be measured to within a ~15% fractional error or better using 3.5 pN order waveforms; on the other hand, we find the accuracy decreases to ~100% fractional error using 2 pN waveforms, except for a narrow band of values of the best-fit non-Kerr quadrupole moment.