In two dimensions, magnetic higher-order topological insulators (HOTIs) are characterized by excess boundary charge and a compensating bulk ``filling anomaly.'' At the same time, without additional noncrystalline symmetries, the boundaries of two-dimensional HOTIs are gapped and featureless at low energies, while the bulk of the system is predicted to have a topological response to the insertion of lattice (particularly disclination) defects. Until recently, a precise connection between these effects has remained elusive. In this work, we point the direction towards a unifying field-theoretic description for the bulk and boundary response of magnetic HOTIs. By focusing on the low-energy description of the gapped boundary of a two-dimensional magnetic HOTI with no time-reversing symmetries, we show that the boundary charge and filling anomaly arise from the gravitational ``Gromov-Jensen-Abanov'' (GJA) response action [A. Gromov et al., Phys. Rev. Lett. 116, 126802 (2016)] in the context of the quantum Hall effect. As in quantum Hall systems, the GJA action cancels apparent anomalies associated with the bulk response to disclinations, allowing us to derive a concrete connection between the bulk and boundary theories of HOTIs. We show how our results elucidate the connection between higher-order topology and geometric response in band insulators, and point towards a route to understanding interacting higher-order topological phases beyond the simple cases considered here.