Topological semimetals are a new class of metallic materials, which exist at band fillings that ordinarily correspond to insulators or compensated accidental semimetals with zero Luttinger volume. Their metallicity is a result of nontrivial topology in momentum space and crystal symmetry, wherein topological charges may be assigned to point band-touching nodes, preventing gap opening, unless protecting crystal symmetries are violated. These topological charges, however, are defined from noninteracting band eigenstates, which raises the possibility that the physics of topological semimetals may be modified qualitatively by electron-electron interactions. Here we ask the following question: what makes the topological semimetals nontrivial beyond band theory? Alternatively, can strong electron-electron interactions open a gap in topological semimetals without breaking the protecting symmetries or introducing topological order? We demonstrate that the answer is generally no, and what prevents it is their topological response, or quantum anomalies. While this is familiar in the case of magnetic Weyl semimetals, where the topological response takes the form of an anomalous Hall effect, analogous responses in other types of topological semimetals are more subtle and involve crystal symmetry as well as electromagnetic gauge fields. Physically these responses are detectable as fractional symmetry charges induced on certain gauge defects. We discuss the cases of type-I Dirac semimetals and time-reversal invariant Weyl semimetals in detail. For type-I Dirac semimetals, we also show that the anomaly vanishes, in a nontrivial manner, if the momenta of the Dirac nodes satisfy certain exceptional conditions.