We study substructures of the Weyl group of conformal transformations of the metric of (pseudo)Riemannian manifolds. These substructures are identified by differential constraints on the conformal factors of the transformations which are chosen such that their composition is associative. Mathematically, apart from rare exceptions, they are partial associative groupoids, not groups, so they do not have an algebra of infinitesimal transformations, but this limitation can be partially circumvented using some of their properties cleverly. We classify and discuss the substructures with two-derivatives differential constraints, the most famous of which being known as the harmonic or restricted Weyl group in the physics literature, but we also show the existence of a lightcone constraint which realizes a proper subgroup of the Weyl group. We then show the physical implications that come from invariance under the two most important substructures, concentrating on classical properties of the energy-momentum tensor and a generalization of the quantum trace anomaly. We also elaborate further on the harmonic substructure, which can be interpreted as partial gauge fixing of full Weyl invariance using BRST methods. Finally, we discuss how to construct differential constraints of arbitrary higher-derivative order and present, as examples, generalizations involving scalar constraints with four and six derivatives.