In modular-invariant models of flavour, hierarchical fermion mass matrices may arise solely due to the proximity of the modulus τ to a point of residual symmetry. This mechanism does not require flavon fields, and modular weights are not analogous to Froggatt-Nielsen charges. Instead, we show that hierarchies depend on the decomposition of field representations under the residual symmetry group. We systematically go through the possible fermion field representation choices which may yield hierarchical structures in the vicinity of symmetric points, for the four smallest finite modular groups, isomorphic to S3, A4, S4, and A5, as well as for their double covers. We find a restricted set of pairs of representations for which the discussed mechanism may produce viable fermion (charged-lepton and quark) mass hierarchies. We present two lepton flavour models in which the charged-lepton mass hierarchies are naturally obtained, while lepton mixing is somewhat fine-tuned. After formulating the conditions for obtaining a viable lepton mixing matrix in the symmetric limit, we construct a model in which both the charged-lepton and neutrino sectors are free from fine-tuning.