We study the realization of non-Abelian discrete gauge symmetries in 4d field theory and string theory compactifications. The underlying structure generalizes the Abelian case, and follows from the interplay between gaugings of non-Abelian isometries of the scalar manifold and field identifications making axion-like fields periodic. We present several classes of string constructions realizing non-Abelian discrete gauge symmetries. In particular, compactifications with torsion homology classes, where non-Abelianity arises microscopically from the Hanany-Witten effect, or compactifications with non-Abelian discrete isometry groups, like twisted tori. We finally focus on the more interesting case of magnetized branes in toroidal compactifications and quotients thereof (and their heterotic and intersecting duals), in which the non-Abelian discrete gauge symmetries imply powerful selection rules for Yukawa couplings of charged matter fields. In particular, in MSSM-like models they correspond to discrete flavour symmetries constraining the quark and lepton mass matrices, as we show in specific examples.