We study novel three-dimensional gapped quantum phases of matter which support quasiparticles with restricted mobility, including immobile "fracton" excitations. So far, most existing fracton models may be instructively viewed as generalized Abelian lattice gauge theories. Here, by analogy with Dijkgraaf-Witten topological gauge theories, we discover a natural generalization of fracton models, obtained by twisting the gauge symmetries. Introducing generalized gauge transformation operators carrying an extra phase factor depending on local configurations, we construct a plethora of exactly solvable three-dimensional models, which we dub "twisted fracton models." A key result of our approach is to demonstrate the existence of rich non-Abelian fracton phases of distinct varieties in a three-dimensional system with finite-range interactions. For an accurate characterization of these novel phases, the notion of being inextricably non-Abelian is introduced for fractons and quasiparticles with one-dimensional mobility, referring to their new behavior of displaying braiding statistics that is, and remains, non-Abelian regardless of which quasiparticles with higher mobility are added to or removed from them. We also analyze these models by embedding them on a three-torus and computing their ground state degeneracies, which exhibit a surprising and novel dependence on the system size in the non-Abelian fracton phases. Moreover, as an important advance in the study of fracton order, we develop a general mathematical framework which systematically captures the fusion and braiding properties of fractons and other quasiparticles with restricted mobility.