In this work we prove a bound for the torsion in Mordell-Weil groups of smooth elliptically fibered Calabi-Yau 3- and 4-folds. In particular, we show that the set of torsion groups which can occur on a smooth elliptic Calabi-Yau n-fold is contained in the set of subgroups which appear on a rational elliptic surface if n ≥ 3 and is slightly larger for n = 2. The key idea in our proof is showing that any elliptic fibration with sufficiently large torsion1 has singularities in codimension 2 which do not admit a crepant resolution. We prove this by explicitly constructing and studying maps to a modular curve whose existence is predicted by a universal property. We use the geometry of modular curves to explain the minimal singularities that appear on an elliptic fibration with prescribed torsion, and to determine the degree of the fundamental line bundle (hence the Kodaira dimension) of the universal elliptic surface which we show to be consistent with explicit Weierstrass models. The constraints from the modular curves are used to bound the fundamental group of any gauge group G in a supergravity theory obtained from F-theory. We comment on the isolated 8-dimensional theories, obtained from extremal K3’s, that are able to circumvent lower dimensional bounds. These theories neither have a heterotic dual, nor can they be compactified to lower dimensional minimal SUGRA theories. We also comment on the maximal, discrete gauged symmetries obtained from certain Calabi-Yau threefold quotients.