It is well known that Dirac quantization of gauge theories is not, in general, equivalent to reduced quantization. When both approaches are self-consistent some additional criterion must be found in order to decide which approach is more natural, or correct. Now, in most cases quantization on the physical degrees of freedom is properly curved-space quantization, with a curvature that is neither constant nor Ricci flat. In a series of two papers, this being the first, we show that, unlike reduced quantization, Dirac quantization (acting in the physical Hilbert space) corresponds in a natural way to such a curved-space quantization scheme, and has remarkable similarities with other curved-space quantization schemes proposed elsewhere. We begin here with an in-depth analysis of the geometry of the classical configuration space of gauge theories. In particular, the existence of a metric on the full configuration space establishes a Yang-Mills connection with respect to the orbits---we emphasize the importance of this connection, as well as the condition the gauge theory must satisfy in order to define it. We also discuss, in Kaluza-Klein-like fashion, three Levi-Civita connections, their associated Ricci tensors, and interrelationships among them.