We report on the magnetic and the electronic properties of the prototype dilute magnetic semiconductor Ga$_{1-x}$Mn$_x$As using infrared (IR) spectroscopy. Trends in the ferromagnetic transition temperature $T_C$ with respect to the IR spectral weight are examined using a sum-rule analysis of IR conductivity spectra. We find non-monotonic behavior of trends in $T_C$ with the spectral weight to effective Mn ratio, which suggest a strong double-exchange component to the FM mechanism, and highlights the important role of impurity states and localization at the Fermi level. Spectroscopic features of the IR conductivity are tracked as they evolve with temperature, doping, annealing, As-antisite compensation, and are found only to be consistent with an Mn-induced IB scenario. Furthermore, our detailed exploration of these spectral features demonstrates that seemingly conflicting trends reported in the literature regarding a broad mid-IR resonance with respect to carrier density in Ga$_{1-x}$Mn$_x$As are in fact not contradictory. Our study thus provides a consistent experimental picture of the magnetic and electronic properties of Ga$_{1-x}$Mn$_x$As.