Though data concerning gold’s optical properties differ substantially in the literature, the causes for these discrepancies are poorly understood. Surface quality affects the optical response considerably, not only through crystal defects but also through the existing morphology. If optical data analysis is done under the assumption of ideally flat surfaces, the obtained dielectric function or dynamic conductivity and any model parameters represent an effective description that may differ from bulk values according to various preparation conditions. To show this finding in detail, we performed spectroscopic ellipsometry measurements of evaporated gold films in the mid-infrared range, below the onset of the interband transitions, and investigated the sample morphology by means of atomic force microscopy. This study yields effective Drude-model parameters that vary with film morphology over a range that includes most of the published values. Introducing a Bruggeman effective medium to model rough films as a mixture of bulk metal and empty volume makes it possible to find a relation between metal volume fraction and effective plasma frequency. In such a model, the plasma frequency and also the dielectric background resulting from interband transitions decrease as the fraction of empty volume inclusions increases. In contrast, while metal volume fraction is much less influential to relaxation time, the density of the gold crystallites’ grain boundaries yields a strong effect. We thus found a plasma frequency, relaxation rate, and dielectric background for the most ideal gold films at room temperature of 7.37(40) × 104 cm–1, 221(1) cm–1, and 9.6(3), respectively.