The annoyance attributable to complex noises consisting of multiple pure tones (which may or may not be harmonically related) and substantial broadband noise is not easily quantifiable. It has been shown that dBA is a poor indicator of loudness [Hellman and Zwicker, J. Acoust. Soc. Am. 82, 1700–1705 (1987)], and a number of other measures have been proposed. In this study, the results of subjective preference tests of passenger compartment noise are compared to objective measures of loudness and annoyance. Eight synthesized variations of the noise experienced at full throttle acceleration were ranked by 40 subjects according to their desirability. All samples were normalized to 101‐dB linear. These results are compared to the objective rating methods: AI (articulation index), ISO 532B (Zwicker method), Stevens Mark VII, CRP (Composite Rating of Preference), and PNL (perceived noise level). The CRP offers the best correlation with subjective results. The results of Mark VII, and ISO 532B are markedly improved by the addition of empirically derived corrections.