Music and language are two intricately linked communication modalities in humans. A deficit in music pitch processing as manifested in the condition of congenital amusia has been related to difficulties in lexical tone processing for both tone and non-tonal languages. However, it is still unclear whether amusia also affects the perception of vowel phonemes in quiet and in noise. In this study, we examined vowel-plus-tone identification in quiet and noise conditions among Mandarin-speaking amusics with and without speech tone difficulties (tone agnosics and pure amusics, respectively), and IQ- and age-matched controls. Overall, pure amusics showed vowel and tone identification comparable to the controls in both quiet and noise conditions. Compared to pure amusics and controls, tone agnosics showed deficits in tone perception in both quiet and noise conditions. More importantly, their vowel perception was lower than pure amusics and controls in noise conditions, e.g., at a signal-to-noise ratio of −4 dB, although they showed normal-like performance in quiet and at a signal-to-noise ratio of −8 dB. These results suggest that when amusia affected speech tone processing (e.g., tone agnosics), it could also compromise vowel processing in noise. However, amusia alone does not affect tone or vowel perception in Mandarin Chinese either in quiet or in noise. Overall, the current study highlights the necessity of taking heterogeneity within the amusic group into account when considering the related speech deficits in this group.