Timbre has been identified as a potential component in the communication of affect in music. Although its function as a carrier of perceptually useful information about sound source mechanics has been established, less is understood about whether and how it functions as a carrier of information for communicating affect in music. To investigate these issues, listeners trained in Chinese and Western musical traditions were presented with Phrases, Measures, and Notes of recorded excerpts interpreted with a variety of affective intentions by performers on instruments from the two cultures. Results showed greater accuracy and more extreme responses in Chinese musician listeners and lowest accuracy in nonmusicians suggesting that musical training plays a role in listeners’ decoding of affective intention. Responses were more differentiated and more accurate with more musical information. Excerpts were also analyzed to determine acoustic features that are correlated with timbre characteristics. Temporal, spectral, and spectrotemporal attributes were consistently used in judging affective intent in music, suggesting purposeful use of these properties by listeners. Comparison between listeners’ use of acoustic features reveals a greater number of shared features between Western musicians and nonmusicians compared to Chinese musicians for valence, although the three groups shared more features for arousal. How timbre is utilized in musical communication appears to be different across musical traditions, and valence responses seem to be more culture-specific and arousal responses more similar across cultures.