Exploring the relation between fundamental frequency and spectral envelope in the perception of musical instrument sounds

Abstract

Pitch and timbre perception centrally depend on fundamental frequency (F0) and spectral envelopes (SE), respectively. However, timbre perception as a function of F0 is less explored in the literature. An analysis-synthesis approach was used to compare sounds with congruent and incongruent F0 and SE parameters via sound pleasantness, brightness, and plausibility ratings. Analyzing 1900 musical instruments and voice sounds, a two-dimensional synthesis space was derived. The space was sampled in its inner and outer regions, reflecting high and low similarity to existing acoustical instrument sounds. Sounds with congruent and incongruent F0-SE relations were synthesized, defining congruency as the co-occurrence of F0 and SE in the analyzed set of sounds. In Experiment 1, inner regions of the space were rated as more pleasant compared to outer regions. Further, sounds with incongruent F0-SE relation were rated as less pleasant only for F0 discrepancies above two octaves. Finally, the first dimension of the space strongly reflected differences in brightness when F0 was controlled for. In Experiment 2, a spectrally more fine-grained resynthesis approach revealed effects of congruency on pleasantness and plausibility for smaller F0 separations. These results provide evidence for a co-dependence of F0 and SE properties in the perception of harmonic sounds.