Perceptual effects of simplifying musical instrument sound time-frequency representations

Abstract

The perceptual salience of several outstanding features of musical instrument quasi-harmonic time-variant spectra were investigated. These are: amplitude spectrum shape variation, harmonic amplitude (or frequency) micro-variations, spectral envelope irregularity, harmonic frequency incoherence, and frequency inharmonicity. Seven musical instrument sounds (clarinet, flute, oboe, trumpet, violin, harpsichord, and marimba) were spectrum-analyzed to produce time-varying harmonic amplitude and frequency data. Tones resynthesized from these data were equalized in pitch, loudness, and duration. Six basic data simplifications were applied together with five combinations of them: amplitude versus time smoothing, spectrum shape fixing, spectral envelope smoothing, harmonic frequency coherence, frequency versus time smoothing, and harmonic frequency fixing. Twenty subjects participated in a discrimination study to determine the ease of distinguishing sounds synthesized from simplified data from sounds resynthesized from the full data. Averaged over the seven instruments, the mean discrimination scores were: spectral envelope smoothing, 96%; spectrum shape fixing, 91%; harmonic frequency fixing, 71%; frequency smoothing, 70%; frequency coherence, 69%; and amplitude smoothing, 66%. In conclusion, spectral envelope smoothing had a profound effect (discrimination >95%) for all instruments except the trumpet. Spectrum shape fixing, which eliminates any spectral centroid variation, had a very large effect except for the clarinet and the oboe.