This study evaluates the rustling sounds of polyester warp knitted fabrics using a psychophysiological technique. We generate and record the rustling sounds of three structural types—reverse locknit, double denbigh, and sharkskin—of warp knitted fabrics, all made from 75D polyester filament yarn at the same density. To obtain quantitative information about rustling sounds, Zwicker’s psychoacoustic parameters—loudness( z), sharpness( z), roughness( z), and fluctuation strength( z)—are calculated from sound spectra obtained by FFT analysis. We acquire physiological signals—electroencephalogram, the ratio of low frequency to high frequency (LF/HF) from the power spectrum of heart rate variability, respiration rate (RESP), heart rate (HR)—from fourteen female participants when each sound is presented to them, and they rate seven sensations—softness, pleasantness, comfort, noisiness, annoyance, clearness, and highness—by free modulus magnitude estimation. Loudness( z), roughness( z), and fluctuation strength( z) are highest for the sharkskin and lowest for the reverse locknit. The rustling sound of double denbigh appears to be related to a positive sensation, with the highest softness, pleasantness, and comfort, and the lowest beta wave and LF/HF. On the other hand, the sharkskin sound appears to evoke negative sensations, with the highest annoyance and noisiness and most increasing beta wave, RESP, HR, and LF/HF. Therefore, we conclude that psychoacoustic parameters such as loudness( z), roughness( z), and fluctuation strength( z) affect psychological and physiological responses evoking negative sensations for warp knitted fabric sounds.