The aim of this study is to investigate the impact of various factors on neutron spectrum unfolding, including track parameters (major axis, minor axis and depth), polyethylene (PE) thickness, and etching time. The Monte Carlo code Geant4 and proton track parameter calculation program TRACK_P were applied to simulate the response matrix and track parameters spectrum. Four test spectra—252Cf spectrum, Gaussian broadened deuterium–deuterium fusion neutron spectrum, 241Am–Be spectrum, and Gaussian broadened deuterium–tritium fusion neutron spectrum—were studied using the maximum-likelihood expectation maximization algorithm for neutron spectrum unfolding. The unfolded spectra were in good agreement with the reference spectra. Results showed that the major and minor axes were more suitable than the track depth for neutron spectrum unfolding. For high-energy neutrons, a thick PE layer is required. Moreover, etching time does not significantly affect the unfolded spectra. This study explores the factors influencing the unfolding of fast neutron spectrum based on simulation calculations and preliminary experiments that can be useful for applying CR-39 to neutron spectrum measurements.