The helical planetary transmission system (HPTS) is widely used in wind power, hydropower, and shield machine, which operating conditions are extremely complicated, and the existence of component installation errors will result in response deviations to the theoretical design. A dynamic model for slice type bending-torsional-pendulum of the HPTS is established in the lumped parameter method considering the installation error of shaft (IEOS) for the 1st planet gear, helical meshing characteristics, and bearing support characteristics. The vibration response characteristics of the HPTS under the IEOS for the 1st planet gear are studied. The results have shown that the FFT amplitudes for the horizontal vibration acceleration of the sun gear increased by 8.3%, and the axial vibration acceleration of the ring gear increased by 11.5%, respectively. The maximum magnitude increase of FFT amplitude for the vibration acceleration under a positive 0.13 mm IEOS for the 1st planet gear is the axial direction of the ring gear. The results indicate that the IEOS needs to be thoroughly introduced into the HPTS to simulate the actual operating conditions of the system.