Stress relaxation manifests itself as the most fundamental mechanical response of bulk amorphous polymers. We characterized polymer repeating units with four-set interaction parameters separately representing the thermodynamic and kinetic aspects of local intrachain and interchain interactions, and evaluated their relative importance in stress relaxation with linear viscoelasticity at high temperatures by means of kinetic Monte Carlo simulations (DOI: 10.1016/j.polym.2021.123740). Hereby we continued to evaluate their relative importance in the initiation and the deviation extent of nonlinear viscoelasticity at low temperatures. We observed that thermodynamic interchain interactions seem no effective, while the other three local interactions retard the later stage of stress relaxation for nonlinear viscoelasticity, responsible for the KWW indexes of polymer rubbery states around 0.5. Both the kinetic aspects of intrachain and interchain interactions dominate the initiation temperatures of nonlinear viscoelasticity, and their high barriers suppress the deviation extent of nonlinear viscoelasticity towards the strong liquids like metals and ceramics, while their low barriers leave the fragile liquids to more thermodynamically rigid polymers. Our work facilitates a better understanding of structure-property relationship in nonlinear viscoelasticity of bulk amorphous polymers.