The power six-branch herringbone gear transmission system has the advantages of large power transmission and transmission ratio. Because of its multi-way transmission and over-constraint structure, to prevent loaded tooth interference, there are large backlashes between teeth. The system shows complex nonlinear dynamic characteristics under the influence of backlashes, which seriously affects meshing performance. The modification technology can effectively improve gear meshing performance. Hence, on the basis of optimizing meshing performance of the active pair of the six-branch herringbone gear transmission system, this article will combine 3D modification with system nonlinear vibration characteristics and propose an analysis method of the system nonlinear vibration characteristics under 3D modification to reduce vibration and noise. First, the 3D modified tooth surface equation is determined by forming grinding, the loaded transmission error (LTE) of the system's active pair is obtained by tooth contact analysis (TCA) and loaded tooth contact analysis (LTCA) technology, and the optimal modification parameters are obtained by Ant Lion Optimizer (ALO) with the minimum error amplitude as optimization objective. Then, the time-varying meshing stiffness of system under the optimal modification is calculated, and the pure torsional nonlinear dynamic model with backlashes, meshing stiffness, and static transmission error (STE) is established. Finally, the global vibration characteristics in parameter field are studied through time domain and bifurcation diagram of system. Results show that the 3D modification can eliminate edge contact of tooth and improve tooth contact performance. With the increase of input power, the root mean square (RMS) values of acceleration increase and the RMS values and jump decrease after 3D modification. With the increase of input speed, the RMS curves appear multiple resonance peaks and jumps at low input speed. After 3D modification, the RMS, jump, and resonance peak values decrease. Compared with level І, the backlashes and STEs of level II and phase difference ϛsII have great influence on system dynamic characteristics and easily make system in chaotic motion, while the 3D modification reduces their influence and makes system motion periodic.