Nonlinear friction and interaction between bit and rock often result in a lower weight on bit (WOB) than the intended design value. The analysis of the dynamic transmission mechanism of a drillstring in complex horizontal wells is a crucial engineering problem that aims to enhance drilling efficiency. In this paper, a mechanical model was developed, integrating nonlinear bit penetration behavior and variable axial loads, aiming to unveil the nonlinear axial vibration characteristics and elucidate the transmission mechanisms within the drillstring. Notably, the mean value of axial force distributed along the drillstring is dictated by wellhead pulling force, while the amplitude is shaped by the interaction between bit and rock, and rotary speed. In calculated examples for horizontal wells, the analysis reveals that significant losses in power axial transmission, resulting in WOB is only 38% of the intended value. And reducing the wellhead pulling force by 30 kN leads to a maximum increase in WOB only of 9.51 kN. Extended region in drillstring with negative axial forces exhibit maximum values 5–6 times higher than the WOB. Specific zones, such as 20 m–40 m and 0–20 m from the bit, experience axial force fluctuation amplitudes of 40.5% and 138.8% of the mean value respectively, with the latter identified as an intense vibration zone, while the acceleration values 0–10 m from the bit signal a high-risk zone. In prolonged horizontal section, stability issues manifest, with spiral buckling observed in the holding section and sinusoidal buckling in the horizontal section. Mitigating the synchronous vibrations between the bit and the surrounding drillstring can effectively reduce the intensity of axial vibration. To ensure successful drilling to target formation, adjusting the wellhead pulling force to keep the WOB in the range of 60–90 kN and limiting the rotary speed to 90 r/min are recommended practices. This comprehensive analysis offers valuable insights for optimizing drilling parameters, pinpointing high-vibration zones, and mitigating risks associated with drilling, providing a holistic approach to enhancing drilling efficiency.