This study performs a systematic analysis of the dynamic behaviors of a gear pair system mounted on rotor–bearing system with strongly nonlinear effects including nonlinear suspension effect, nonlinear couple-stress fluid film force, nonlinear rub-impact force and nonlinear gear mesh force. The dynamic orbits of the system are observed using bifurcation diagrams plotted using the dimensionless rotational speed ratio as a control parameter. The onset of chaotic motion is specified from the phase diagrams, Poincare maps, Lyapunov exponents and fractal dimension of the system. There exist various forms of periodic, sub-harmonic and chaotic motions at different bifurcation parameters. The simulation results revealed that highly non-periodic motions do exist in gear-rotor–bearing systems under those nonlinear effects. The results also confirm that the couple-stress fluid can suppress non-periodic vibrations, especially at lower rotational speeds. The results of this study allow suitable system parameters to be defined such that undesirable behaviors of the system can be avoided and the machinery life extended as a result.