The nonlinear effects in gearboxes are a key concern to describe accurately their dynamic behavior. This task is difficult for complex gear systems such as planetary gearboxes. The main aim of this work is to provide responses to overcome this difficulty especially in nonstationary operating regimes by investigating a back-to-back planetary gearbox in steady conditions and in the run-up regime. The nonlinear Hertzian contact of teeth pair is modeled in stationary and nonstationary run-up regime. Then it is incorporated in to a torsional model of the planetary gearbox through different mesh stiffness functions. In addition, motor torque and external load variation are taken into account. The nonlinear equations of motion of the back-to-back planetary gearbox are computed through the Newmark- β algorithm combined with the method of Newton–Raphson. An experimental validation of the proposed numerical model is done through a test bench for both stationary and run-up regimes. The vibration characteristics are extracted and correlated to speed and torque. Time–frequency analysis is implemented to characterize the transient regime during the run-up.