The dynamic characteristics of a rotor system with an initial bow and coupling fault of imbalance-rub are investigated in this work. The geometrical nonlinearity of shafts becomes significant due to the large-amplitude whirling motion. Then, the influences of the initial bow and geometrical nonlinearity on the natural frequencies corresponding to the linear part of the rotor system are studied. Moreover, the coupling faults of imbalance-rub are introduced to the rotor system. Complicated nonlinear phenomena are revealed by bifurcation diagrams, time histories, Poincare sections, and spectra. The influences of several key design parameters, such as initial bow, shaft radius, and casing stiffness, are analyzed. One of the main findings of this investigation is that when initial bow and geometrical nonlinearity coexist in a system, the resonance characteristics are obviously affected by the initial bow’s degree. This coexistence can result in the jump phenomenon, which rapidly increases the amplitude of whirling motion. These findings are useful in the fault diagnosis and feature recognition of rotating machines.