This study is dedicated to investigate the nonlinear dynamics of a system composed of a flexible rotor shaft carrying a longitudinally dispositioned unbalanced rigid disc. The less-attended multi-modal nonlinear vibration analysis of rotor structures, considering the effect of geometric and inertial nonlinearities due to the large deflection of the flexible structure, is investigated in this study. To this end, the flexible shaft is modelled using the Euler–Bernoulli beam theory. In order to model the unbalanced rigid disc, it is considered as combination of a symmetric disc with an additional eccentric lump mass. The equations of motion of the system are derived using the extended Hamilton principle. Assuming the inextensional shaft and considering high twisting rigidity of the shaft, the flexural-flexural vibrations of the system of rotor shaft-disc are studied in this paper. The equations of motion are discretized using the Galerkin method exploiting various reduced order models including single-mode, two-mode, and three-mode approximations. In order to obtain the steady-state response of the system, two methods are used in this study. The first method is to use the semi-analytical modified complex averaging technique along with the numerical arc-length continuation method. The second method is using the direct integration in MATLAB. The nonlinear dynamics of different systems such as a system consisting only a flexible shaft and a system of flexible shaft carrying an unbalanced rigid disc at different positions along the shaft are investigated. The results show that geometric nonlinearity of the flexible shaft is dominating the inertial nonlinearity and hence, the system demonstrates a hardening nonlinear behaviour at the vicinity of the first two resonances. Besides, it is shown that dispositioning the unbalanced disc along the flexible shaft results in change in the nonlinear dynamics of the system at various modes. It is illustrated that if the unbalanced disc is located at the node of one of the modes of the vibration of the system, that mode is not excited, as expected. The results of the paper reveal that the longitudinal disposition plays a key role in the instability analysis of the system of rotor shaft-disc, as well as the radial unbalance. In the second part of the paper, nonlinear vibration of the system of a flexible rotor shaft with a longitudinally dispositioned unbalanced rigid disc is mitigated using nonlinear energy sink (NES). Two different conditions of the unbalanced disc are considered for the system. For each case, the NES is designed to control the undesired nonlinear vibration of the system close to the first two natural frequencies. The results show that the designed NESs are capable of controlling the system with different dynamic responses including periodic, quasi-periodic, chaotic, and also the system with unstable dynamic response that may lead to collapse of the system.