For a dynamic system of a rigid hub and a flexible cantilever beam, the traditional hybrid coordinate model assumes the small deformation in structural dynamics where axial and transverse displacements at any point in the beam are uncoupled. This traditional hybrid coordinate model is referred as the zeroth-order approximation coupling model in this paper, which may result in divergence to the dynamic problem of some rigid–flexible coupling systems with high rotational speed. In this paper, characteristics of a flexible hub-beam system with a tip mass is studied. Based on the Hamilton theory and the finite element discretization method, and in consideration of the second-order coupling quantity of the axial displacement caused by the transverse displacement of the beam, the rigid–flexible coupling dynamic model (referred as the first-order approximation coupling (FOAC) model in this paper) and the corresponding model in non-inertial system for the flexible hub-beam system with a tip mass are presented firstly, then the dynamic characteristics of the system are studied through numerical simulations under twos cases: the large motion of the system is known and is unknown. Simulation and comparison studies using both the traditional zeroth-order model and the proposed first-order model show that even small tip mass may affect dynamic characteristics of the system significantly, which may result in the largening of vibrating amplitude and the descending of vibrating frequency of the beam, and may affect end position of the hub-beam system as well. The effect of the tip mass becomes large along with the increasing of rotating speed of large motion of the system. When the large motion of the system is at low speed, the traditional ZOAC model may lead to a large error, whereas the proposed FOAC model is valid. When the large motion is at high speed, the ZOAC model may result in divergence to the dynamic problem of the flexible hub-beam system, while the proposed second model can still accurately describe the dynamic hub-beam system.