This paper establishes a nonlinear dynamic model for a suspended riser under the condition of considering realistic configurations, based on the principles of virtual work and Hamilton's variational principle. The nonlinearity of the model is manifested by considering vortex-induced vibration in the cross flow (CF) and in-line (IL) directions, as well as large deformations, taking into account the variable diameter of the riser in the compliant riser system. The model is discretized using the finite element method and solved using the Newmark-β method. Experimental validation is conducted to verify the effectiveness of the model. Utilizing the parameters of a riser in a well in the South China Sea, the paper explores the influence of different environmental and structural parameters on the VIV response characteristics of a compliant riser. Results show that the vibration frequency is controlled by both the bare tube and buoyant block when the buoyant block coverage rate is 25%, which can significantly inhibit VIV. With the increase of flow velocity, the VIV energy jumps from low order to high order, and the riser mode also transitions from low order to high order. The increase in suspension length transfers the vibration energy from high to low frequency, thus realizing energy transition and mode shape transformation. With the increase in LMRP weight, the vibration and natural frequencies of the riser increase synchronously, the vibration amplitude decreases, and the mode shape remains unchanged.