For its special shape of floating body with oblique side and external expansion, the heave and pitch motions of the sandglass-type Floating Production Storage and Offloading unit (FPSO) have coupling influence and instability phenomenon under harmonic wave excitation force. In order to study the nonlinear characteristic of the sandglass-type floating body, this paper presents the improved incremental harmonic balance (IHB) method in conjunction with the incremental arc-length method, and adopts the Floquet theory to determine the stability of the periodic solutions. Therein, the discrete Fourier transformation is introduced in the traditional IHB method to solve the problem of no analytical expressions for nonlinear FK and hydrostatic forces of the sandglass-type floating body. Considering the dependence of the heave and pitch hydrodynamic coefficients on wave frequency, the coefficients of the Fourier series in the IHB method are modified to cope with the memory effect. Taking a classic Spar platform as the test case, the accuracy of the method in this paper is validated and can be used to analyze the nonlinear coupling stability of the large-scale floating body. Then, the nonlinear responses corresponding to single-DOF heave and single-DOF pitch respectively are studied, and the nonlinear characteristics of hardening-type stiffness are verified by considering the different effects of damping and wave amplitude. Next, the interactive effects between pitch and heave motions of the sandglass-type model are further discussed, which are mainly reflected in two frequency bands representing the hardening-type nonlinearity of pitch and heave motions. Finally, the influence of different upper inclination angle of the floating body on heave and pitch coupling motions is analyzed. The results show that the smaller the upper angle, the larger the nonlinear motion responses. Consequently, the nonlinear characteristics of the heave and pitch motions should be fully noticed and considered in the shape design of the sandglass-type FPSO.