Serpentine belt accessory drive system (SBADS) is a typical coupled vibration system with rigid body vibrations and continuum vibrations. Vibration behaviors of a SBADS consist of rotational vibrations of pulleys and tensioner arm and transverse vibrations of belt spans. In modeling of a SBADS, transverse vibration of a belt span consisting of time and space coordinates is discretized into multiple degrees of freedom with only time variable. As a result, number of degrees of freedom of a SBADS is large, and computational efficiency of dynamic responses of a SBADS is low. To solve this issue, a calculation method for nonlinear dynamic characteristics of a SBADS is proposed based on the harmonic balance method with alternating frequency/time domain technique in the paper. An engine front end accessory drive system is taken as a study example, the coupling vibration model of the SBADS is established, and hysteretic behavior of the tensioner is considered in the model. In addition, a dimensionality reduction method for the dynamic matrix is proposed based on local nonlinearity of the model. A calculation method for nonlinear dynamic responses of the SBADS is subsequently developed by combining the harmonic balance method and alternating frequency/time domain technique. Nonlinear dynamic responses estimated by the proposed method are validated by comparing with numerical simulations. Finally, amplitude-frequency responses of the SBADS are calculated using arc-length continuation method, and influences of nonlinear tensioner on frequency domain responses and internal resonance phenomena of the SBADS are analyzed.