To investigate the nonlinear seismic response of the reef-coral sand site in the South China Sea, a visco-elasto-plastic dynamic constitutive model and corresponding subroutine for coral sand material are developed in this study. The Matasovic skeleton curve and the derived hysteretic curve are refined in this model by introducing a critical shear modulus. Hysteresis criteria under irregular cyclic loads are established based on the improved Masing criteria. Moreover, increments of the octahedral strain and stress are utilized to construct the mapping rules between the three-dimensional stress-strain relationship and the one-dimensional stress-strain space. On that basis, a nonlinear finite element model of the reef-coral sand site is established, and the distribution pattern of the ground motion field and key ground motion parameters, such as the seismic amplification ratio and response spectrum of the reef-coral sand site, are investigated. The findings suggest that the coral sand layer significantly increases the site seismic responses, especially near the edge of the reef flat, thus posing a threat to the seismic safety of the buildings and construction in this area. Furthermore, the deformation of the reef-coral sand site tends to be concentrated on one side, resulting in a certain residual displacement after the earthquake.