Coastal box-girder bridges have wide applications in the near-shore transportation systems and many low-lying bridges are under serious threats from extreme waves. However, few researchers paid attention to the dynamic structural behaviors of coastal low-lying bridges under extreme wave conditions, which is crucial to the rational design of these bridges. In this study, a three-dimensional finite element model, including the local nonlinear bearing material model and wave loading importation from the computational fluid dynamics (CFD) simulations, is established to investigate the dynamic behavior of the bridge structure. Specifically, to provide a more scientific and precise evaluation for the dynamic behavior of coastal bridges, the laminated rubber bearing model is established at joints between the superstructure/box-girder and bent cap using solid elements. Three common failure modes, including the large lateral shear deformation of the rubber bearing, sliding between the girder and bearings, and overturning of the superstructure, are investigated in detail. To consider the uncertainty under different loading conditions, one surrogate model based on the support vector machine (SVM) is adopted to enhance the calculation efficiency of the vulnerability assessment for the coastal box-girder bridge. It is revealed that (a) The overturning and sliding failures are more likely to occur for the box-girder in submerged states, while the large lateral shear deformation for the bearing is prone to arise when the girder is elevated; (b) The wave height to water depth ratio has a significant influence on the safety of the bridge; and (c) The high failure probability indicates that appropriate countermeasures are necessary to improve the safety margin of the bridge.