In the seismic analysis of arch dams, the heterogeneity of the foundation rock is always not included with the use of assumption of a homogeneous half-space. This study is devoted to the analysis of dam-water-foundation rock interaction considering a complex layered half-space. For this purpose, a novel scaled boundary finite element method (SBFEM) for the analysis of 3D dam-foundation interaction is developed. The novel SBFEM can satisfy the radiation damping rigorously of a complex layered half-space, which overcomes the limit of scaling requirements and broadens the application of the original SBFEM. In addition, two efficient models in the framework of SBFEM have been established in the early research, respectively, to solve hydrodynamic analysis of dam-water interaction and free-field motion of an infinite canyon. Thus, based on the substructure method, a SBFEM procedure and the corresponding program, named SBFEM-LEACD, is developed to solve the earthquake analysis of arch dams considering a complex layered half-space. The response of an arch dam at the design stage in southwest China, where the foundation rock is idealized as three kinds of model: homogeneous half-space, horizontal layered half-space and inclined half-space, is analyzed to demonstrate the effect of the complex layered half-space.