Purpose – The purpose of this paper is to develop an easy-to-implement and accurate fast boundary element method (BEM) for solving large-scale elastodynamic problems in frequency and time domains. Design/methodology/approach – A newly developed kernel-independent fast multipole method (KIFMM) is applied to accelerating the evaluation of displacements, strains and stresses in frequency domain elastodynamic BEM analysis, in which the far-field interactions are evaluated efficiently utilizing equivalent densities and check potentials. Although there are six boundary integrals with unique kernel functions, by using the elastic theory, the authors managed to accelerate these six boundary integrals by KIFMM with the same kind of equivalent densities and check potentials. The boundary integral equations are discretized by Nyström method with curved quadratic elements. The method is further used to conduct the time-domain analysis by using the frequency-domain approach. Findings – Numerical results show that by the fast BEM, high accuracy can be achieved and the computational complexity is brought down to linear. The performance of the present method is further demonstrated by large-scale simulations with more than two millions of unknowns in the frequency domain and one million of unknowns in the time domain. Besides, the method is applied to the topological derivatives for solving elastodynamic inverse problems. Originality/value – An efficient KIFMM is implemented in the acceleration of the elastodynamic BEM. Combining with the Nyström discretization based on quadratic elements and the frequency-domain approach, an accurate and highly efficient fast BEM is achieved for large-scale elastodynamic frequency domain analysis and time-domain analysis.