In this paper, a typical brick masonry loess cave (BMLC) is selected, and finite element analysis software is used to establish a model and analyse its seismic performance. The model was verified by the test results of the BMLC on a 3D shaking table, and the damage quantification index was proposed. On this basis, seismic vulnerability analysis was carried out. A method based on nonlinear incremental dynamic analysis (IDA) was proposed to solve the rapid damage assessment of the traditional building of BMLC structures after earthquake action. The proposed method relies on an accurate numerical finite element model and pre-run multidimensional nonlinear IDA simulations with vibration data recorded during earthquakes. The IDA curves were built with different positions of the BMLC structure being considered. Moreover, meaningful local damage parameters were correlated with relevant seismic intensity measures. The results showed that the proposed method immediately and accurately estimated damage after seismic action if the IDA curve set had been previously constructed using a suitable model.