The evaluation of the out-of-plane (OOP) behavior for infilled walls is a critical aspect of regional seismic and fire resilience assessment. This study presents a detailed finite element (FE) model that comprehensively captures the seismic and fire behaviors of masonry walls. The FE model incorporates key factors governing the thermo-mechanical behavior of masonry walls, including temperature-dependent material properties, nonlinearities in geometry and material, block cracking and crushing, and mortar damage. Then, an energy-based damage criterion is proposed based on the principle of energy conservation, leveraging the combined mixed fracture energy and internal energy of elements. In addition, this study defines the damage levels of masonry walls under OOP loading and fire and determines corresponding damage indices using the proposed energy-based damage criterion. This criterion is further compared with a stiffness-based damage criterion. Finally, the proposed damage assessment method is employed to evaluate the damage evolution of infilled walls under seismic, fire and post-earthquake fire scenarios, and to conduct a fragility analysis. The results demonstrate a high degree of agreement between the predicted and measured damage states of masonry walls under seismic and fire loads. More severe seismic damage does not always result in diminished fire resilience. Evaluating post-earthquake fire damage requires considering the impact of displacement directions caused by earthquakes and fires.