Infilled reinforced concrete (IRC) frames are a very common construction typology, not only in developing countries such as Pakistan but also in southern Europe and Western countries, due to their ease of construction and less technical skills required for the construction. Their performance during past earthquakes has been in some cases satisfactory and in other cases inadequate. Significant effort has been made among researchers to improve such performance, but few have highlighted the influence of construction materials used in the infill walls. In some building codes, infills are still considered as non-structural elements, both in the design of new buildings and, sometimes, in the assessment of existing buildings. This is mainly due to some difficulties in modeling their mechanical behavior and also the large variety of typologies, which are difficult to categorize. Some building codes, for example, Eurocode, already address the influence of infill walls in design, but there is still a lack of homogeneity among different codes. For example, the Pakistan building code (PBC) does not address infills, despite being a common construction technique in the country. Past earthquake survey records show that construction materials and infill types significantly affect the seismic response of buildings, thus highlighting the importance of investigating such parameters. This is the object of this work, where a numerical model for infill walls is introduced, which aims at predicting their failure mode, as a function of some essential parameters, such as the friction coefficient between mortar and brick surface and mortar strength, usually disregarded in previous models. A comprehensive case study is presented of a three-story IRC frame located in the city of Mirpur, Pakistan, hit by an earthquake of magnitude 5.9 on 24 September 2019. The results obtained from the numerical model show good agreement with the damage patterns observed in situ, thus highlighting the importance of correctly modeling the infill walls when seismically designing or assessing Pakistani buildings that make use of this technology.