Volcanic tuffs have a historical tradition of usage in Northern Hungary as dimension stones for monumental construction, Ottoman architecture, common dwellings, etc., admirable at its best in the medieval castles of Eger and Sirok. This research explores tuff deterioration in the castle walls, dealing with the mineralogical composition, microstructure, trace-element geochemistry, and microporosity of the surface weathering products and the near-surface stone substrate. The classic microscopic and mineralogical techniques–optical microscopy, SEM-EDS, and XRD–were supported by ICP-MS and nitrogen adsorption analyses. The textures and mineral assemblages of the tuffs are partly diverse, and so are the weathering characteristics, although including common features such as secondary crystallization of gypsum, swelling clay minerals, and iron oxides-hydroxides; deposition of airborne pollutants, i.e., carbon particles and heavy metals; formation of crusts and patinas; decreased surface microporosity. Nonetheless, the entity of deterioration varies, in relation to air pollution–involving changing emissions from road and rail transport–and the specific tuff texture, porosity, and durability–affecting pollutant absorption. The studied stone monuments offer the possibility to examine materials with analogue composition and petrogenesis but utilized in different environmental contexts, which allow pointing out the environmental and lithological constraints and cause-effect relationships related to surface weathering.