Heat pump technology offers a path towards reducing the use of fossil fuels to heat space, providing energy bill savings and reducing air pollution and GHG emissions. The choice of heating method is based on costs; hence, this study examines the gains from operating heat pump systems in public buildings as well as alternative systems using electricity, LPG, and heating oil. The study focuses on the Ruda-Huta municipality in Poland that, as is common in rural areas, lacks access to a district heating system or piped gas. The empirical analysis includes heat pump installations in eight municipal buildings. The study found that the use of ground source heat pumps proved competitive with existing heating systems in terms of payback time. Calculations for three heating energy source scenarios, i.e., electricity, LPG, and heating oil, used the Simple Pay Back Time (SPBT) and the Levelized Cost of Heat (LCOH) methods and the average prices of the three energy types for the period 2012–2021. The payback period calculations disregarded the EU subsidies for heating systems utilizing renewable energy sources (RES). The payback time for electric, LPG, and heating oil were, respectively, 6.7–7.8 years, 4.1–6.1 years, and 6.7–6.9 years. Much larger spreads favoring heat pumps were calculated using the LCOH, and the costs in the case of electric heating were nearly three times higher and doubled when using heating oil and LPG. The gains from investing in heat pump systems have been offset by the increase in electricity, LPH, and heating oil prices, which have been predicted to continue to increase in the foreseeable future supporting the use of heat pumps in rural areas lacking access to, for example, district heating systems. The switch to heat pumps reduced local air pollution by eliminating the burning of fossil fuels to heat space in public buildings.