We describe the quantum theory of isolated horizons with electromagnetic or non-Abelian gauge charges in a setting in which both gauge and gravitational field are quantized. We consider the distorted case, and its spherically symmetric limit. We show that the gravitational horizon degrees of freedom give rise to the Bekenstein-Hawking relation, with lower order terms giving some corrections for small black holes. We also demonstrate that one can include matter degrees of freedom into the state counting. We show that one can expect (potentially divergent) contributions proportional to the area, as well as logarithmic correction proportional to the horizon charge. This is qualitatively similar to results on matter contributions obtained with other methods in the literature.