We construct a new bigraded Hopf algebra whose bases are indexed by square matrices with entries in the alphabet {0,1,…,k}, k⩾1, without null rows or columns. This Hopf algebra generalizes the one of permutations of Malvenuto and Reutenauer, the one of k-colored permutations of Novelli and Thibon, and the one of uniform block permutations of Aguiar and Orellana. We study the algebraic structure of our Hopf algebra and show, by exhibiting multiplicative bases, that it is free. We moreover show that it is self-dual and admits a bidendriform bialgebra structure. Besides, as a Hopf subalgebra, we obtain a new one indexed by alternating sign matrices. We study some of its properties and algebraic quotients defined through alternating sign matrices statistics.