We introduce a class of permutation centralizer algebras which underly the combinatorics of multi-matrix gauge invariant observables. One family of such non-commutative algebras is parametrised by two integers. Its Wedderburn-Artin decomposition explains the counting of restricted Schur operators, which were introduced in the physics literature to describe open strings attached to giant gravitons and were subsequently used to diagonalize the Gaussian inner product for gauge invariants of 2-matrix models. The structure of the algebra, notably its dimension, its centre and its maximally commuting sub-algebra, is related to Littlewood-Richardson numbers for composing Young diagrams. It gives a precise characterization of the minimal set of charges needed to distinguish arbitrary matrix gauge invariants, which are related to enhanced symmetries in gauge theory. The algebra also gives a star product for matrix invariants. The centre of the algebra allows efficient computation of a sector of multi-matrix correlators. These generate the counting of a certain class of bi-coloured ribbon graphs with arbitrary genus.