We construct a non-Abelian world volume effective action for a system of multiple M-theory gravitons. This action contains multipole moment couplings to the eleven-dimensional background potentials. We use these couplings to study, from the microscopical point of view, giant graviton configurations where the gravitons expand into an M2-brane, with the topology of a fuzzy 2-sphere, that lives in the spherical part of the AdS 7× S 4 background or in the AdS part of AdS 4× S 7. When the number of gravitons is large we find perfect agreement with the Abelian, macroscopical, description of giant gravitons given in the literature.