The classical Pauli group can be obtained as the central product of the dihedral group of 8 elements with the cyclic group of order 4. Inspired by this characterization, we introduce the notion of central product of Cayley graphs, which allows to regard the Cayley graph of a central product of groups as a quotient of the Cartesian product of the Cayley graphs of the factor groups. We focus our attention on the Cayley graph Cay(Pn,SPn) of the generalized Pauli group Pn on n-qubits; in fact, Pn may be decomposed as the central product of finite 2-groups, and a suitable choice of the generating set SPn allows us to recognize the structure of central product of graphs in Cay(Pn,SPn).Using this approach, we are able to recursively construct the adjacency matrix of Cay(Pn,SPn) for each n≥1, and to explicitly describe its spectrum and the associated eigenvectors. It turns out that Cay(Pn,SPn) is a (3n+2)-regular bipartite graph on 4n+1 vertices, and it has integral spectrum. This is a highly nontrivial property if one considers that, by choosing as a generating set for P1 the three classical Pauli matrices, one gets the so-called Möbius-Kantor graph, belonging to the class of generalized Petersen graphs, whose spectrum is not integral.