We propose a generalization of the Saad-Shenker-Stanford duality relating matrix models and JT gravity to the case in which the bulk includes higher spin fields. Using a PSL(N, ℝ) BF theory we compute the disk and generalization of the trumpet partition function in this theory. We then study higher genus corrections and show how this differs from the usual JT gravity calculations. In particular, the usual quotient by the mapping class group is not enough to ensure finite answers and so we propose to extend this group with additional elements that make the gluing integrals finite. These elements can be thought of as large higher spin diffeomorphisms. The cylinder contribution to the spectral form factor then behaves as TN−1 at late times T, signaling a deviation from conventional random matrix theory. To account for this deviation, we propose that the bulk theory is dual to a matrix model consisting of N − 1 commuting matrices associated to the N − 1 conserved higher spin charges.We find further evidence for the existence of the additional mapping class group elements by interpreting the bulk gauge theory geometrically and employing the formalism developed by Gomis et al. in the nineties. This formalism introduces additional (auxiliary) boundary times so that each conserved charge generates translations in those new directions. This allows us to find an explicit description for the PSL(3, ℝ) Schwarzian theory for the disk and trumpet and view the additional mapping class group elements as ordinary Dehn twists, but in higher dimensions.