In this paper, we introduce new sets of boundary conditions for higher spin gravity in $AdS_3$ where the boundary dynamics of spin two and other higher spin fields are governed by the interacting collective field theory Hamiltonian of Avan and Jevicki. We show that the time evolution of spin two and higher spin fields can be captured by the classical dynamics of folded fermi surfaces in the similar spirit of Lin, Lunin and Maldacena. We also construct infinite sequences of conserved charges showing the integrable structure of higher spin gravity (for spin 3) under the boundary conditions we considered. Further, we observe that there are two possible sequences of conserved charges depending on whether the underlying boundary fermions are non-relativistic or relativistic.