The gravito-magneto-modulational (GEM) instability of a self-gravitating system is investigated on the basis of magnetic-type Maxwell-Vlasov equations. Analytical and numerical results indicate that the quasistatic self-excited gravitomagnetic field ismodulationally unstable with respect to the uniform state of a self-gravitating system. Such an instability would strongly localize and amplify low-frequency self-excited gravitomagnetic fields, the density ofmatter and gravitoelectric fields. This new and interesting state of matter is in favor of the formation of a new rotating object. The characteristic scale and maximum growth rate of quasistatic self-excited gravitomagnetic fields depend on the strength of a pump wave.