We perform an extensive linear investigation of the nonaxisymmetric disk modes referred to in the literature as P, I, and J modes in self-gravitating polytropic toroids with power law angular velocity distributions. For selected models, we also follow the development of instability from the linear regime through the quasi-linear regime to deep into the nonlinear regime. We consider modes with azimuthal dependence eimφ, where m is an integer and φ is the azimuthal angle. We find that instability sets in through m=2 barlike I modes at T/|W|∼0.16–0.18 depending upon the chosen angular velocity law where T is the rotational kinetic energy and W is the gravitational energy of the toroid. Instability in the barlike I mode peaks in strength around T/|W|=0.22–0.23 after which it weakens, eventually stabilizing around T/|W|∼0.25–0.26. One-armed modes (m=1 modes) become unstable just after instability in the m=2I modes sets in; instability in m=1 modes sets in at T/|W|∼0.19. They dominate the barlike I modes in toroids with T/|W|≳0.25. However, almost immediately after the m=1 mode overtakes the barlike I mode, higher-mJ modes appear. J modes with m=2, 3, and 4 become unstable for T/|W|≳0.25–0.26, 0.23–0.25, and 0.25–0.26, respectively. m≥3J modes dominate the m=1 mode in toroids with T/|W|≳0.27. As T/|W| increases further, nonaxisymmetric instability sets in through higher and higher m modes. We find quantitative agreement between the early nonlinear behavior of the tested unstable toroids and our linear results. Quasi-linear modeling suggests that a gravitational self-interaction torque which arises early in the nonlinear regime saturates growth of the mode and leads to significant transport of mass and angular momentum. Neither I mode nor J mode instabilities produce prompt fission in toroids.