ABSTRACT We examine the onset of viscous overstability in dense self-gravitating particle rings. This oscillatory instability offers a plausible explanation for the periodic radial density variations seen at several locations of Saturn’s B and inner-A ring. So far, the theoretical understanding of overstable ring systems relies mainly on analytical results based on approximate treatments of ring self-gravity which omit the emergence of self-gravity wakes (small-scale gravitational instabilities). Therefore, the interplay between the two mechanisms, self-gravity and overstability, is still not well understood. Here, we address numerically the factors that determine the onset of overstability in self-gravitating rings. We confirm that weak self-gravity promotes overstability whereas strong self-gravity, with prominent wake structure, weakens the overstable pattern. This strong gravity regime corresponds to optical depths beyond a threshold value. In systems where overstable oscillations and wakes co-exist, we detect a strong anticorrelation between the strength of wakes and overstable oscillations. It is this interaction that eventually leads to the suppression of overstability, with strong transient wake structure preventing the phase synchronization between density and velocity oscillations required to maintain a coherent overstable pattern. Finally, we derive a composite criterion for the onset of overstability valid even in the regime of strong gravity wakes, framed in terms of collective properties of the system: the central-plane filling factor and the ratio of gravitational and collisional viscosities. In the limit where gravity wakes are omitted, our new criterion agrees with that found by the kinetic theory analysis of linear stability.