Abstract Zonal jets and nonzonal large-scale flows are often present in forced–dissipative barotropic turbulence on a beta plane. The dynamics underlying the formation of both zonal and nonzonal coherent structures is investigated in this work within the statistical framework of stochastic structural stability theory (S3T). Previous S3T studies have shown that the homogeneous turbulent state undergoes a bifurcation at a critical parameter and becomes inhomogeneous with the emergence of zonal and/or large-scale nonzonal flows and that these statistical predictions of S3T are reflected in direct numerical simulations. In this paper, the dynamics underlying the S3T statistical instability of the homogeneous state as a function of parameters is studied. It is shown that, for weak planetary vorticity gradient β, both zonal jets and nonzonal large-scale structures form from upgradient momentum fluxes due to shearing of the eddies by the emerging infinitesimal flow. For large β, the dynamics of the S3T instability differs for zonal and nonzonal flows but in both the destabilizing vorticity fluxes decrease with increasing β. Shearing of the eddies by the mean flow continues to be the mechanism for the emergence of zonal jets while nonzonal large-scale flows emerge from resonant and near-resonant triad interactions between the large-scale flow and the stochastically forced eddies. The relation between the formation of large-scale structure through modulational instability and the S3T instability of the homogeneous state is also investigated and it is shown that the modulational instability results are subsumed by the S3T results.