Context. The vertical shear instability (VSI) is a hydrodynamical instability predicted to produce turbulence in magnetically inactive regions of protoplanetary disks. The regions in which this instability can occur and the physical phenomena leading to its saturation are a current matter of research. Aims. We explore the secondary instabilities triggered by the nonlinear evolution of the VSI and their role in its saturation. We also expand on previous investigations on stability regions by considering temperature stratifications enforced by stellar irradiation and radiative cooling, and including the effects of dust-gas collisions and molecular line emission. Methods. We modeled the gas-dust mixture in a circumstellar disk around a T Tauri star by means of high-resolution axisymmetric radiation-hydrodynamical simulations including stellar irradiation with frequency-dependent opacities, considering different degrees of depletion of small dust grains. Results. The flow pattern produced by the interplay of the axisymmetric VSI modes and the baroclinic torque forms bands of nearly uniform specific angular momentum. In the high-shear regions in between these bands, the Kelvin–Helmholtz instability (KHI) is triggered. A third instability mechanism, consisting of an amplification of eddies by baroclinic torques, forms meridional vortices with Mach numbers up to ∼0.4. Our stability analysis suggests that protoplanetary disks can be VSI-unstable in surface layers up to tens of au for reasonably high gas emissivities. Conclusions. The significant transfer of kinetic energy to small-scale eddies produced by the KHI and possibly even the baroclinic acceleration of eddies limit the maximum energy of the VSI modes, likely leading to the saturation of the VSI. Depending on the gas molecular composition, the VSI can operate at surface layers even in regions where the midplane is stable. This picture is consistent with current observations of disks showing thin midplane millimeter-sized dust layers while appearing vertically extended in optical and near-infrared wavelengths.