The stability prediction and unsteady-flow characteristic explanations of centrifugal compressors mostly focus on radial vaneless diffusers with parallel walls (PWs). However, vaneless diffusers with curved shroud profiling are more often adopted in a practical compression system owing to their better performance or larger stability margin. In this work, the stability properties of the unsteady flow in the radial vaneless diffusers with parallel walls and curved shroud profiling were studied in the context of numerical simulations and global modes. First, the unsteady simulations of a full annular compressor, including vaneless diffusers with parallel walls and constant area walls, were carried out. The simulation results were validated against experimental data, and the dominant modes of the diffuser flow under stall conditions were captured using dynamic mode decomposition (DMD) analysis. Then, a frozen eddy stability analysis was performed on the vaneless diffusers, and the global mode associated with the flow perturbations was investigated. Finally, a continuous adjoint method was adopted to reveal the receptivity of the instability perturbations. By focusing on the effect of the curved shroud profiling on the stability characteristics, the global and adjoint eigenmodes corresponding to the onset of instability were used to provide a physical explanation for the stall mechanism. The simulation results indicated that the stability margin for diffuser with constant area walls was 2.15% wider than that with PWs. The DMD and global stability-analysis results showed that the diffuser with constant area walls had a stall frequency of 298 Hz, which was larger than that of diffuser with PWs. But both cases shared the same stall number of 4. The value of growth rate came to its minimum at 30% span and maximum at 95% span in both cases, indicating the dominant flow fields were near the shroud walls. For the vaneless diffuser with PWs, the reverse flow at 95% span within the radius range of r = 1.3–1.4 was the dominant reason for the occurrence of instability with large value of direct and adjoint mode. Meanwhile, flow mode indicated that flow instability of the diffuser with constant area walls was caused by the reverse flow at 90% span with the radius range of r = 1.02–1.1.