Spectrum sensing aims to sense the potential spectrum resources available in the cognitive radio environment. It is also the premise of spectrum management and spectrum sharing in cognitive radio systems. To perceive the primary user’s activity and make full use of spectrum holes, rapid detection of a broad frequency span is an essential part of cognitive radio technology. Reducing the observation time for the data collection, the data storage requirements, and hardware/software computational complexity are urgent and challenging issues in wideband spectrum sensing. High accuracy power spectral density estimation is not the primary requirement; of course, the accuracy must be controlled within the appropriate range and can support the primary user activity’s determination. This paper proposes a sub-Nyquist wideband spectrum sensing method based on compressive covariance sensing for the rapid wideband spectrum sensing. Compared with the traditional Nyquist-rate method, this method can use low-speed ADC to detect wideband signals and effectively control the observation time and computational complexity. This paper’s main contributions include: (1) developing a sub-Nyquist sampling structure based on the multi-coset sampling banks, (2) proposing a coarse-grained power spectral density estimation method for wideband spectrum sensing with short observation time and low complexity. Simulations show that the proposed method exhibits this method is suitable for fast spectral detection. At the same time, the error of spectrum analysis is basically within the acceptable range.