We study the phase fluctuations in the normal state of generic two-dimensional superconducting systems with $s$-wave pairing. The effect of phase fluctuations of the pairing fields can be dealt with perturbatively using disorder averaging, after we treat the local superconducting order parameter as a static disordered background. It is then confirmed that the phase fluctuations above the two-dimensional Berezinskii-Kosterlitz-Thouless transition lead to a significant broadening of the single-particle spectrum, giving birth to the pseudogap phenomenon. Quantitatively, the broadening of spectral weights near the BCS gap is characterized by the ratio of the superconducting coherence length and the spatial correlation length of the superconducting pairing order parameter. Our results are tested on the fermionic attractive-$U$ Hubbard model on the square lattice, using the unbiased determinant quantum Monte Carlo method and stochastic analytic continuation.