For a smooth, stationary, planar Gaussian field, we consider the number of connected components of its excursion set (or level set) contained in a large square of area $R^2$. The mean number of components is known to be of order $R^2$ for generic fields and all levels. We show that for certain fields with positive spectral density near the origin (including the Bargmann-Fock field), and for certain levels $\ell$, these random variables have fluctuations of order at least $R$, and hence variance of order at least $R^2$. In particular, this holds for excursion sets when $\ell$ is in some neighbourhood of zero, and it holds for excursion/level sets when $\ell$ is sufficiently large. We prove stronger fluctuation lower bounds of order $R^\alpha$, $\alpha \in [1,2]$, in the case that the spectral density has a singularity at the origin. Finally, we show that the number of excursion/level sets for the Random Plane Wave at certain levels has fluctuations of order at least $R^{3/2}$, and hence variance of order at least~$R^3$. We expect that these bounds are of the correct order, at least for generic levels.