We address the non-Markovian character of quantum maps describing the interaction of a qubit with a random classical field. In particular, we evaluate trace- and capacity-based non-Markovianity measures for two relevant classes of environments showing non-Gaussian fluctuations, described respectively by random telegraph noise and colored noise with spectra of the the form $1/f^\alpha$. We analyze the dynamics of both the trace distance and the quantum capacity, and show that the behavior of non-Markovianity based on both measures is qualitatively similar. Our results show that environments with a spectrum that contains a relevant low-frequency contribution are generally non-Markovian. We also find that the non-Markovianity of colored environments decreases when the number of fluctuators realizing the environment increases.