Active colloidal microswimmers serve as archetypical active fluid systems, and as models for biological swimmers. Here, by studying in detail their velocity traces, we find robust power-law intermittency with system-dependent exponential cut off. We model the intermittent motion by an interplay of the field gradient-dependent active force, which depends on a fluid gradient and is reduced when the swimmer moves, and the locally fluctuating hydrodynamic drag, that is set by the wetting properties of the substrate. The model closely describes the velocity distributions of two disparate swimmer systems: AC field activated and catalytic swimmers. The generality is highlighted by the collapse of all data in a single master curve, suggesting the applicability to further systems, both synthetic and biological.