Kelvin probe force microscopy (KPFM) has been used for the characterization of metals, insulators, and semiconducting materials on the nanometer scale. Especially in semiconductors, the charge dynamics are of high interest. Recently, several techniques for time-resolved measurements with time resolution down to picoseconds have been developed, many times using a modulated excitation signal, e.g., light modulation or bias modulation that induces changes in the charge carrier distribution. For fast modulation frequencies, the KPFM controller measures an average surface potential, which contains information about the involved charge carrier dynamics. Here, we show that such measurements are prone to artifacts due to frequency mixing, by performing numerical dynamics simulations of the cantilever oscillation in KPFM subjected to a bias-modulated signal. For square bias pulses, the resulting time-dependent electrostatic forces are very complex and result in intricate mixing of frequencies that may, in some cases, have a component at the detection frequency, leading to falsified KPFM measurements. Additionally, we performed fast Fourier transform (FFT) analyses that match the results of the numerical dynamics simulations. Small differences are observed that can be attributed to transients and higher-order Fourier components, as a consequence of the intricate nature of the cantilever driving forces. These results are corroborated by experimental measurements on a model system. In the experimental case, additional artifacts are observed due to constructive or destructive interference of the bias modulation with the cantilever oscillation. Also, in the case of light modulation, we demonstrate artifacts due to unwanted illumination of the photodetector of the beam deflection detection system. Finally, guidelines for avoiding such artifacts are given.