In recent years, some approaches to realize photonic analog-to-digital conversion based on spectral encoding have been proposed. In these approaches, the amplitude information of input analog signals is first converted into the frequency information of the optical carrier via a nonlinear process, which is equivalent to optical frequency modulation; then, the spectral information of the optical carrier is quantized and encoded with the help of an optical filter group. In this letter, we propose, for the first time to our knowledge, that the time-frequency uncertainty principle puts a fundamental limit on this type of photonic analog-to-digital converters (ADCs). The condition for achieving acceptable performance in terms of effective number of bits is discussed. The presented theoretical results are verified by numerical simulations and a proof-of-concept experiment in electrical domain. We believe that the presented result is important to the design of spectral-encoding-based photonic ADCs.