As realized recently, the success of delayed feedback control methods may be significantly restricted by control loop latency, i.e., by an additional delay which acts on the control force. We show within a linear stability analysis that such a limitation is caused by the shift of frequency splitting points. Our analytical results are in good quantitative agreement with numerical ``exact'' calculations of the Toda oscillator and with data from an electronic circuit experiment.