An effective time-dependent Hamiltonian can be implemented by making a quantum system fly through an inhomogeneous potential, realizing, for example, a quantum gate on its internal degrees of freedom. However, flying systems have a spatial spread that will generically entangle the internal and spatial degrees of freedom, leading to decoherence in the internal state dynamics, even in the absence of any external reservoir. We provide formulas valid at all times for the dynamics, fidelity, and change of entropy for ballistic particles with small spatial spreads, quantified by Δx. This non-Markovian decoherence can be significant for ballistic flying qubits (scaling as Δx^{2}) but usually not for flying qubits carried by a moving potential well (scaling as Δx^{6}). We also discuss a method to completely counteract this decoherence for a ballistic qubit later measured.