Systems subject to high-frequency driving exhibit Floquet prethermalization, that is, they heat exponentially slowly on a timescale that is large in the drive frequency, τ_{h}∼exp(ω). Nonetheless, local observables can decay much faster via energy conserving processes, which are expected to cause a rapid decay in the fidelity of an initial state. Here we show instead that the fidelities of eigenstates of the time-averaged Hamiltonian, H_{0}, display an exponentially long lifetime over a wide range of frequencies-even as generic initial states decay rapidly. When H_{0} has quantum scars, or highly excited eigenstates of low entanglement, this leads to long-lived nonthermal behavior of local observables in certain initial states. We present a two-channel theory describing the fidelity decay time τ_{f}: the interzone channel causes fidelity decay through energy absorption, i.e., coupling across Floquet zones, and ties τ_{f} to the slow heating timescale, while the intrazone channel causes hybridization between states in the same Floquet zone. Our work informs the robustness of experimental approaches for using Floquet engineering to generate interesting many-body Hamiltonians, with and without scars.