We analyze the rate of convergence of the local statistics of Dyson Brownian motion to the GOE/GUE for short times t = o(1) with deterministic initial data V. Our main result states that if the density of states of V is bounded both above and away from 0 down to scales $${\ell \ll t}$$ in a small interval of size $${G \gg t}$$ around an energy $${E_0}$$ , then the local statistics coincide with the GOE/GUE near the energy $${E_0}$$ after time t. Our methods are partly based on the idea of coupling two Dyson Brownian motions from Bourgade et al. (Commun Pure Appl Math, 2016), the parabolic regularity result of Erdős and Yau (J Eur Math Soc 17(8):1927–2036, 2015), and the eigenvalue rigidity results of Lee and Schnelli (J Math Phys 54(10):103504, 2013).