An approach is presented for analysis of real-time X-ray reflectivity (XRR) process data not just as a function of the magnitude of the reciprocal-space vector q, as is commonly done, but as a function of both q and time. The real-space structures extracted from the XRR curves are restricted to be solutions of a physics-informed growth model and use state-of-the-art convolutional neural networks (CNNs) and differential evolution fitting to co-refine multiple time-dependent XRR curves R(q, t) of a thin film growth experiment. Thereby it becomes possible to correctly analyze XRR data with a fidelity corresponding to standard fits of individual XRR curves, even if they are sparsely sampled, with a sevenfold reduction of XRR data points, or if the data are noisy due to a 200-fold reduction in counting times. The approach of using a CNN analysis and of including prior information through a kinetic model is not limited to growth studies but can be easily extended to other kinetic X-ray or neutron reflectivity data to enable faster measurements with less beam damage.