Parametrized quantum optical circuits are a class of quantum circuits in which the carriers of quantum information are photons and the gates are optical transformations. Classically optimizing these circuits is challenging due to the infinite dimensionality of the photon number vector space that is associated to each optical mode. Truncating the space dimension is unavoidable, and it can lead to incorrect results if the gates populate photon number states beyond the cutoff. To tackle this issue, we present an algorithm that is orders of magnitude faster than the current state of the art, to recursively compute the exact matrix elements of Gaussian operators and their gradient with respect to a parametrization. These operators, when augmented with a non-Gaussian transformation such as the Kerr gate, achieve universal quantum computation. Our approach brings two advantages: first, by computing the matrix elements of Gaussian operators directly, we don't need to construct them by combining several other operators; second, we can use any variant of the gradient descent algorithm by plugging our gradients into an automatic differentiation framework such as TensorFlow or PyTorch. Our results will find applications in quantum optical hardware research, quantum machine learning, optical data processing, device discovery and device design.