Estimating the impact of systematic uncertainties in particle physics experiments is challenging, especially since the detector response is unknown analytically in most situations and needs to be estimated through Monte Carlo (MC) simulations. Typically, detector property variations are parameterized in ways that implicitly assume a specific physics model, which can introduce biases on quantities measured by an analysis. In this paper, we present a method to recover a model-independent, event-wise estimation of the detector response variation by applying a likelihood-free inference method to a set of MC simulations representing discrete detector realizations. The method provides a re-weighting scheme for every event, which can be used to apply the effects of detector property variations fully decoupled from the assumed physics model. Using a toy MC example inspired by fixed-baseline neutrino oscillation experiments, we demonstrate the performance of our method. We show that it fully decouples the modeling of the detector response from the physics parameters to be measured in a MC forward-folding analysis.