Predictive design and optimization methods for controlled quantum systems depend on the accuracy of the system model. Any distortion of the input fields in an experimental platform alters the model accuracy and eventually disturbs the predicted dynamics. These distortions can be nonlinear with a strong frequency dependence so that the field interacting with the microscopic quantum system has limited resemblance to the input signal. We present an effective method for estimating these distortions, which is suitable for nonlinear transfer functions of arbitrary lengths and magnitudes provided the available training data has enough spectral components. Using a quadratic estimation, we have successfully tested our approach for a numerical example of a single Rydberg atom system. The transfer function estimated from the presented method is incorporated into an open-loop control optimization algorithm allowing for high-fidelity operations in quantum experiments.