The reconstruction of cosmic ray-induced air showers from measurements of radio waves constitutes a major challenge. In this work, we focus on recovering the full three-dimensional electromagnetic field from two recorded signal traces of an antenna station covering two horizontal polarization directions. The simulated field is folded by a direction and frequency-dependent characteristic antenna response pattern, resulting in voltage signal traces as a function of time. Both signal traces are contaminated by simulated background noise. We use conditional Invertible Neural Networks (cINNs) to learn posterior distributions, from which the most likely electromagnetic field given a measured signal trace can be inferred. To improve robustness, we extend the method with an autoencoder by reducing the parameter phase space and decoupling the cINN from specific data shapes. Thereby, each signal trace is condensed into a small number of abstract parameters in the latent space on which the cINN operates. The presented method shows promising results and can be transferred to other unfolding problems where the recovery of the pre-measurement state is of interest.