Although the magnetic fields in the quiet Sun account for the majority of the magnetic energy in the solar photosphere, inferring their exact spatial distribution, origin, and evolution poses an important challenge because the signals lie at the limit of today's instrumental precision. This severely hinders and biases our interpretations, which are mostly made through nonlinear model-fitting approaches. Our goal is to directly compare simulated and observed polarization signals in the Fe I $ angstrom $ and $ angstrom $ spectral lines in the very quiet Sun, the so-called solar internetwork (IN). This way, we aim to constrain the mechanism responsible for the generation of the quiet Sun magnetism while avoiding the biases that plague other diagnostic methods. We used three different three-dimensional radiative magneto-hydrodynamic simulations representing different scenarios of magnetic field generation in the internetwork: small-scale dynamo, decay of active regions, and horizontal flux emergence. We synthesized Stokes profiles at different viewing angles and degraded them according to the instrumental specifications of the spectro-polarimeter (SP) on board the Hinode satellite. Finally, we statistically compared the simulated spectra to the Hinode/SOT/SP observations at the appropriate viewing angles. Of the three simulations, the small-scale dynamo best reproduced the statistical properties of the observed polarization signals. This is especially prominent for the disk center viewing geometry, where the agreement is excellent. Moving toward more inclined lines of sight, the agreement worsens slightly. The agreement between the small-scale dynamo simulation and observations at the disk center suggests that small-scale dynamo action plays an important role in the generation of quiet Sun magnetism. However, the magnetic field around 50 km above the continuum layer in this simulation does not reproduce observations as well as at the very base of the photosphere.