The transport equations for polarized radiation transfer in non-Riemannian, Weyl-Cartan type space-times are derived, with the effects of both torsion and non-metricity included. To obtain the basic propagation equations we use the tangent bundle approach. The equations describing the time evolution of the Stokes parameters, of the photon distribution function and of the total polarization degree can be formulated as a system of coupled first order partial differential equations. As an application of our results we consider the propagation of the cosmological gamma ray bursts in spatially homogeneous and isotropic spaces with torsion and non-metricity. For this case the exact general solution of the equation for the polarization degree is obtained, with the effects of the torsion and non-metricity included. The presence of a non-Riemannian geometrical background in which the electromagnetic fields couple to torsion and/or non-metricity affect the polarization of photon beams. Consequently, we suggest that the observed polarization of prompt cosmological gamma ray bursts and of their optical afterglows may have a propagation effect component, due to a torsion/non-metricity induced birefringence of the vacuum. A cosmological redshift and frequency dependence of the polarization degree of gamma ray bursts also follows from the model, thus providing a clear observational signature of the torsional/non-metric effects. On the other hand, observations of the polarization of the gamma ray bursts can impose strong constraints on the torsion and non-metricity and discriminate between different theoretical models.