Context. Numerous studies have reported significant displacements in the coordinates of active galactic nuclei (AGNs) between measurements using the very-long-baseline-interferometry (VLBI) technique and those obtained by the Gaia space observatory. There is consensus that these discrepancies do indeed manifest astrometrically resolved sub-components of AGNs rather than random measurement noise. Among other evidence, it has been reported that AGNs with VLBI-to-Gaia displacements (VGDs) pointing downstream of their parsec-scale radio jets exhibit higher optical polarization compared to sources with the opposite (upstream) VGD orientation. Aims. We aim to verify the previously reported connection between optical polarization and a VGD-jet angle using a larger dataset of polarimetric measurements and updated Gaia DR3 positions. We also seek further evidence supporting the disk-jet dichotomy as an explanation of such a connection by using millimeter-wave polarization and multiband optical polarization measurements. Methods. We performed optical polarimetric observations of 152 AGNs using three telescopes. These data are complemented by other publicly available polarimetric measurements of AGNs. We cross-matched public astrometric data from VLBI and Gaia experiments, obtained corresponding positional displacements, and combined this catalog with the polarimetric and jet direction data. Results. Active galactic nuclei with downstream VGDs are confirmed to have significantly higher optical fractional polarization than the upstream sample. At the same time, the millimeter-wavelength polarization of the two samples shows very similar distributions. Conclusions. Our results support the hypothesis that the VGDs pointing down the radio jet are likely caused by a component in the jet emitting highly polarized synchrotron radiation and dominating in the overall optical emission. The upstream-oriented VGDs are likely to be produced by the low-polarization emission of the central engine’s subcomponents, which dominate in the optical.