The kinetic Sunyaev Zel'dovich (kSZ) and polarized Sunyaev Zel'dovich (pSZ) effects are temperature and polarization anisotropies induced by the scattering of CMB photons from structure in the post-reionization Universe. In the case of the kSZ effect, small angular scale anisotropies in the optical depth are modulated by the cosmic microwave background (CMB) dipole field, i.e. the CMB dipole observed at each spacetime point, which is sourced by the primordial dipole and especially the local peculiar velocity. In the case of the pSZ effect, similar small-scale anisotropies are modulated by the CMB quadrupole field, which receives contributions from both scalar and tensor modes. Statistical anisotropies in the cross correlations of CMB temperature and polarization with tracers of the inhomogeneous distribution of electrons provide a means of isolating and reconstructing the dipole and quadrupole fields. In this paper, we present a set of unbiased minimum variance quadratic estimators for the reconstruction of the dipole and quadrupole fields, and forecast the ability of future CMB experiments and large scale structure surveys to perform this reconstruction. Consistent with previous work, we find that a high fidelity reconstruction of the dipole and quadrupole fields over a variety of scales is indeed possible, and demonstrate the sensitivity of the pSZ effect to primordial tensor modes. Using a principle component analysis, we estimate how many independent modes could be accessed in such a reconstruction. We also comment on a few first applications of a detection of the dipole and quadrupole fields, including a reconstruction of the primordial contribution to our locally observed CMB dipole, a test of statistical homogeneity on large scales from the first modes of the quadrupole field, and a reconstruction technique for the primordial potential on the largest scales.