The stochastic gravitational wave background (SGWB) is a rich resource of cosmological information, encoded both in its source statistics and anisotropies induced by propagation effects. We provide a theoretical description of it, without employing tools which rely on the geometric optics assumption. Our formalism is based on the so-called classical matter approximation and it is able to capture wave-optics effects, such as interference and diffraction. We show that the interaction between the gravitational waves and the cosmic structures along the line-of-sight produce observable scalar and vector polarization modes, on top of modulating the tensorial ones. We build the two point correlation function describing the statistics of the SGWB, and introduce the intensity and gravitational Stokes parameters for all its components. In the case of an unpolarized, Gaussian and statistically homogeneous SGWB, we show that the interaction with matter modulates its intensity and does not generate a net difference between left- and right-helicity tensor and vector modes, as expected. We demonstrate that, in order to produce QT- and UT-tensor polarization modes the background must possess a hexadecapole anisotropy, while a quadrupole anisotropy can source the vector QV- and UV-Stokes parameters.