Because mixtures containing arbitrary ratios of ortho ($J=1$) and para ($J=0$) concentrations can be prepared, the condensed phases of hydrogen afford a unique opportunity to study interactions between isotropic and anisotropic molecules which are chemically identical. Detailed measurements are reported here of the rotational Raman spectra in the liquid and solid phases of ortho-para hydrogen mixtures, with para concentrations between 25% and 96% at temperatures between 10 and 23\ifmmode^\circ\else\textdegree\fi{}K. Changes in the configuration and dynamics of the molecular environment with temperature and ortho concentration are clearly reflected in the rotational Raman line shapes. Logarithmically recorded spectra, covering an intensity dynamic range of typically ${10}^{4}$, reveal two relatively independent types of interaction which respectively govern the near (30 ${\mathrm{cm}}^{\ensuremath{-}1}$) and far (>30 ${\mathrm{cm}}^{\ensuremath{-}1}$) wings. Quantum calculations of the low-frequency mechanism, electric quadrupole-quadrupole molecular pair interactions, are presented. They are found to produce values for the second frequency moments for all three ($J=1\ensuremath{\rightarrow}1$, $J=1\ensuremath{\rightarrow}3$, and $J=0\ensuremath{\rightarrow}2$) rotational transitions which agree well both in absolute magnitude and in concentration dependence with experimental observations. The far wings in the solid spectra consist of phonon sidebands whose structure is in good agreement with the phonon density of states previously observed by neutron scattering. In the liquid mixtures the far-wing line shapes are insensitive to temperature or concentration variation and are very similar to the solid wings.