We study the circularly polarized photoluminescence of negatively charged (${\mathrm{NV}}^{\ensuremath{-}}$) and neutral (${\mathrm{NV}}^{0}$) nitrogen-vacancy ensembles and neutral vacancies (${\mathrm{V}}^{0}$) in diamond crystals exposed to magnetic fields of up to 10 T. We determine the orbital and spin Zeeman splitting as well as the energetic ordering of their ground and first-excited states. The spin-triplet and -singlet states of the ${\mathrm{NV}}^{\ensuremath{-}}$ are described by an orbital Zeeman splitting of about 9 $\ensuremath{\mu}\mathrm{eV}/\mathrm{T}$, which corresponds to a positive orbital $g$-factor of ${g}_{L}=0.164$ under application of the magnetic field along the (001) and (111) crystallographic directions, respectively. The zero-phonon line (ZPL) of the ${\mathrm{NV}}^{\ensuremath{-}}$ singlet is defined as a transition from the ${}^{1}{E}^{\ensuremath{'}}$ states, which are split by ${g}_{L}{\ensuremath{\mu}}_{\text{B}}B$, to the ${}^{1}{A}_{1}$ state. The energies of the zero-phonon triplet transitions show a quadratic dependence on intermediate magnetic field strengths, which we attribute to a mixing of excited states with nonzero orbital angular momentum. Moreover, we identify slightly different spin Zeeman splittings in the ground (gs) and excited (es) triplet states, which can be expressed by a deviation between their spin $g$-factors: ${g}_{S,\text{es}}={g}_{S,\text{gs}}+\mathrm{\ensuremath{\Delta}}g$ with values of $\mathrm{\ensuremath{\Delta}}g=0.014$ and 0.029 in the (001) and (111) geometries, respectively. The degree of circular polarization of the ${\mathrm{NV}}^{\ensuremath{-}}$ ZPLs depends significantly on the temperature, which is explained by an efficient spin-orbit coupling of the excited states mediated through acoustic phonons. We further demonstrate that the sign of the circular polarization degree is switched under rotation of the diamond crystal. A weak Zeeman splitting similar to $\mathrm{\ensuremath{\Delta}}g{\ensuremath{\mu}}_{\text{B}}B$ measured for the ${\mathrm{NV}}^{\ensuremath{-}}$ ZPLs is also obtained for the ${\mathrm{NV}}^{0}$ zero-phonon lines, from which we conclude that the ground state is composed of two optically active states with compensated orbital contributions and opposite spin-1/2 momentum projections. The zero-phonon lines of the ${\mathrm{V}}^{0}$ show Zeeman splittings and degrees of the circular polarization with opposite signs. The magnetophotoluminescence data indicate that the electron transition from the $^{1}T_{2}$ states to the $^{1}A$ ground state defines the zero-phonon emission at 1.674 eV, while the $^{1}T_{2}\ensuremath{\rightarrow}^{1}E$ transition is responsible for the zero-phonon line at 1.666 eV. The $^{1}T_{2}$ ($^{1}E$) states are characterized by an orbital Zeeman splitting with ${g}_{L}=0.071$ (0.128).