The magnetism in the sawtooth lattice of Mn in the olivine chalcogenides, ${\mathrm{Mn}}_{2}{\mathrm{SiS}}_{4\ensuremath{-}x}{\mathrm{Se}}_{x}$ ($x=1--4$), is studied in detail by analyzing their magnetization, specific heat, and thermal conductivity properties and complemented with density functional theory calculations. The air-stable chalcogenides are antiferromagnets and show a linear trend in the transition temperature ${T}_{N}$ as a function of Se content ($x$), which shows a decrease from ${T}_{N}\ensuremath{\approx}86\phantom{\rule{0.16em}{0ex}}\mathrm{K}$ for ${\mathrm{Mn}}_{2}{\mathrm{SiS}}_{4}$ to 66 K for ${\mathrm{Mn}}_{2}{\mathrm{SiSe}}_{4}$. Additional magnetic anomalies are revealed at low temperatures for all the compositions. Magnetization irreversibilities are observed as a function of $x$. The specific heat and the magnetic entropy indicate the presence of short-range spin fluctuations in ${\mathrm{Mn}}_{2}{\mathrm{SiS}}_{4\ensuremath{-}x}{\mathrm{Se}}_{x}$. A spin-flop antiferromagnetic phase transition in the presence of applied magnetic field is present in ${\mathrm{Mn}}_{2}{\mathrm{SiS}}_{4\ensuremath{-}x}{\mathrm{Se}}_{x}$, where the critical field for the spin flop increases from $x=0$ towards 4 in a nonlinear fashion. Density functional theory calculations show that an overall antiferromagnetic structure with ferromagnetic coupling of the spins in the $ab$ plane minimizes the total energy. The band structures calculated for ${\mathrm{Mn}}_{2}{\mathrm{SiS}}_{4}$ and ${\mathrm{Mn}}_{2}{\mathrm{SiSe}}_{4}$ reveal features near the band edges similar to those reported for Fe-based olivines suggested as thermoelectrics; however the experimentally determined thermal transport data do not support superior thermoelectric features. The transition from long-range magnetic order in ${\mathrm{Mn}}_{2}{\mathrm{SiS}}_{4}$ to short-range order and spin fluctuations in ${\mathrm{Mn}}_{2}{\mathrm{SiSe}}_{4}$ is explained using the variation of the Mn-Mn distances in the triangle units that constitutes the sawtooth lattice upon progressive replacement of sulfur with selenium. Overall, the results presented here point towards the role played by magnetic anisotropy and geometric frustration in the antiferromagnetic state of the sawtooth olivines.