We present a comprehensive structural study of the charge-orbital ordering and magnetic phase transitions observed in the $A$-site ordered $\mathrm{SmBa}{\mathrm{Mn}}_{2}{\mathrm{O}}_{6}$ perovskite combining synchrotron radiation x-ray powder diffraction and symmetry-adapted modes analysis. In $\mathrm{SmBa}{\mathrm{Mn}}_{2}{\mathrm{O}}_{6}$, successive phase transitions in charge, spin, and lattice degrees of freedom take place with decreasing temperature at ${T}_{\mathrm{CO}1}\ensuremath{\approx}380\phantom{\rule{0.16em}{0ex}}\mathrm{K}, {T}_{\mathrm{CO}2}\ensuremath{\approx}190\phantom{\rule{0.16em}{0ex}}\mathrm{K}$, and ${T}_{\mathrm{N}}\ensuremath{\approx}250\phantom{\rule{0.16em}{0ex}}\mathrm{K}$. The main difference between the two charge-ordered phases concerns the stacking sequence along the $c$ axis, which is double for the high temperature charge-ordered phase and has led to controversy in the literature. We show that both charge-ordered phases are pseudosymmetric with respect to the ideal undistorted tetragonal structure of $A$-site ordered $R\mathrm{Ba}{\mathrm{Mn}}_{2}{\mathrm{O}}_{6}$ perovskites and lead to two nonequivalent Mn sites. However, the charge segregation stabilizes at about $0.35{e}^{\ensuremath{-}}$ in the low temperature charge-ordered phase, clearly below the nominal separation of one charge unit between ${\mathrm{Mn}}^{3+}$ and ${\mathrm{Mn}}^{4+}$ and undergoes a prominent increase in the high temperature charge-ordered phase when warming above $\ensuremath{\approx}250\phantom{\rule{0.16em}{0ex}}\mathrm{K}$. The two Mn sites are anisotropic in both charge-ordered phases but the analysis of the active modes discloses that only the low temperature charge-ordered phase displays a Jahn-Teller-like distortion for one of the Mn sites. In addition, this low temperature charge-ordered phase has polar symmetry compatible with ferroelectricity along the $a$ axis.