We show that conservative distributional solutions to the Semi-Geostrophic systems in a rigid domain are in some well-defined sense critical points of a time-shifted energy functional involving measure-preserving rearrangements of the absolute density and momentum, which arise as one-parameter flow maps of continuously differentiable, compactly supported divergence free vector fields. We also show directly, with no recourse to Monge-Kantorovich theory, that the convexity requirement on the modified pressure potentials arises naturally if these critical points are local minimizers of said energy functional for any admissible vector field. The obligatory connection with the Monge-Kantorovich theory is addressed briefly.