The extraordinary optical properties of single-layer graphene have spurred the development of a variety of photonic components. We have previously demonstrated a scalable and versatile platform to facilitate the integration of graphene and other 2-D materials with chalcogenide glass-based planar photonics. In this paper, we detail the design criteria and optimization guidelines towards high-performance graphene-integrated thermo-optic (TO) switches based on the chalcogenide glass-on-graphene platform. Notably, absorption loss of graphene can be reduced to < 20 dB/cm when it is sandwiched inside photonic structures capitalizing on the anisotropic absorption property of graphene. We quantify energy efficiency of the TO switch, showing that the choice of cladding materials plays a critical role in improving device efficiency. Furthermore, we report a record TO switching efficiency of 10 nm/mW via judicious engineering of the overlap between optical mode and thermal profile.