Advances in additive manufacturing have been paving the way for unconventional topologies in electromechanical systems. As such, topology optimization can be employed to uncover topologies that were once considered impractical. This article investigates enhancing the shear stress of the reluctance trans-rotary magnetic gear (TROMAG) by optimizing the topology of the gear’s variable reluctance component. Reluctance TROMAG is a strong contender for force-intensive linear motion applications where a long stroke is required. Topology optimization is performed on the reluctance TROMAG with radially magnetized permanent magnet (PM) configuration as well as with quasi-Halbach magnetized configuration. Results show that the use of elliptical iron teeth backed by non-ferromagnetic core can make a 40% increase compared to the shear stress of the conventional (rectangular teeth with iron core) reluctance TROMAG with radial magnetization. The Halbach-magnetized TROMAG with elliptical iron teeth backed by non-ferromagnetic core achieves the same shear stress as the conventional reluctance TROMAG with Halbach magnetization, and the use of lightweight non-ferromagnetic material for the variable reluctance component can make significant reductions in the total gear weight.