Multi-plane light converters (MPLCs) are an emerging class of optical devices capable of converting a set of input spatial light modes to a new target set of output modes. This operation represents a linear optical transformation—a much sought after capability in photonics. MPLCs have potential applications in both the classical and quantum optics domains, in fields ranging from optical communications to optical computing and imaging. They consist of a series of diffractive optical elements (the “planes”), typically separated by a free space. The phase delays imparted by each plane are determined by the process of inverse-design, most often using an adjoint algorithm known as the wavefront matching method (WMM), which optimizes the correlation between the target and actual MPLC outputs. In this work, we investigate high mode capacity MPLCs to create arbitrary spatial mode sorters and linear optical circuits. We focus on designs possessing low numbers of phase planes to render these MPLCs experimentally feasible. To best control light in this scenario, we develop a new inverse-design algorithm, based on gradient ascent with a specifically tailored objective function, and show how, in the low-plane limit, it converges to MPLC designs with a substantially lower modal cross-talk and higher fidelity than those achievable using the WMM. We experimentally demonstrate several prototype few-plane high-dimensional spatial mode sorters, operating on up to 55 modes, capable of sorting photons based on their Zernike mode or orbital angular momentum state, or an arbitrarily randomized spatial mode basis. We discuss the advantages and drawbacks of these proof-of-principle prototypes and describe future improvements. Our work points to a bright future for high-dimensional MPLC-based technologies.