We investigate self-homodyne coherent transmission employing space division multiplexing (SDM), where a continuous wave laser source is divided to produce spatially multiplexed signals and a co-propagating pilot-tone (PT), which serves as a local oscillator for self-homodyne detection at the receiver side. The beat interference caused by the spatial coupling between PT and the signals during propagation is investigated through derivations and numerical modelling. Both conventional multiple-input–multiple-output (MIMO) and unreplicated parallel interference canceler (UPIC)-assisted MIMO suffer from the effects of beat interference and additional mode dependent loss introduced by PT-signal crosstalk, which limits the performance of SDM-based self-homodyne coherent systems. We propose two novel MIMO processing schemes employing digital conjugate input branches to handle the beat interference and demonstrate system performance improvement in both simulations and experiments. For a 240-Gbps self-homodyne coherent system transmitting mode-multiplexed QPSK signals over a 3-mode multimode fiber, after adapting the two enhanced MIMO schemes, simulations show a up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\sim$</tex-math></inline-formula> 4 dB improvement in PT-signal crosstalk level tolerance, and experiments show undistorted constellations as well as BER improvements from <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\sim 10^{-2}$</tex-math></inline-formula> to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\sim 10^{-5}$</tex-math></inline-formula> with PT-signal delay alignment after the mode demultiplexer.