Abstract A new cost-effective mobile fronthaul architecture has been proposed by designing promising analog radio over fiber (ARoF) transmission system compliant with the next generation ultra-dense wavelength division multiplexing passive optical network (UDWDM-PON). It is conceived for standalone 5G/5G+ mobile communication systems driven by broadband multi-input multi-output (MIMO) millimeter wave data signals using two high spectral efficient modulations. The feasibility of the proposed system was firstly proven in terms of spectral tracing supported by theoretical analysis that disclosed good agreement. Furthermore, the obtained simulation results regarding the system performance evaluation after 22 km bidirectional optical fiber links were revealed high reliable characteristics including error-free transmissions at low receiver sensitivities, and measured power penalties less than 1 dB. In addition, the recovered eye diagrams maintained wide openings with high-quality patterns.