Molecular beam epitaxy (MBE) is demonstrated as an excellent growth technique for growing a low-defect GaN channel layer, which is crucial for controlling vertical leakage current and improving breakdown voltage (BV) in GaN-based high-electron mobility transistors (HEMTs). The 3D islands to 2D growth mode transition approach was induced by modulating substrate growth temperature (Tsub), displaying an overall improvement in film quality. A comprehensive investigation was conducted into the effects of Tsub on surface morphologies, crystal quality, and the optical and electrical properties of GaN films. Optimal results were achieved with a strain-relaxed GaN film grown at 690 °C, exhibiting significantly improved surface characteristics (root-mean-square roughness, Rq = 0.3 nm) and impressively reduced edge dislocations. However, the film with the smoothest surface roughness, attributed to the effect of the Ga-rich condition, possessed a high surface pit density, negatively affecting optical and electrical properties. A reduction in defect-related yellow emission further confirmed the enhanced crystalline quality of MBE GaN films. The optimized GaN film demonstrated outstanding electrical properties with a BV of ~1450 V, surpassing that of MOCVD GaN (~1180 V). This research significantly contributes to the advancement of MBE GaN-based high electron mobility transistor (HEMT) applications by ensuring outstanding reliability.