The effects of barrier layer thickness, Al component of barrier layer, and passivation layer thickness of high-resistance Si (111)-based AlGaN/GaN heterojunction epitaxy on the knee-point voltage (Vknee), saturation current density (Id-sat), and cut-off frequency (ft) of its high electron mobility transistor (HEMT) are simulated and analyzed. A novel optimization factor OPTIM is proposed by considering the various performance parameters of the device to reduce the Vknee and improve the Id-sat on the premise of ensuring the ft. Based on this factor, the optimized AlGaN/GaN epitaxial structure was designed with a barrier layer thickness of 20 nm, an Al component in the barrier layer of 25%, and a SiN passivation layer of 6 nm. By simulation, when the gate voltage Vg is 0 V, the designed device with a gate length of 0.15 μm, gate-source spacing of 0.5 μm, and gate-drain spacing of 1 μm presents a high Id-sat of 750 mA/mm and a low Vknee of 2.0 V and presents ft and maximum frequency (fmax) as high as 110 GHz and 220 GHz, respectively. The designed device was fabricated and tested to verify the simulation results. We demonstrated the optimization factor OPTIM can provide an effective design method for follow-up high-frequency and low-voltage applications of GaN devices.