Rare-earth (RE) elements doped III-nitride semiconductors have garnered attention for their potential in advanced high-frequency and high-power electronic applications. We report on the molecular beam epitaxy of quaternary alloy ScAlInN, which is an encouraging strategy to improve the heterointerface quality when grown at relatively low temperatures. Monocrystalline wurtzite phase and uniform domain structures are achieved in ScAlInN/GaN heterostructures, featuring atomically sharp interface. ScAlInN (the Sc content in the ScAlN fraction is 14%) films with lower In contents (less than 6%) are nearly lattice matched to GaN, exhibiting negligible in-plane strain, which are excellent barrier layer candidates for GaN high electron mobility transistors (HEMTs). Using a 15-nm-thick Sc0.13Al0.83In0.04N as a barrier layer in GaN HEMT, a two-dimensional electron gas density of 4.00 × 1013 cm−2 and a Hall mobility of 928 cm2/V s, with a corresponding sheet resistance of 169 Ω/□, have been achieved. This work underscores the potential of alloy engineering to adjust lattice parameters, bandgap, polarization, interfaces, and strain in emerging RE-III-nitrides, paving the way for their use in next-generation optoelectronic, electronic, acoustic, and ferroelectric applications.