Mg-doped AlxGa1-xN (x = 0.23 and 0.35) alloys have been grown on GaN templates with high temperature AlN (HT-AlN) interlayer by metalorganic chemical vapor deposition (MOCVD). A combination of secondary ion mass spectrometry (SIMS) and transmission electron microscopy (TEM) indicates the formation of more inversion domains in the high Al mole fraction Mg-doped AlGaN alloys at Mg concentration ∼1020 cm−3. For Mg-doped Al0.23Ga0.77N epilayer, the analysis of cathodoluminescence (CL) spectra supports the existence of self-compensation effects due to the presence of intrinsic defects and Mg-related centers. The energy level of Mg is estimated to be around 193 meV from the temperature dependence of the resistivity measured by Hall effect experiments. And hole concentration and mobility are measured to be 1.2 × 1018 cm−3 and 0.56 cm2/V at room temperature, respectively. The reduction of acceptor activation energy and low hole mobility are attributed to inversion domains and self-compensation. Moreover, impurity band conduction is dominant in carrier transport up to a relatively higher temperature in high Al content Mg-doped AlGaN alloys.