Nanoscopic potential fluctuations of Si-doped AlGaN epitaxial layers with the AlN molar fraction varying from 0.42 to 0.95 and Si-doped Al0.61Ga0.39N epitaxial layers with Si concentrations of 3.0–37 × 1017 cm−3 were investigated by cathodoluminescence (CL) imaging combined with scanning electron microscopy. The spot CL linewidths of AlGaN epitaxial layers broadened as the AlN molar fraction was increased to 0.7, and then narrowed at higher AlN molar fractions. The experimental linewidths were compared with the theoretical prediction from the alloy broadening model. The trends displayed by our spot CL linewidths were consistent with calculated results at AlN molar fractions of less than about 0.60, but the spot CL linewidths were markedly broader than the calculated linewidths at higher AlN molar fractions. The dependence of the difference between the spot CL linewidth and calculated line broadening on AlN molar fraction was found to be similar to the dependence of reported S values, indicating that the vacancy clusters acted as the origin of additional line broadening at high AlN molar fractions. The spot CL linewidths of Al0.61Ga0.39N epitaxial layers with the same Al concentration and different Si concentrations were nearly constant in the entire Si concentration range tested. From the comparison of reported S values, the increase of VAl did not contribute to the linewidth broadening, unlike the case of the VAl clusters.