Abstract It is well-known that nonlinear dry friction damping has the potential to bound the otherwise unboundedly growing vibrations of self-excited structures. An important technical example are the flutter-induced friction-damped limit cycle oscillations of turbomachinery blade rows. Due to symmetries, natural frequencies are inevitably closely spaced, and they can generally be multiples of each other. Not much is known on the nonlinear dynamics of self-excited friction-damped systems in the presence of such internal resonances. In this work, we analyze this situation numerically by regarding a two degrees-of-freedom system. We demonstrate that in the case of closely-spaced natural frequencies, the self-excitation of the lower-frequency mode gives rise to non-periodic oscillations, and the occurrence of unbounded behavior well before reaching the maximum friction damping value. If the system is close to a 1:3 internal resonance, limit cycles associated with much higher frictional damping appear, however, most of these are unstable. If more than one mode is subjected to self-excitation, the maximum resistance against self-excitation is at least given by the damping capacity of the most weakly friction-damped mode. These results are of high technical relevance, as the prevailing practice is to analyze only periodic limit states and argue the stability solely by the slope of the damping-amplitude curve. Our results demonstrate that this practice leads to considerable mis- and overestimation of the resistance against self-excitation, and a more rigorous stability analysis is required.