This paper describes the theory of Lie group operators in a form suitable for the applied dynamics community. In particular, it is adapted to analyzing the dynamic behavior of nonlinear systems in the presence of different resonance conditions. A key ingredient of the theory is the Hausdorff formula, which is found to be implicitly reproduced in most averaging techniques during the transformation process of the equations of motion. The method is applied to examine the nonlinear modal interaction in a coupled oscillator representing a double pendulum. The system equations of motion are reduced to their simplest (normal) form using operations with the linear differential operators according to Hausdorff's formula. Based on the normal form equations, different types of resonance regimes are considered. It is shown that the energy of the parametrically excited first mode can be regularly (or nonregularly) shared with the other mode due to the internal resonance condition. If the second mode is parametrically excited, its energy is localized and is not transferred to the first mode, even in the presence of internal resonance.