Differing from a two-level voltage source converter (TL-VSC), the internal dynamics of a modular multilevel converter (MMC) are much more complicated. Since the fluctuation of submodule capacitor voltage is unavoidable when transmitting power, the MMCs output voltages cannot track the references perfectly. The interaction between the time-varying capacitor dynamics and controllers may lead to self-instability of the MMCs. In order to address this self-instability issue, this paper first establishes the small-signal closed loop transfer function matrices (TFMs) of the external and internal control loop by the harmonic linearization method. Further, a novel design principle of MMC controllers is proposed. Then, the critical impact factors on the stability of circulating current controller and AC voltage controller are investigated based on the generalized Nyquist criterion(GNC). The proposed models and analysis methods are validated by time-domain simulations in PSCAD/EMTDC.