In recent years, transverse mode instability (TMI) has been widely observed in fiber laser amplifier systems. The transverse mode instability phenomenon in fiber laser oscillators is less studied. Here, we focus on the dynamical output properties, i.e., its temporal signal and modal characteristics in a 30-μm-core-diameter ytterbium (Yb)-doped fiber laser oscillator. The TMI occurs at a pumping power around 310 W. Different from amplifiers, the basic oscillation frequency is quite low, at around 100 Hz, changing with time and pump power. When the fiber laser oscillator operates beyond TMI threshold at 357 W or 377 W for a while, the temporal fluctuation slowly disappears together with a decreased oscillation frequency, and appears again later. Based on the mode decomposition technique, we find that during the period of fluctuation disappearance at 357 W, the power output stays low and the output beam is still a mixture of fundamental mode and higher-order modes. The fundamental mode content is calculated to be averagely higher when temporal fluctuation disappears, increasing from ∼57% to ∼63%. Our results indicate complex interaction between the fiber laser oscillation and the TMI effect, and calls for more attention into understanding TMI in fiber laser oscillators.