<sec>Yb-doped fiber amplifiers and their applications in radiation environments have become more and more attractive in recent years. However, the radiation effect will cause damage to the Yb-doped fibers, which can give negative effect on the output properties of Yb-doped fiber amplifiers. In this work, the influence of radiation effect on the transverse mode instability (TMI) of Yb-doped fiber amplifier is studied. TMI can couple the single light from the fundamental mode to high-order mode, thereby degenerating the beam quality of fiber amplifier. TMI is considered a key limitation of power up-scaling of fiber amplifiers.</sec><sec>In this work, the radiation effect on the TMI is studied theoretically, and a formula of TMI threshold is presented by taking the radiation-induced attenuation (RIA), the most important radiation effect for the TMI, into account. The formula is deduced by introducing the loss of signal light induced by RIA into the formerly reported TMI-threshold formula which can be obtained by the linear stability analysis of the numerical model studying the TMI. Then, the relationship between the TMI and radiation dose is also given with the help of Power-Law describing the relationship between the RIA and radiation dose.</sec><sec>With the formula, the variations of TMI threshold with the radiation dose and RIA are studied. It is found, as expected, that the TMI threshold decreases monotonically with the increase of RIA or radiation dose. Nevertheless, it is unexpectedly found that, to some extent, the gain coefficient of fiber amplifiers will also affect the radiation effect on TMI threshold. The results reveal that the increase of gain coefficient will lower the sensitivity of TMI threshold to the radiation dose. However, it is also implied that the gain coefficient cannot be too large because it can also make the TMI threshold lowered. Therefore, in order to maintain a high TMI threshold in a radiation environment, sufficient radiation resistance of Yb-doped fiber is essential.</sec><sec>Because the RIA can affect not only the TMI threshold but also the output power or efficiency of Yb-doped fiber amplifier, the comparison between two effects of RIA is also discussed. It is found that the threshold of TMI is more sensitive to the radiation than to the output power or efficiency (see the figure attached below), which means that the TMI can exist in the irradiated Yb-doped fiber amplifier, although the output power is reduced because of RIA. This result can be verified by the experimental observation reported formerly. As a result, TMI can become a key limitation to the output power of Yb-doped fiber amplifier in radiation environments. The relevant results can provide significant guidance for the applications of Yb-doped fiber amplifiers in radiation environments.</sec>