The well known soft theorems state the specific factorizations of tree level gravitational (GR) amplitudes at leading, sub-leading and sub-sub-leading orders, with universal soft factors. For Yang-Mills (YM) amplitudes, similar factorizations and universal soft factors are found at leading and sub-leading orders. Then it is natural to ask if the similar factorizations and soft factors exist at higher orders. In this note, by using transformation operators proposed by Cheung, Shen and Wen, we reconstruct the known soft factors of YM and GR amplitudes, and prove the nonexistence of higher order soft factor of YM or GR amplitude which satisfies the universality.