We have theoretically investigated the polarization-dependent ultrashort pulse amplification in erbium-doped fluoride fibers. The numerical model is based on the coupled generalized nonlinear Schrödinger equations and the rate equations. It is found that the Raman effect can be weakened by converting the seed polarization from linear to circular. Compared with the linearly polarized seed pulses, amplification of the circularly polarized seed pulses can generate Raman solitons with a larger pulse energy of the same central wavelength, although more pump power is needed. Furthermore, the amplification dynamics when no pulse splitting occurs have also been investigated. We demonstrate that amplification of the circularly polarized seed pulses allows the main pulse to accumulate more energy while maintaining a single-pulse profile. It is predicted that sub-two-cycle laser pulses at 2.8 μm with megawatt peak power and hundreds of nanojoules pulse energy can be directly generated from the erbium-doped fluoride fiber amplifiers. The investigations conducted in this paper can provide guidelines for the design of Raman-soliton-based tunable fiber laser systems and high-power few-cycle fiber laser systems.