To study the performance of the PbS quantum dots (QDs)-doped silica fiber amplifier (PQDFA), here, a model of PQDFA with ultra-broadband gain and low noise is theoretically proposed. Based on this model, the impact of fiber structure, pump/signal power, and PbS QDs properties on the noise figure (NF) of PQDFA is investigated. The simulation results show that pump/signal light power changes the NF of PQDFA by affecting the population inversion level. Besides, the Stokes-shift and doping concentration proportion of QDs also affect the performance of fiber amplifiers. In addition, the gain and NF spectrum of single- and multi-sized PQDFA are simulated and compared. The simulation results show that the working band of PQDFA is determined by the QDs’ emission spectra, and wavelength tunability can be achieved by changing the QDs’ distribution in fiber. For the multi-sized PQDFA proposed in this paper, under −10 dBm input signal power, the 3 dB bandwidth, highest gain, and lowest NF in 1300–1500 nm band are 107 nm, 29 dB, and 3.84 dB, respectively.