Phase-generated carrier (PGC) demodulation technology has been widely used in fiber-optic interferometric sensors system in recent years. Nonlinear errors caused by interference noise and parasitic amplitude modulation (AM) are key factors that affect the reliability of traditional PGC demodulation system based on arctangent (PGC-Arctan) algorithm. In order to enhance the reliable performance of PGC demodulation system under the conditions of low signal-to-noise ratio (SNR) and high parasitic AM interference, an ellipse fitting algorithm based on the Gauss–Newton iteration (GNI) is proposed, which is called the PGC-Arctan-GNI demodulation algorithm. The proposed algorithm uses the Euclidean distance to accurately estimate the geometric parameters of ellipse, which can correct the nonlinear distortion of the demodulated signal. In addition, the PGC demodulation system designed based on GNI algorithm has the advantages of strong antinoise ability, good antiparasitic AM ability, and widely dynamic range of the input signal amplitude. Experimental results show that, compared with the direct demodulation algorithm and the ellipse fitting algorithm based on the least squares (LS) method, the SINAD value of the proposed algorithm is always greater than 30 dB and higher than the other two algorithms under the condition of low SNR. Under the condition that the parasitic AM index <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$m$</tex-math></inline-formula> changes from 0 to 0.5, the signal-to-noise and distortion ratio (SINAD) curve of the proposed algorithm is more stable than the curves of the other two algorithms, and the fluctuation range of the SINAD value does not exceed 3 dB. This algorithm also has a wider input signal amplitude range than the LS algorithm. Especially, when the amplitude of the measured signal is less than <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\pi$</tex-math></inline-formula> /2, the relative amplitude error of the GNI algorithm is less than 2% and significantly smaller than the LS algorithm. Finally, the demodulation results using real measured data in the actual system show that the frequency and amplitude of the demodulated signal are same as the original signal. At the same time, the SNIAD value of this system reaches 67.40 dB, the spurious-free dynamic range value reaches 68.82 dB, and the total harmonic distortion value reaches -68.40 dB@1 KHz. In summary, the proposed PGC-Arctan-GNI algorithm can eliminate system nonlinear errors caused by the parasitic AM, harmonics, and noise interference by estimating the ellipse parameters, thereby effectively enhancing the stability and reliability of the PGC demodulation system.