In order to reduce the calculation time and improve the performance of X-ray pulsar-based navigation (XNAV), we choose the bispectral feature points by the glowworm swarm optimization (GSO) algorithm, and to estimate the time delay for the pulsar integrated pulse profile. The proposed method is used to extract the bispectral feature points of the standard pulsar integrated pulse profile by the GSO algorithm on the ground control center, and store these feature points into the spacecraft's database. Then to estimate the time delay, it is only necessary to calculate the self-bispectrum and the cross-bispectrum of the standard pulsar integrated pulse profile and the observed one at the extracted bispectral feature points during flying. The results showed that the proposed method not only has the advantages of the traditional bispectrum algorithm that it can suppress the Gauss noise completely, but also reduced the calculation time greatly.