With the increasing demands for high-accuracy autonomous navigation, the integrated strapdown inertial navigation system and refractive celestial navigation system (SINS/RCNS) has received close attention. Aided by stellar refraction information, it can correct position and velocity errors. However, atmospheric density error is a critical element that restricts the accuracy of SINS/RCNS. To solve this problem, this paper proposes a novel tightly-coupled method. In this scheme, the atmospheric density error is estimated and compensated to accommodate for the unknown model perturbation. Additionally, a tight integration model based on star pixel coordinates is derived to handle the time-varying measurement noise. The proposed method is evaluated by representative suborbital flight vehicle navigation simulations, and the results indicate that the introduction of atmospheric density error compensation can improve the navigation accuracy without adding extra sensors and enhance the performance.