This paper considers the control problem of spacecraft line-of-sight (LOS) relative motion with thrust saturation in the presence of unmodeled dynamics, external disturbance and unknown mass property. By using skew-symmetric property, reference trajectory generator and anti-windup technique, a novel passivity-based adaptive sliding mode control (SMC) scheme is proposed without prior knowledge of uncertainty/disturbance bound. Within the Lyapunov framework, the establishment of a real sliding mode (which induces the practical stability of closed-loop error system) is validated. The main contributions are that a new control gain adaptive algorithm is adopted to attenuate the overestimation of switching gain and a differentiable projection-based parameter adaptive algorithm is proposed to force the mass approximator to remain in a desired domain, then the adaptive control law is modified by the reference trajectory generator and anti-windup technique to compensate for the effect of thrust saturation. Finally, simulations are conducted to show the fine performance of proposed control scheme.