ABSTRACT It is clarified that the parabolic/hyperbolic restricted three-body problem (PRTBP/HRTBP) can be adopted to provide a simple description of the dynamics of fly-by asteroids or the close encounters between different galaxies. For these reasons, PRTBP and HRTBP have been investigated for long intervals of time. However, they are quite different from the circular restricted three-body problem due to the time-dependent and non-periodic dynamics. The Lagrangian coherent structures (LCSs), as a useful tool to analyse the time-dependent dynamical system, could be applied to explain some mechanics of the PRTBP and HRTBP. In this paper, we verify the invariant manifolds on the boundary manifolds of PRTBP by analysing the LCSs in proper Poincaré sections, which shows that it works in such a non-periodic problem. One of the contributions is to investigate the LCSs in the complete phase space of PRTBP, and then some natural escape and capture trajectories from or to the two main bodies can be obtained in this way. Another contribution is to establish and study the dynamics of HRTBP and its boundary. The LCSs can be introduced into this system, reasonably, to work as the analogues of the invariant manifolds, and the similar natural escape and capture trajectories corresponding to the two main bodies can also be obtained in the complete phase space of HRTBP. As a typical technique applied to fluid, flows to identify transport barriers in the time-dependent system, the LCSs provide an effective way to determine the time-dependent analogues of invariant manifolds for the PRTBP/HRTBP.