The goal of this work is to numerically investigate the mechanism of flow separation in a thrust-optimized parabolic nozzle during high-altitude testing. Both startup and shutdown processes of the Korea Space Launch Vehicle-II (KSLV-II) third-stage rocket engine are examined by axisymmetric computations. In particular, unconventional transitional process between the two representative separation patterns, free-shock separation (FSS) and restricted-shock separation (RSS) during high-altitude engine testing, is explored. It is observed that the attachment of nozzle flow to the diffuser wall leads to the upstream jump of the shock structure during the FSS-to-RSS transition for the engine startup process. For the engine shutdown process, the straight transition from full-flow to RSS is induced by the large trapped vortex which pushes nozzle flow towards the diffuser wall, and the pressure waves caused by the breakdown of vacuum region leads to the RSS-to-FSS transition. Furthermore, the entrained flow effect on flow separation is investigated. During the engine startup process, the entrained flow stabilizes the flow around the diffuser inlet, thereby inducing a smooth RSS-to-FSS transition. While the entrained flow delays the transition from full-flow to RSS during the engine shutdown process, but this delay eventually promotes the RSS-to-FSS transition. Consequently, the entrained flow significantly reduces the time in which flow separation resides inside the nozzle.