Time-accurate three-dimensional Reynolds-averaged Navier-Stokes simulations have been performed to explore the flow unsteadiness of the subscale J-2S thrust optimized contour rocket nozzle in over-expansion at prescribed air supply conditions in the chamber. The results show the development of periodic low-frequency, high-amplitude, self-sustained oscillations at the simulated nozzle pressure ratios (NPRs). The cap-shock pattern is responsible for the two-coaxial jet topology that appears. The complex three-dimensional evolution of the flowfield is characterized by a periodical jet flapping, axial oscillations, and azimuthal fluctuations with or without swirl, depending on the NPR. According to the simulations, jet flapping quasi-restricted to a preferred plane may occur during the firings, being supported by the reported thrust and side-load measurements. The computed dominant frequencies corresponding to the longitudinal and transverse oscillation modes show an excellent agreement with the experimental counterpart. The analysis of the forces spectra and flowfields has identified that synchronization between side-load (transverse) and thrust (longitudinal) oscillations develops in consonance with the observations. Besides, their strong coupling is favoured by the response of the shocks to the pulsating pressure near the large recirculation bubble that arises behind the cap-shock. A discussion on the self-oscillating, high-coherent flow phenomenon is addressed. The present three-dimensional study has captured for the first time the phenomenon of cyclic transition between reattached flow separation mode and non-reattached mode, where the alternating bursting and formation of a separation bubble is accompanied by the large amplitude axial displacement of the shock pattern. Furthermore, the role of the separation bubble dynamics on triggering high side-load peaks at fixed NPRs is numerically confirmed.