Background: Pulmonary hypertension (PH) is known to alter the biventricular shape and temporal phases of the cardiac cycle. The presence of interventricular septal (IVS) displacement has been associated with the severity of PH. There has been limited cardiac magnetic resonance (CMR) data regarding the temporal parameters of the cardiac cycle in PH. This study aimed to quantify the temporal changes in the cardiac cycle derived from CMR in PH patients with and without IVS displacement and sought to understand the mechanism of cardiac dysfunction in the cardiac cycle. Methods: Patients with PH who had CMR and right heart catheterization (RHC) examinations were included retrospectively. Patients were divided into an IVS non-displacement (IVSND) group and an IVS displacement (IVSD) group according to IVS morphology, as observed on short-axis cine CMR images. Additionally, age-matched healthy volunteers were included as the health control (HC). Temporal parameters, IVS displacement, ventricular volume and functional parameters were obtained by CMR, and pulmonary hemodynamics were obtained by RHC. The risk stratification of the PH patients was also graded according to the guidelines. Results: A total of 70 subjects were included, consisting of 33 IVSD patients, 15 IVSND patients, and 22 HC patients. In the IVSND group, only the right ventricle ejection fraction (RVEF) was decreased in the ventricular function, and no temporal change in the cardiac cycle was found. A prolonged isovolumetric relaxation time (IRT) and shortened filling time (FT) in both ventricles, along with biventricular dysfunction, were detected in the IVSD group (p < 0.001). The IRT of the right ventricle (IRTRV) and FT of the right ventricle (FTRV) in the PH patients were associated with pulmonary vascular resistance, right cardiac index, and IVS curvature, and the IRTRV was also associated with the RVEF in a multivariate regression analysis. A total of 90% of the PH patients in the IVSD group were stratified into intermediate- and high-risk categories, and they showed a prolonged IRTRV and a shortened FTRV. The IRTRV was also the predictor of the major cardiovascular events. Conclusions: The temporal changes in the cardiac cycle were related to IVS displacement and mainly impacted the diastolic period of the two ventricles in the PH patients. The IRT and FT changes may provide useful pathophysiological information on the progression of PH.