The stretching and subsidence characteristics of passive continental margins have been investigated for decades but are still controversial. We focused our study on the deep water sub-basins of the Pearl River Mouth Basin in the northern South China Sea, the only well-preserved passive margin in the South China Sea. Inverse modeling and forward modeling based on a flexural cantilever model were employed. The stretching factors of the upper crust, lower crust, whole crust, and mantle lithosphere were mapped, and the anomalous post-rift subsidence was isolated from the McKenzie model. The Liwan Sub-basin showed highly depth-dependent stretching that significantly increased with depth. The stretching factor of the lower crust in the center of the Baiyun Sub-basin was higher than that of the mantle lithosphere. Two significant anomalous rapid subsidence events were revealed. The 23.03-19.8 Ma event was characterized by anomalous subsidence at the center, located in the southern Baiyun Sub-basin, with a maximum value in excess of 1200 m. Strong negative P-wave abnormal and magma addition may indicated an upwelling mantle under the Baiyun Sub-basin around 23 Ma. We suggest that the 23.6 Ma southward ridge jump in the South China Sea could have weakened or eliminated the mantle upwelling under the Baiyun Sub-basin and caused rapid subsidence at 23.03-19.8 Ma. The 14.3-11.9 Ma event was characterized by a subsidence zone orientated northeast along the Baiyun Sub-basin, with a maximum value in excess of 600 m. We propose that rapid depositional sediment during this period would have driven the lower crustal flow away from the Baiyun Sub-basin center, causing rapid vertical subsidence and leading to the stretching factor of the lower crust being higher than that of the mantle lithosphere.