To accurately investigate the evolution characteristics and generation mechanism of retained oil, the study analyzed organic-rich lacustrine shale samples from the Paleogene Kongdian Formation in Cangdong Sag, Bohai Bay Basin. This analysis involves Rock-Eval pyrolysis, pyrolysis simulation experiments, Gas Chromatograph Mass Spectrometer (GC–MS), and reactive molecular dynamics simulations (ReaxFF). The results revealed the retained oil primarily consisted of n-alkanes with carbon numbers ranging from C14 to C36. The generation of retained oil occurred through three stages. A slow growth stage of production rate was observed before reaching the peak of oil production in Stage I. Stage II involved a rapid increase in oil retention, with C12–C17 and C24–C32 serving as the primary components, increasing continuously during the pyrolysis process. The generation process involved the cleavage of weak bonds, including bridging bonds (hydroxyl, oxy, peroxy, imino, amino, and nitro), ether bonds, and acid amides in the first stage (Ro =0.50%–0.75%). The carbon chains in aromatic ring structures with heteroatomic functional groups breaks in the second stage (Ro =0.75%–1.20%). In the third stage (Ro =1.20%–2.50%), the ring structures underwent ring-opening reactions to synthesize iso-short-chain olefins and radicals, while further breakdown of aliphatic chains occurred. By coupling pyrolysis simulation experiments and molecular simulation technology, the evolution characteristics and bond breaking mechanism of retained oil in three stages were revealed, providing a reference for the formation and evolution mechanism of retained oil.