In this work, a molecular-level kinetic model was built to simulate the vacuum residue (VR) coking process in a semi-batch laboratory-scale reaction kettle. A series of reaction rules for heavy oil coking were summarized and formulated based on the free radical reaction mechanism. Then, a large-scale molecular-level reaction network was automatically generated by applying the reaction rules on the vacuum residue molecules. In order to accurately describe the physical change of each molecule in the reactor, we coupled the molecular-level kinetic model with a vapor–liquid phase separation model. The vapor–liquid phase separation model adopted the Peng-Robinson equation of state to calculate vapor–liquid equilibrium. A separation efficiency coefficient was introduced to represent the mass transfer during the phase separation. We used six sets of experimental data under various reaction conditions to regress the model parameters. The tuned model showed that there was an excellent agreement between the calculated values and experimental data. Moreover, we investigated the effect of reaction temperature and reaction time on the product yields. After a comprehensive evaluation of the reaction temperature and reaction time, the optimal reaction condition for the vacuum residue coking was also obtained.