Active cooling technology serves as one of the promising and effective methods for addressing the thermal protection strategy for hypersonic aircraft. While the coke generated through pyrolysis, dehydrogenation and coking reactions of fuel will reduce the heat transfer efficiency, even clog the heat exchange channel. In this work, three-component RP-3 aviation fuel was adopted as model compounds to investigate the cracking coking reaction mechanism and explore the method of inhibiting coking. It is found that the three-component RP-3 molecules generate an intermediate system dominated by C2 products through bond breaking, reforming and bimolecular reactions (mainly hydrogen abstraction reactions) under supercritical state, in which the ·H radicals, as represented radicals, play a crucial role in the opening of benzene rings. The coking process is analyzed in details furtherly, and detailed soot formation mechanisms are revealed. Additionally, a suitable “core-shell structure” model is proposed to describe the coke formation process. Meanwhile, it is found that the addition of a small number of addition agents can effectively inhibit the formation of coke and thus prolong the residence time of fuel molecules in the heat transfer channel for the coking process mainly consisted of addition reaction. The study attempts to provide a theoretical basis for the high-temperature pyrolytic coking mechanism of endothermic hydrocarbon fuel, as well as the significance to understanding the inhibition of coke formation.