RBOE is a new type of DNAN-based high-energy melt-cast mixed explosive, whose safety under thermal stimulation is significantly affected by heating conditions and venting area of the warhead. Based on the thermal decomposition reaction characteristics and combustion characteristics of each component of RBOE explosive, the cook-off calculation models of RBOE warhead before and after ignition were established. In addition, closed and vented warheads were designed, as well as fast and slow cook-off test devices. The cook-off characteristics and thermal safety venting area of RBOE warhead were extensively studied. The results showed that the closed RBOE warhead underwent deflagration reaction under both slow and fast cook-off conditions. The calculation result of the shell wall temperature before slow cook-off ignition response of the warhead was 454.06K, with an error of +1.75% compared to the test result of 462.15K, and the temperature rise rate calculated was in good agreement with the test. The calculated ignition time of RBOE warhead under fast cook-off was 161s, with an error of + 8.8% compared to the test result of 148 s, which verified the accuracy of cook-off model of RBOE warhead before ignition. According to the cook-off calculation model of the warhead after ignition and cook-off test of the vented warhead, it was determined that the thermal safety venting area was 1124.61 mm2 for fast cook-off and 530.66 mm2 for slow cook-off, effectively preventing the reaction of warhead above combustion. Therefore, this study provides a scientific basis for the thermal safety design and evaluation of insensitive warheads.