Shale oil resources, noted for their broad distribution and significant reserves, are increasingly recognized as vital supplements to traditional oil resources. In response to the high fracturing costs and swift decline in productivity associated with shale oil horizontal wells, this research introduces a novel approach utilizing CO2 for enhanced shale oil recovery in radial boreholes. A compositional numerical simulation method is built accounted for component diffusion, adsorption, and non-Darcy flow, to explore the viability of this technique. The study examines how different factors—such as initial reservoir pressure, permeability, numbers of radial boreholes, and their branching patterns—influence oil production and CO2 storage. Our principal conclusions indicate that with a constant CO2 injection rate, lower initial reservoir pressures predominantly lead to immiscible oil displacement, hastening the occurrence of CO2 gas channeling. Therefore, maintaining higher initial or injection pressures is critical for effective miscible displacement in CO2-enhanced recovery using radial boreholes. Notably, the adsorption of CO2 in shale oil results in the displacement of lighter hydrocarbons, an effect amplified by competitive adsorption. While CO2 diffusion tends to prompt earlier gas channeling, its migration towards areas of lower concentration within the reservoir reduces the extent of channeling CO2. Nonetheless, when reservoir permeability falls below 0.01 mD, the yield from CO2-enhanced recovery using radial boreholes is markedly low. Hence, selecting high-permeability “sweet spot” regions within shale oil reservoirs for the deployment of this method is advisable. To boost oil production, utilizing longer and broader radial boreholes, increasing the number of boreholes, or setting the phase angle to 0° are effective strategies. Finally, by comparing the production of shale oil enhanced by CO2 with that of a dual horizontal well fracturing system enhanced by CO2, it was found that although the former’s oil production is only 50.6% of the latter, its cost is merely 11.1%, thereby proving its economic viability. These findings present a new perspective for the economically efficient extraction of shale oil, offering potential guidance for industrial practices.