The development of carbon capture, utilization, and storage technologies has notably advanced CO2-enhanced oil recovery (EOR) in shale oil reservoirs, which are characterized by abundant nanopores. These nanopores induce unique phase behaviors in hydrocarbons, challenging traditional phase equilibrium calculation methods. This paper presents a novel three-phase thermodynamic model (vapor–liquid-adsorbed three-phase equilibrium calculation) that addresses these challenges by considering the nanopore capillary pressure, critical parameter transitions, and material exchange between the adsorbed and bulk phases. Grounded in the multicomponent Langmuir–Freundlich adsorption equation and the Peng Robinson equation of state, this model is integrated into the MATLAB Reservoir Simulation Toolbox using an embedded discrete fracture model framework, enabling detailed study of CO2 and hydrocarbon phase behaviors within shale oil nanopores. The results reveal that there are significant nano-constrained effects on multicomponent fluid phase behavior, particularly in pores smaller than 20 nm, leading to notable changes in bubble and dew point pressures, as well as critical condensation pressures and temperatures. CO2 injection further complicates the system, enhancing interactions and expanding the coexistence region of the liquid and gas phases on the pressure–temperature diagram, especially across varying pore sizes. Optimization research on CO2 huff and puff technical parameters for shale oil reservoirs suggests the following optimal settings: a CO2 injection rate of 100 t/day, a shut-in time of 30 days, and six huff and puff cycles. The results of this study offer critical insights into CO2-EOR mechanisms in shale oil reservoirs and emphasize the importance of nanopore properties in EOR.