Summary On a global scale, shale oil/gas has become an important alternative energy source for conventional oil and gas. The potential advantages of supercritical CO2 (ScCO2) make it an ideal alternative to hydraulic fracturing, used for shale reservoir transformation and production increase while also promoting the geological storage of CO2, which is in-line with today’s carbon capture, utilization, and storage technology and helps to address the challenges of global climate change. To further study the fracture propagation and optimization of a complex fracture network (CFN) in ScCO2 fracturing under complex geological conditions using the cohesive module of ABAQUS to establish a fluid structure coupling model and completing indoor and field experimental verification, we introduce the global embedded cohesion zone model (CZM) combined with Python to generate two natural-fracture (NF) distribution models, conjugate and power law, to establish a dispersed mesh model. Based on this model, we studied the fracture propagation problem of ScCO2 fracturing under different engineering and geological conditions. The simulation results will be used as data-driven data to establish an optimization model of the random forest-particle swarm optimization algorithm (RF-PSO) and optimize the CFN. Research has shown that (1) ScCO2 is more inclined to pass through NFs and propagate in the rock matrix, and hydraulic fractures (HFs) combine better with NFs. Compared with hydraulic fracturing, ScCO2 fracturing has significant advantages (only the fracture width is lower than hydraulic fracturing, its initiation pressure and fracture length are much better than hydraulic fracturing, and there are more small fractures, making it easier to form a CFN). (2) During the process of fracture propagation, once dominant fractures form, the trend of the “Matthew effect” is inevitable. The process of fracture propagation is influenced by multiple factors, especially the distribution of NFs; the larger the reservoir filtration coefficient is, the more ScCO2 fracturing fluid that is lost, which is more unfavorable for fracturing construction. While maintaining the same amount of fracturing fluid injection, as the displacement increases, the fracture complexity increases, and the fracturing control range expands. Compared with other parameters, the effect of fracturing fluid temperature (FFT) on the expansion of ScCO2 fracturing fractures is not significant. (3) The established RF-PSO optimization model has an error of 2.89%, which can well adapt to CFN optimization problems under complex NF conditions and reduce uncertainty. We propose in this article a research method for fracture network optimization from fracture modeling, dynamic simulation, and optimization modeling. By combining numerical simulation and machine learning, the CFN optimization design of ScCO2 fracturing under CFN conditions is achieved, providing a research approach for the optimization of fracturing in fractured reservoirs.