Abstract The filled broken rock mass in a geothermal reservoir under high-pressure work fluid and overburden stress is of great significance to the efficient operation of the geothermal system. The pore structure and hydraulic properties of the broken rock mass were quantitatively investigated by the nuclear magnetic resonance (NMR) technology and the non-Darcy models. The broken rock mass at the lower compressive stress levels shows stronger compressibility, and the broken gangue with more small particles shows stronger stress sensitivity. The seepage pores in broken rock can be characterized by the fractal theory, and the fractal dimension ranges from 2.749 to 2.861, which is positively correlated with compressive stress. The increase in Dp can be attributed to the shrinkage of macropores affected by the increasing compressive stress. The flow in broken rock mass under high pressure gradient shows significant nonlinearity verified by the Forchheimer equation and the Barree-Conway equation. We proposed a logistic regression model to characterize the effects of compressive stress and GSD on the permeability and the characteristic parameter of nonlinear flow in broken rock mass, which provides a potential method for evaluating the performance of the geothermal system based on the pore structure of the reservoir rock mass.