Organic matter (OM)-hosted pores are the most prevalent pores in organic-rich shale, in which shale gas is generated and stored. Ascertaining the pore structure of OM and its contribution to the shale pore network provides guidance for understanding deeply the complex pore network, as well as the shale gas flow and storage mechanisms. In this study, the pore structure and heterogeneity of Wufeng–Longmaxi shales and the corresponding isolated OM samples were studied comparatively by scanning electron microscopy (SEM) observations and gas (CO2 and N2) physisorption quantification. The Wufeng–Longmaxi shales are rich in OM, with total organic carbon (TOC) contents of 1.48–3.59 wt.%. The SEM-observed microscopic pores were primarily OM-hosted pores and intra-particle pores within clays. Compared to the pore structure per gram of bulk shale, that of isolated OM showed a significantly larger micropore (d < 2 nm) volume and specific surface area (SSA), meso-pore (d = 2–50 nm) and fine macro-pore (50 < d ≤ 80 nm) volume, and Brunauer–Emmett–Teller SSA. Moreover, the pore structure heterogeneity of the bulk shale was much stronger than that of the isolated OM, revealing that pores associated with minerals could enhance the heterogeneity of shale to some extent. The pore size distributions of the bulk shale (1 g) and the corresponding weight-normalized isolated OM (1 g × TOC) were compared. It was revealed that shale pores with diameters smaller than 20 nm were provided by OM together with minerals, while almost all pores with diameters of 20–80 nm occurred within the OM. Overall, the OM and OM-hosted pores provide a vital contribution to the entire pore network of the Wufeng–Longmaxi shale.