Due to the influence of terrigenous debris, the internal pore structure of continental shale is highly heterogeneous, and the controlling factors are complex. This paper studies the structure and controlling factors of shale reservoirs in the first member of the Qingshankou Formation in the Southern Songliao Basin using core data and various analytical test data. The results show that the original deposition and subsequent diagenesis comprehensively determine the shale reservoirs’ pore structure characteristics and evolution law. According to the severity of terrigenous debris, the shale reservoirs in the study area are divided into four categories and six subcategories of lithofacies. By comparing the characteristics of different shale lithofacies reservoirs, the results show that the lithofacies with a high brittle mineral content have more substantial anti-compaction effects, more primary pores to promote retention and a relatively high proportion of mesopores/macropores. Controlling the organic matter content when forming high-quality reservoirs leads to two possibilities. An excessive organic matter content will fill pores and reduce the pore pressure resistance. A moderate organic matter content will make the inorganic diagenesis and organic hydrocarbon generation processes interact, and the development of organic matter mainly affects the development of dissolution pores. The comprehensive results show that A3 (silty laminated felsic shale) reservoirs underwent the pore evolution process of “two drops and two rises” of compaction, cementation and pore reduction, dissolution and pore increase, and organic matter cracking and pore increase, and they are the most favourable lithofacies of the shale reservoirs in the study area.