Phytoplankton plays a key role in the stabilization of aquatic ecosystems. However, there is a lack of research on the structure of phytoplankton communities and their influencing factors in shallow lakes in Southeast Hubei. In this study, four typical lakes were selected in the summer of 2019, and the phytoplankton community structure and its drivers were investigated for each lake. The results showed that the number of phytoplankton species, dominant species, biomass, and abundance varied among lakes. A comprehensive trophic level index (TLI) assessment showed that Lake Xiandao was oligotrophic, Lake Baoan was light eutrophic, and Lake Wang and Lake Ci were moderate eutrophic. The Shannon–Wiener, Margalef, and Pielou indices showed that the phytoplankton communities of Lake Wang and Lake Ci are less stable, and the water bodies are more severely contaminated. In contrast, the communities of the Lake Baoan and Lake Xiandao are more stable and have superior water quality. A redundancy analysis (RDA) indicated that the primary variables influencing phytoplankton community structures were pH and water temperature (WT) (Lake Ci); permanganate index (CODMn) and pH (Lake Xiandao); WT and total nitrogen (TN) (Lake Wang); and WT and total phosphorus (TP) (Lake Baoan).