Membrane-bound protein complexes involving pore forming toxins (PFTs) released by virulent bacteria are known to form transmembrane pores leading to host cell lysis. Developing alternative strategies against PFT mediated bacterial virulence factors requires an understanding of the cellular membrane response. However, membrane disruption and related lipid reorganization events during attack by PFTs remain largely unexplored. We report counterintuitive and nonmonotonic variations in lipid diffusion, measured using confocal fluorescence correlation spectroscopy, due to interplay of lipid ejection and crowding by membrane-bound oligomers of a prototypical cholesterol-dependent cytolysin, listeriolysin O (LLO). The observed dynamical crossover is correlated with concentration dependent transitions of LLO oligomeric state populations from rings to arc-like pore complexes, predicted using a proposed two-state free area-based diffusion model. At low PFT concentrations, a hitherto unexplored regime of increased lipid diffusivity is attributed to lipid ejection events because of a preponderance of ring-like pore states. At higher protein concentrations in which membrane-inserted arc-like pores dominate, lipid ejection is less efficient and the ensuing crowding results in a lowering of lipid diffusion. These variations in lipid dynamics are corroborated by macroscopic rheological response measurements of PFT bound vesicles. Our study correlates PFT oligomeric state transitions, membrane remodeling, and mechanical property variations, providing unique insights into the pore forming mechanisms of cholesterol-dependent cytolysins.