Grassland is an important component of terrestrial ecosystems and plays a crucial role in the global carbon cycle. PhytOC (phytolith-occluded organic carbon) is an extremely important long-term and stable carbon pool in terrestrial ecosystems. Southwest China karst soil exhibits obvious characteristics of alkalinity, high silicon content, and rich calcium, which can significantly influence the characteristics and mechanisms of PhytOC sequestration in vegetation. To elucidate the sequestration characteristics and mechanisms of PhytOC in the karst grasslands, three typical karst grasslands of tropical shrub tussock (TST), warm-temperate shrub tussock (WST), and mountain meadow (MM) from Guizhou province of southwest China were studied. The following results and conclusions were obtained that: 1) the range of PhytOC content of aboveground plant parts, underground roots, and soil in the karst grasslands was 4.03–16.54 g·kg−1, 10.67–33.92 g·kg−1, and 0.63–1.89 g·kg−1, respectively. The underground roots are an important site for phytolith carbon sequestration in grassland ecosystems, and the PhytOC content of underground roots may be higher than that of the aboveground parts. 2) The PhytOC sequestration rate of vegetation was 7.34–15.93 kg·ha−1·yr−1, and the annual sequestration amount of PhytOC of the whole grasslands in southwest China could reach 0.48 × 103–1.48 × 103 t CO2. Compared to grasslands in non-karst regions of China, karst grasslands in southwest China have a higher sequestration rate of PhytOC in vegetation and a greater capacity for phytolith carbon sequestration. 3) Soil available silicon, pH, and stoichiometric characteristics of C, N and P nutrients significantly affected the phytolith carbon sequestration of vegetation and the soil accumulation of PhytOC in the karst grasslands. The research results are of great significance for estimating the phytolith carbon sequestration capacity of grassland ecosystems and for grassland construction and management based on enhancing carbon sequestration.