Investigating social vulnerability is a crucial aspect of attaining an in-depth understanding of disasters and disaster risk. This study developed a measurement framework for social vulnerability to geological disasters, obtained a comprehensive social vulnerability index, and compared the spatial distribution patterns of fatalities, social vulnerability, and geological disaster-prone zones. These efforts assisted in determining the strength and character of the relationship between geological disaster fatality risk and social vulnerability at the county level. The findings show that: (1) the social vulnerability to geological disasters index varies significantly between counties. High social vulnerability is primarily concentrated in Bijie City, Guizhou province, which shows concentrated clustering. Conversely, low social vulnerability is mainly concentrated in Guiyang City: (2) the spatial distribution patterns reveal the limitations of disaster exposure in explaining fatality risk and demonstrate the ability of social vulnerability to amplify disaster risk; (3) the zero-inflated negative binomial regression model fit the data significantly better than the negative binomial regression model, suggesting that for every one unit increase in the social vulnerability index, there is a 6.33% increase in the risk of fatalities, holding all other variables constant. This study establishes a specific framework for measuring social vulnerability to geological disasters and provides new evidence of how social vulnerability influences disaster risk intensity. The county-level spatial measurement results can further assist disaster management personnel in formulating more localized mitigation strategies. The findings will further support the identification of core remediation sites and the implementation of management and relocation (avoidance) projects during Guizhou Province’s 14th Five-Year Plan for geological disaster prevention and control.