The agriculture sector in the Savanna region of Togo is especially vulnerable to weather fluctuations, which have an impact on crop production levels. However, farmers’ decisions to implement adaptation strategies are directly related to their perceptions of climate change risk. The current study employed a participatory workshop and household survey of 425 farmers to examine the drivers of specific climate change risks of interest (risk of loss of livelihood for farmers) and measure farmers’ level of climate change risk perception. A climate change risk perception score (CCRPS), descriptive statistics, principal component analysis, and K-means cluster analysis were used to analyze the data collected. The findings revealed that the most important changes in climate conditions affecting agricultural production in the study area were mainly the increased duration of dry spells, erratic rainfall, and an increase in extreme rainfall events. These climatic variations cause more floods and droughts, which, when coupled with socio-ecological vulnerability drivers, increase the impact of these events on agricultural livelihood, expose more farmers and their farmland, and contribute to the risk of farmers’ livelihood loss in the study area. Based on farmers’ appraisals of the occurrence of hazards, their exposure, and their vulnerability, farmers’ perceptions of climate risk have been classified into three categories: high, moderate, and low. This finding sheds some light on farmers’ climate change risk perception, which may influence their adaptation decision. These findings can be used to increase the uptake of adaptation strategies and thus the resilience of Savanna region agriculture to climate change.