Climate change has recently induced an increase in natural disasters worldwide, which has affected agriculture, a sector highly sensitive to climatic alterations. The paper measures agricultural disaster vulnerability of China in the period 2000–2020 by establishing vulnerability assessment system, and examines how agricultural infrastructure correlates with agricultural disaster vulnerability by constructing grey relational analysis model. Research results indicate that China's agricultural disaster vulnerability presents an overall decreasing trend during the research period. This is mainly attributed to the decreased disaster exposure and increased adaptability to disaster. Spatially, the agricultural disaster vulnerability in the northeastern, central and southwestern regions of China is higher than other regions. Agricultural transportation and communication infrastructure are found to be more influential to agricultural disaster vulnerability followed by irrigation and power facilities. Implications are proposed to enhance agricultural resilience to climate disasters in China.