The rapid development of modern cold chain logistics technology has greatly expanded the sales market of agricultural products in rural areas. However, due to the uncertainty of agricultural product harvesting, relying on the experience values provided by farmers for vehicle scheduling can easily lead to low utilization of vehicle capacity during the pickup process and generate more transportation cost. Therefore, this article adopts a non-linear improved grey prediction method based on data transformation to estimate the pickup demand of fresh agricultural products, and then establishes a mathematical model that considers the fixed vehicle usage cost, the damage cost caused by non-linear fresh fruit and vegetable transportation damage and decay rate, the cooling cost generated by refrigerated transportation, and the time window penalty cost. In order to solve the model, a hybrid simulated annealing algorithm integrating genetic operators was designed to solve this problem. This hybrid algorithm combines local search strategies such as the selection operator without repeated strings and the crossover operator that preserves the best substring to improve the algorithm’s solving performance. Numerical experiments were conducted through a set of benchmark examples, and the results showed that the proposed algorithm can adapt to problem instances of different scales. In 50 customer examples, the difference between the algorithm and the standard value in this paper is 2.30%, which is 7.29% higher than C&S. Finally, the effectiveness of the grey prediction freight path optimization model was verified through a practical case simulation analysis, achieving a logistics cost savings of 9.73%.