Falling damage is the most common form of damage sustained by kiwifruit during the process of picking and post-processing, and it is difficult to conduct a quantitative analysis of this phenomenon through traditional experimental methods. In order to deeply understand the sensitivity of kiwifruit to falling collision damage, the finite element numerical simulation method was used to evaluate and predict the sensitivity of kiwifruit to falling collision damage during harvesting. First, we obtained the appearance characteristics of kiwifruit through reverse engineering technology and determined the geometric and mechanical property parameters of kiwifruit through physical mechanics experiments. Then, according to the characteristics of fruit tissue structure, a multiscale finite element model, including the skin, pulp, and core, was constructed to simulate the effects of different falling heights, collision angles, and contact surface materials on fruit damage, and the accuracy of the model was verified through falling experiments. Finally, based on the simulation results, the Box–Behnken design was employed within the response surface methodology to establish a sensitivity prediction model for the drop damage sensitivity of kiwifruit across different contact materials. The results showed that the maximum relative error between the speed change obtained using finite element simulation and the speed obtained by the high-speed camera was 5.19%. The model showed high rationality in energy distribution, with the maximum value of hourglass energy not exceeding 0.08% of the internal energy. On the contact surface material with a large elastic modulus, a higher falling height and larger collision angle will significantly increase the risk of fruit bruise. When the contact surface material was a steel plate, the falling height was 1 m, and the collision angle was 90°; the maximum bruise sensitivity of kiwifruit reached 6716.07 mm3 J−1. However, when the contact surface material was neoprene, the falling height was 0.25 m, and the collision angle was 0°, the damage sensitivity was the lowest, at 1570.59 mm3 J−1. The multiscale finite element model of kiwifruit falling collision constructed in this study can accurately predict the damage of kiwifruit during falling collision and provide an effective tool for the quantitative analysis of kiwifruit falling collision damage. At the same time, this study can also provide guidance for the design and optimization of the loss reduction method of the harvesting mechanism, which has important theoretical significance and practical value.