The big data provided by unmanned aerial vehicle (UAV) visual sensors offers essential information resources for activities across various industries. However, various adversarial threats are inevitable throughout the lifecycle of data generation, transmission, and utilization, leading to serious security risks. Trust assessment of visual sensors is a prerequisite for securing UAVs, but the multidimensionality of the trust elements and the uncertainty of the evidence limit its practical application. To advance this research, we innovatively propose a trust management scheme based on multi-granularity evidence fusion within the framework of belief functions (BFs) theory to adaptively respond to both known and unknown threats. We first propose a direct trust assessment model for known threats, which constructs multidimensional coarse-grained trust elements (MCTEs) and integrates multiple lightweight sub-models for basic belief assignment (BBA) to meet the need for fast response. Then, to address the unknown threats, we introduce pre-trained models to build multidimensional fine-grained trust elements (MFTEs) to construct trust recommendation models for indirect trust assessment for visual sensors. In addition, to accurately characterize the trustworthiness of visual sensors, we also introduce a BBA-weighted fusion method to achieve more reasonable trust aggregation by weakening highly conflicting evidence sources. Finally, to validate the effectiveness of the proposed method, we conducted a comprehensive trust assessment and security experiment on UAV aerial images. The results indicate that the proposed method demonstrates excellent performance and is beneficial for enhancing UAV security in adversarial attack scenarios.