With the exponential growth of data traffic, the use of caching and device-to-device (D2D) communication has been recognized as an effective approach for mitigating the backhaul bottleneck in unmanned aerial vehicle (UAV)-assisted networks. In this article, we investigate the issue of secure transmission in a cache-enabled UAV-relaying network with D2D communications in the presence of an eavesdropper. Specifically, both UAVs and D2D users are equipped with cache memory, which can prestore some popular content to collaboratively serve users. Considering the fairness among users, we formulate an optimization problem to maximize the minimum secrecy rate among users, by jointly optimizing the user association and UAV scheduling, transmission power, and UAV trajectory over a finite period. The joint design problem is a nonconvex mixed-integer programming problem. To efficiently solve this problem, we propose an alternating iterative algorithm based on the block alternating descent and successive convex approximation methods. Specifically, the user association and UAV scheduling, UAV trajectory, and transmission power are optimized alternately in each iteration, and the convergence of the algorithm is proven. Extensive numerical results show that the proposed joint design scheme significantly outperforms other benchmark schemes in terms of the secrecy rate.