Unmanned Aerial Vehicles (UAVs) are commonly equipped with multiple sensors to facilitate control and navigation. However, due to the complexity of the flight environment, the UAV sensors are prone to damage, which can cause serious accidents or even economic losses. Therefore, it is of great importance to propose an effective UAV sensor fault diagnosis method. The following weaknesses are obtained by the existing fault diagnosis methods: 1) The model-based methods, such as parameter estimation, state estimation, and equivalent space method, require introducing high-precision system models and expert knowledge. 2) Conventional graph convolutional network (GCN)-based methods fail to guarantee the accuracy of association graphs. To address the above two challenges, the dynamics model of quadrotor as prior knowledge is proposed in this paper to construct the association graph first. Then, a spatial-temporal difference graph convolutional network (STDGCN) consisting of a difference layer and multiple spatial-temporal graph convolutional modules is built. The difference layer enhances the feature extraction of the graph nodes, while the spatial-temporal graph convolutional modules extract the spatial-temporal dependencies of sensor data. Finally, the robustness and effectiveness of STDGCN are demonstrated in fault diagnosis compared with the commonly used data-driven methods. The experimental result shows that the proposed STDGCN method exhibits outstanding feature extraction ability with the highest diagnostic accuracy.