The deployment of unmanned aerial vehicle (UAV) in scenarios with limited or no infrastructure can provide services for mobile devices flexibly. This letter investigates the UAV transmission energy minimization problem with high data rate and reliability requirements in a UAV-based Orthogonal Frequency Division Multiple Access (OFDMA) network. Considering that the formulated problem is a non-convex problem, we decompose the original problem into two subproblems: (i) joint subcarrier allocation and modulation mode selection subproblem, and (ii) UAV trajectory design subproblem. By solving these two subproblems, we propose the joint subcarrier allocation, modulation mode selection, and trajectory design (JSMT) algorithm. Simulation results show that the proposed JSMT algorithm can efficiently reduce the transmission energy of the UAV while meeting high data rate and reliability requirements.