In complex environments, path planning is the key for unmanned aerial vehicles (UAVs) to perform military missions autonomously. This paper proposes a novel algorithm called flight cost-based Rapidly-exploring Random Tree star (FC-RRT*) extending the standard Rapidly-exploring Random Tree star (RRT*) to deal with the safety requirements and flight constraints of UAVs in a complex 3D environment. First, a flight cost function that includes threat strength and path length was designed to comprehensively evaluate the connection between two path nodes. Second, in order to solve the UAV path planning problem from the front-end, the flight cost function and flight constraints were used to inspire the expansion of new nodes. Third, the designed cost function was used to guide the update of the parent node to allow the algorithm to consider both the threat and the length of the path when generating the path. The simulation and comparison results show that FC-RRT* effectively overcomes the shortcomings of standard RRT*. FC-RRT* is able to plan an optimal path that significantly improves path safety as well as maintains has the shortest distance while satisfying flight constraints in the complex environment. This paper has application value in UAV 3D global path planning.