Unmanned aerial vehicles (UAV) on Mars operate in a thin atmosphere of carbon dioxide, where the flow Reynolds number and local sound velocity are significantly reduced compared to the Earth's atmosphere because of low gas density and temperature. Therefore, a Mars UAV with a rotary-wing design can operate in a state of low-Reynolds number (Re = 1-5 × 104) and transonic flow (Matip = 0.7–1.2), where flow separation and shock waves occur simultaneously. The interaction between flow separation and shock waves not only makes the entire flow field more complex but also seriously affects the aerodynamic performance of the airfoil, thereby reducing the payload of UAVs, which previous aircraft designs based on the thick atmospheric environment of the Earth have not considered. Therefore, in this study, a numerical simulation method is used to conduct airfoil optimization research on a characteristic airfoil under low-Reynolds number transonic flow (Re = 33200, Ma = 0.85) conditions to acquire additional insight, improve aerodynamic performance of the airfoil, and enhance the payload. The results were compared with the optimization results of subsonic flow (Re = 19500, Ma = 0.50). The Hicks-Henne bump functions method was selected for airfoil parameterization, and non-dominated sorting genetic algorithms (NSGA)-Ⅱ were used for optimization. Through an analysis of the pressure coefficient (Cp), surface friction coefficient (Cf), and density gradient, the number and intensity of oblique shock waves on the upper and lower surfaces were observed to significantly affect the aerodynamic performance of airfoils under low-Reynolds number flow. The changes in flow separation and transition positions do not directly affect the aerodynamic performance but weaken the oblique shock wave intensity induced by the separation point or directly transform strong shock waves into weaker compression waves because of the absence of flow separation. The optimization results for both subsonic and transonic flow point to a thinner overall shape with a flatter surface with the drag greatly reduced due to the disappearance of shock waves on the lower surface, which significantly improves aerodynamic performance. Compared with the original airfoil, at Ma = 0.85, this design can increase the lift coefficient (Cl) by at least 124 % and lift drag ratio (Cl/Cd) by 177 %.