During the conceptual design of the Mars surface nuclear reactor coupled with supercritical carbon dioxide Brayton cycle, the heat rejection system occupies most of the weight, and reducing the weight of heat rejection system can effectively cut down the transportation cost from Earth to Mars. This paper explored the feasibility of adopting a combination of heat pipe cooling device and convective heat exchanger to design the heat rejection system for the Mars surface nuclear power station, established flow and heat transfer model and weight model, developed a thermal-hydraulic design program for the heat rejection system, and verified its accuracy; then, the NSGA-II multi-objective optimization method is used to optimize the weight of heat rejection system. Pareto front shows that with higher heat rejection proportion of convective heat exchanger, corresponding inlet velocity of Mars' atmosphere should also be increased. As the optimization result of heat rejection system, the ratio of weight to heat rejection is between 0.94 kg/kW and 2.92 kg/kW, and the maximum power consumption of convective heat exchanger accounts for 3.29% of the total power generation.