Natural ventilation is one of the most potential passive measures to improve the building energy performance. This study introduces a multi-factor optimization method for energy performance of buildings with natural ventilation. To maximize the utilization of natural ventilation, this method uses natural-ventilation strategy (Strategy NV) as the air conditioning system operation strategy. In Strategy NV, the air conditioning system is only on when the natural ventilation cannot fulfill the occupants’ thermal needs. Two thermal comfort models and airflow network model are combined to obtain Strategy NV. Support vector regression is applied to train a surrogate model which is used to predict the annual energy consumption in Strategy NV. Introducing the surrogate model as objective function, particle swarm optimization is used to find the optimal solution. A case study in hot summer and cold winter climate zone is given to verify the feasibility and correctness of this method. The optimal solution has 656 GJ annual energy consumption in Strategy NV, including 254 GJ annual thermal energy consumption. The optimal solution saves annual thermal energy consumption by 21% in Strategy NV comparing with Strategy FT (full time operation strategy within occupied period). The annual thermal energy consumption of the optimal solution in Strategy NV is at least 26% smaller than that of four commonly used building designs in Strategy NV; and it is at least 43% smaller than that of the four commonly used building designs in Strategy FT. The optimal solution has the least total air conditioning hours comparing with the four commonly used building designs.