In this paper, mixed integer nonlinear programming (MINLP) is optimized by PSO_GA–SQP, the mixed coding of a particle swarm optimization (PSO), and a hybrid genetic algorithm and sequential quadratic programming (GA–SQP). The population is separated into two groups: discrete and continuous variables. The discrete variables are optimized by the adapted PSO, while the continuous variables are optimized by the GA–SQP using the discrete variable information from the adapted PSO. Therefore, the population can be set to a smaller size than usual to obtain a global solution. The proposed PSO_GA–SQP algorithm is verified using various MINLP problems including the designing of retrofit heat exchanger networks. The fitness values of the tested problems are able to reach the global optimum.