In this study, substantial efforts have been done to enhance both the economy and nitrogen dioxide (NOx) emission characteristics of an Atkinson cycle engine (ACE). The integrated simulation model was meticulously formulated and calibrated utilizing GT-Power software based on test data, and four machine learning (ML) models were developed to predict the performance of the ACE. On this basic, the improved multi-objective hybrid particle swarm optimization (IMHPSO) algorithm was proposed to conduct comprehensive optimization of ACE performance. The results highlight the superior prediction performance and generalization ability of the eXtreme Gradient Boosting (XGBoost) model, attaining an average coefficient of determination (R2) of 0.9978 and 0.9966 on the train and test sets, respectively. The IMHPSO algorithm exhibited efficient convergence, uniform distribution and rich diversity across five benchmark test problems, effectively mitigating issues related to local optima and premature convergence. The Optimal Brake Specific Fuel Consumption (BSFC) solution is chosen as the preferred optimized scheme and realizes significant reductions of 8.5 % for BSFC and 96.1 % for NOx emissions, which experiences a remarkable increase of over 1300 times in computational efficiency compared to the GT-Power model. These findings provide theoretical basis, data support and novel insights for multi-objective optimization of ACE performance.