Implementation of flue gas waste heat recovery is an effective way to improve the energy utilization of marine engines. This paper aims to model and optimize a marine four-stroke dual-fuel (DF) engine coupled with a flue gas waste heat recovery system. Firstly, the DF engine and waste heat recovery system were respectively modeled in GT-Power and Simulink environments and verified with experimental data. Then, a regression model was built using the response surface method, with the intake temperature, compression ratio, and pilot fuel injection timing as input parameters and parametric analysis was performed. Finally, multi-objective optimization of the waste heat recovery system was performed using a genetic algorithm. The result showed that the optimal solution is obtained when the intake temperature is 306.18 K, the geometric compression ratio is 14.4, and the pilot fuel injection timing is −16.68 °CA after the top dead center. The corresponding brake-specific fuel consumption was 155.18 g/kWh, reduced by 3.24%, and the power was 8025.62 kW, increased by 0.32%. At the same time, 280.98 kW of flue gas waste heat generation was obtained.