Rim-driven thruster (RDT) is a new integrated thruster that has attracted much attention recently. The RDT provides the benefits of reducing vibration and noise and increasing useable volume by mounting the propulsion motor stator into the duct and fusing the motor rotor with the propeller blades. The calculation method of RDT considering the motor gap was discussed and validated. A simulation-based design optimization framework was proposed for the particularity of RDT ducts. The parametric reconstruction of the duct was executed based on the Bézier curve. The optimal Latin hypercube design method was used to generate samples, and the approximate model was developed based on the radial basis function (RBF) neural network. The fitting R2 of efficiency, minimum pressure, blade thrust, and total thrust of the approximate model reached 0.998, 0.955, 0.998, and 0.990, respectively. On this basis, the NSGA-II was employed to optimize the duct with minimum pressure and efficiency as the optimization objectives. Sensitivity analyses indicate that the optimization objectives are significantly affected by the nozzle size, followed by the internal nozzle profile. The typical Pareto solutions were verified by CFD calculation, and the duct optimization can improve efficiency by 3.3% or minimum pressure by 5.3%.