With the advantages of fast response, good stability, and strong robustness to filter parameter variations as well as load perturbations, the application of passivity-based control (PBC) in the single-phase voltage source inverter (SPVSI) has received extensive attention. However, as a non-linear control strategy, PBC control has a complex coupling structure containing feed-forward, negative feedback, etc., which poses many difficulties for multi-damping design. In addition, ensuring the passivity of the islanded system under various load conditions or filter parameter drift is still an unresolved issue in the damping coefficient design of the PBC controller. Therefore, for SPVSI systems with LC filters, this paper proposes a multi-damping coefficient design method for PBC controllers based on stability domain analysis. The designed PBC controller can ensure system stability under several load conditions or certain deviations of LC filter parameters. Moreover, the design order from the inner to the outer control loop is adopted according to the distribution of individual damping coefficients in the control loop. As a result, each control loop has a good dynamic performance and distinct control bandwidth. Finally, robustness analysis, simulation, and experimental results verify the correctness and effectiveness of the proposed multi-damping coefficients design method for the PBC controller.