The common ship structural optimization design is a deterministic method without considering uncertain factors, whereas the Reliability-Based Design Optimization (RBDO) can compensate for this deficiency. The RBDO of ship structure is a multi-parameter, high-dimensional, and high-nonlinear optimization solver. There exists a difficulty in guaranteeing accuracy and efficiency due to massive computation. In this study, the high-precision agent model for the limit state of ship hold structure is established based on agent model technology, including BP neural network, Radial Basis Function neural network, and Support Vector Machine combined with SMOTE oversampling algorithm. Furthermore, the reliability computation program is developed using Monte Carlo Simulation Method. A river-sea-going ship is considered the research object. The definition of rules, structural direct calculation result, and reliability requirement within all life cycles are considered boundary conditions. The RBDO system is constructed by the simulated annealing algorithm to investigate the lightweight structure. The established system can improve the efficiency and accuracy of the RBDO, which is significant for the ship's structural optimization design.