In order to achieve the design requirements of light weight and high stiffness of the CubeSat modular separation mechanism, a BPNN-GA-PSO size optimization method is proposed to optimize the design of the separation mechanism. Firstly, the separation mechanism of modularization and standardization is analyzed to select the optimization objects. Then, a hierarchical optimization strategy of topology optimization followed by size optimization is adopted to find the global optimum using a hybrid GA-PSO optimization algorithm. Meanwhile, the BPNN surrogate model is introduced to improve the optimization efficiency. The results show that the mass proportion of the optimized separation mechanism is reduced to 18 %, and the maximum deformation of the separation mechanism is 0.123 mm, which meets the design requirements of light weight and high stiffness of the separation mechanism. It proves the applicability of the optimization method to the optimal design of the separation mechanism. The CubeSat modular separation mechanism designed in this paper has been verified by ground verification with overload, vibration, and shock mechanical tests, and successfully deployed the BY-03 satellite in-orbit, which can provide reference for the design and development of subsequent CubeSat modular separation mechanisms.