The use of nuclear-powered Stirling systems for planetary and deep space exploration applications has drawn increasing attention in recent decades. As the critical energy conversion unit in the system, the free-piston Stirling engine (FPSE) is characterized by high efficiency, low mass, compact configuration, robustness, and long lifetime, but difficult to accurately model and analyze. A self-developed numerical model of the whole system, coupled with a third-order model of the FPSE, was proposed for the first time, and a nuclear-powered Stirling system comprising eight 1 kW FPSEs was designed. The methodology for designing a system with high efficiency, high specific power, and compact structure was proposed. Furthermore, the internal heat transfer and loss mechanisms were analyzed. The results revealed that the engines’ performance could be improved by increasing the temperature ratio of the hot end to the cold end. However, this leads to a corresponding increase in the size and mass of the heat exchangers. It was shown that the optimum hot-end temperatures for maximizing the system efficiency exist at different cold-end temperatures due to the radiation heat loss of heat pipes. When the hot-end and cold-end temperatures are respectively 1050 K and 450 K, a maximum system efficiency of 25.64% could be obtained. Moreover, the system efficiency and specific power could be increased by optimizing the FPSEs’ parameters and enhancing the heat exchangers’ heat transfer capabilities, and the adverse effect of the heat sink fluctuation could be inhibited accordingly. The distribution of the exergy losses changes greatly with the operating temperatures, and the primary parameter to be optimized also changes accordingly.