Targets are irradiated in high-flux reactors to produce transplutonium isotopes. Neutron environment of the target is crucial for the production efficiency of transplutonium isotopes. To improve the production efficiency of transplutonium isotopes, it is necessary to research the optimization design of target. Taking the production of Californium-252 as an example, this study analyzed the impact of self-shielding effect in targets on the yield of transplutonium isotope based on the High Flux Isotope Reactor (HFIR) and High-Flux Fast Reactor (HFFR). The self-shielding effect leads to the hardening of the neutron spectrum inside the target and significantly reduces the conversion rate of nuclides. After conducting a refined energy spectrum analysis, we proposed a layered target design method based on the Genetic Algorithm (GA). To reduce computational costs, we propose a fixed source-burnup coupling approximate calculation method, which can avoid tedious burnup calculation and provide optimization direction. Using this method, we designed an optimal layered target scheme. Compared with non-layered target, the production efficiency of Cf-252 was increased by approximately 4.1 times. This study provides technical support for energy spectrum analysis and target design in producing transplutonium isotopes.