In this study, a novel two-stage TEG with different phase change materials (PCMs) embedding modes is proposed to improve the efficiency of TEG systems. The output power and electrical energy of single-stage and two-stage TEG combined with different PCMs is firstly compared at the limit of total length of the PCM containers. Furthermore, the effect of different PCMs embedding modes, including arranged at the heat source side, between two TEGs and heat sink side, on the performance of two-stage TEG system was discussed. Finally, the amount of different PCMs arranged at different locations was optimized. The results showed that although application of high temperature PCM on the heat source side can alleviate the effect of heat source fluctuations to provide more stabilized output power, the total electrical energy is seriously reduced in the two-stage TEG system. Compared with the single-stage TEG system, the two-stage TEG system provides a much higher output power and electrical energy, in which the harvested total electrical energy of the optimum two-stage TEG system is increased by 44.8% with a conversion efficiency of 6.9%. These results are of great significance to the design of advanced TEG system for waste heat recovery.