Serpentine cooling channels have been employed in various industrial applications to control the heat loads in the system by increasing heat transfer while lowering the total pressure loss. As additive manufacturing (AM) enables the fabrication of complex structures, more sophisticated cooling models have been proposed to improve the thermal performance of the serpentine channel. This work designs a high-performance serpentine cooling channel based on topology optimization and triply periodic minimal surface (TPMS). The transient liquid crystal thermography (TLC) is used to reveal the endwall heat transfer characteristics of the optimized serpentine channel infilled with the Diamond TPMS topology within the Reynolds number of 10,000–40,000. The overall and local flow structures are revealed by numerical simulation, and the results are compared with the baseline serpentine channel. The experimental results show that at the Reynolds number of 10,000, the topology-optimized model with the Diamond structure shows a higher wetted-area averaged Nusselt number in the second pass by 70.7 % while reducing the total pressure loss by 19.2 %, compared to the smooth channel. Within the studied Reynolds numbers, the heat transfer enhancement in the second pass is 2.30–2.39 times relative to a smooth channel. The numerical results show that the topology-optimized model significantly improves the thermal performance by up to 82.7 % compared to the baseline because the flow recirculation and the regions of low heat transfer are eliminated. The optimized channel also mitigates the negative effect of the Dean's vortices while keeping high turbulence and improving flow and heat transfer uniformity.