This paper involves implementing and optimizing a novel recessed-extruded endwall design for a rectangular cooling channel equipped with pin-fins. The study encompasses an in-depth exploration of the flow dynamics, pondering the generation and sustenance of vortices within the channel, using Reynolds-averaged Navier-Stokes analysis. The study delves into the evaluation of heat transfer characteristics, considering crucial parameters such as Nusselt number, friction factor, and overall heat transfer performance. These attributes are contrasted with the corresponding values observed in cases featuring flat endwalls, concerning Reynolds numbers spanning from 7400 to 36,000. A parametric study of the channel with recessed-extruded endwalls is conducted, aiming to comprehend the influence of geometric factors on channel heat transfer. A single-objective optimization process focusing on enhancing heat transfer effectiveness, quantified through the heat transfer efficiency index (HTEI), is conducted with three design variables. The results indicate that the new endwall configuration significantly enlarges the high heat transfer areas near the pin-fins compared to the flat endwall across all Reynolds numbers. This novel endwall design substantially elevates the heat transfer levels adjacent to the pin-fins, showcasing an impressive 62.4% increase in HTEI at Re = 21,500 compared to the flat endwall, with a protrusion height of 2.5% of the channel height. Furthermore, by increasing the height, the recessed-extruded endwall configuration can achieve a remarkable 91.1% increase in HTEI compared to the flat endwall. This achievement is reached with a protrusion height equivalent to 5% of the channel height at Re = 7400. Through design optimization using the radial basis neural network (RBNN) surrogate model and particle swarm optimization (PSO) technique, the HTEI of the recessed-extruded endwall design shows enhancements of 91.7% and 18.0% compared with the flat endwall and reference designs, respectively, at a Reynolds number of 21,500. The present work demonstrates the potential for enhancing the heat transfer of pin-fins channels by improving endwall design.