The study of flow around a stationary or rotating circular cylinder holds substantial importance in numerous engineering disciplines, including building, bridge, and structural engineering. This knowledge is utilized to manage flow-induced vibrations, implement blowing or suction techniques, control moving surfaces, and develop acoustic thrusters and synthetic jets. The current aims to analyze the numerical and RSM optimization with Box–Behnken design (BBD) for the buoyancy driven convective flow and heat management features of Magneto hybrid nanofluid filled inside a novel oval-shaped annulus enclosure with concentric hot circular cylinder. The water used as base fluid and iron oxide and Multiwall carbon nanotubes as nanoparticles. The finite element method is employed to solve numerically the coupled equations. The RSM optimization with Box–Behnken design (BBD) is applied to optimize the heat transfer rate across different three factors. The stream lines, isotherms and line graphs are design for different flow parameters effect. The results indicated that heat transfer is enhanced by using of hybrid nanofluid. The velocity is reduced via increment in Hartmann number. The local Nusselt number is increases with increment in the values of the Rayleigh number. Furthermore the Heat transfer rate is boosted up with increases the concentration of nanoparticles.