This study explores the dynamics of a mixed magneto-hybrid nanofluid (HNF) flowing over a rotating cone within a rotational fluid flow, focusing on the effects of joule heating and thermal radiation. Employing water as a base fluid, enhanced with nanoparticles like molybdenum disulfide (MoS2) and graphene oxide (GO), we examine the critical role of liquid–solid interfacial layers on thermal integrity and boundary conditions. The governing model integrates joule heating, thermal radiation, mixed convection, and magnetic effects to fully represent the complexities of the flow dynamics. A system of partial differential equations (PDEs), simplified under boundary layer assumptions, is transformed into ordinary differential equations (ODEs) through similarity transformations. These ODEs are then solved using the Homotopy Analysis Method (HAM) in Mathematica. Notably, our results demonstrate an improvement in heat transfer rates under combined magnetic and rotational influences, compared to conventional fluids. This enhanced cooling efficiency is critical for applications like aeronautical gas turbines and power production turbines, where higher thermal regulation directly correlates with improved performance, durability, and operational efficiency. Moreover, the study reveals a significant sensitivity of the thermal boundary layer to variations in the Prandtl number, indicating that higher Prandtl numbers lead to a lower temperature profile. This finding is vital for optimizing heat transfer processes, particularly in engineering applications where precise control over thermal properties is required. Furthermore, the study provides detailed insights into the friction factors for azimuthal and tangential directions, alongside local Nusselt numbers, offering substantial agreement with prior research. These findings contribute significantly to the design and optimization of modern cooling systems, emphasizing the potential of HNFs in high-temperature environments.