PurposeThis study aims to numerically examine mixed convection of CuO-water nanofluid in a three-dimensional (3D) vented cavity with inlet and outlet ports under the influence of an inner rotating circular cylinder, homogeneous magnetic field and surface corrugation effects. In practical applications, it is possible to encounter some of the considered configurations in a vented cavity such as magnetic field, rotating cylinder and it is also possible to specially add some of the active and passive control means to control the convection inside the cavity such as adding nanoparticles, corrugating the surfaces. The complicated physics with nanofluid under the effects of magnetic field and inclusion of complex 3D geometry make it possible to use the results of this numerical investigation for the design, control and optimization of many thermal engineering systems as mentioned above.Design/methodology/approachThe bottom surface is corrugated with a rectangular wave shape, and the rotating cylinder surface and cavity bottom surface were kept at constant hot temperatures while the cold fluid enters the inlet port with uniform velocity. The complicated interaction between the forced convection and buoyancy-driven convection coupled with corrugated and rotating surfaces in 3D configuration with magnetic field, which covers a wide range of thermal engineering applications, are numerically simulated with finite element method. Effects of various pertinent parameters such as Richardson number (between 0.01 and 100), Hartmann number (between 0 and 1,000), angular rotational speed of the cylinder (between −30 and 30), solid nanoparticle volume fraction (between 0 and 0.04), corrugation height (between 0 and 0.18H) and number (between 1 and 20) on the convective heat transfer performance are numerically analyzed.FindingsIt was observed that the magnetic field suppresses the recirculation zone obtained in the lower part of the inlet port and enhances the average heat transfer rate, which is 10.77 per cent for water and 6.86 per cent for nanofluid at the highest strength. Due to the thermal and electrical conductivity enhancement of nanofluid, there is 5 per cent discrepancy in the Nusselt number augmentation with the nanoadditive inclusion in the absence and presence of magnetic field. The average heat transfer rate of the corrugated surface enhances by about 9.5 per cent for counter-clockwise rotation at angular rotational speed of 30 rad/s as compared to motionless cylinder case. Convective heat transfer characteristics are influenced by introducing the corrugation waves. As compared to number of waves, the height of the corrugation has a slight effect on the heat transfer variation. When the number of rectangular waves increases from N = 1 to N = 20, approximately 59 per cent of the average heat transfer reduction is achieved.Originality/valueIn this study, mixed convection of CuO-water nanofluid in a 3D vented cavity with inlet and outlet ports is numerically examined under the influence of an inner rotating circular cylinder, homogeneous magnetic field and surface corrugation effects. To the best of authors knowledge such a study has never been performed. In practical applications, it is possible to encounter some of the considered configurations in a vented cavity such as magnetic field, rotating cylinder and it is also possible to specially add some of the active and passive control means to control the convection inside the cavity such as adding nanoparticles, corrugating the surfaces. The complicated physics with nanofluid under the effects of magnetic field and inclusion of complex 3D geometry make it possible to use the results of this numerical investigation for the design, control and optimization of many thermal engineering systems as mentioned above.