In recent years, experiments with flows of liquid metals in a helical magnetic field have been actively carried out. The study of the processes of mixing and crystallization of a liquid metal is of practical importance for metallurgy. With the development of nanotechnology, more and more new types of nanofluids (hybrid, ternary nanofluids) are being synthesized, and the thermophysical characteristics of which can compete with liquid metals. This circumstance served as a motivation for conducting this theoretical study. In this study, the criterion for the onset of convection in a Darcy–Brinkman porous medium layer saturated by an electrically conductive nanofluid under a helical magnetic field is investigated. The Brownian motion and thermophoresis effects are combined in the model for nanofluids, whereas the Darcy–Brinkman model is used for porous media. Instead of prescribing the nanoparticle volume fraction on the borders, we adopted a boundary condition in which the nanoparticle flow is considered to be zero. In the absence of a temperature gradient, a new type of instability in a helical magnetic field in a thin layer of a nanofluid is considered. The growth rate and the region of the development of this instability are numerically obtained depending on the profile of the azimuthal magnetic field (the magnetic Rossby number Rb) and the radial wave number k. In the presence of temperature, the stationary regime of nonuniformly rotating magnetoconvection is studied. The accurate analytical equation for the critical Rayleigh–Darcy number in terms of various non-dimensional parameters is determined using the linear stability theory. The results show that rotation and the axial (vertical) part of the helical magnetic field retard the onset of convection. While the azimuthal part of the helical magnetic field has a destabilizing effect at positive Rossby numbers Rb. The conditions for stabilization and destabilization of stationary convection in a helical magnetic field are determined for metal oxide, metallic, and semiconductor nanofluids.