Three-dimensional simulations are performed to identify the flow regimes of the rotating thermal flow of cold water near its density inversion point within a vertical annulus and evaluate the commensurate heat transfer abilities. The development of flow regimes and the variation in heat transfer behaviors in the special case with the density inversion parameter Θm=0.5 are explored over a wide range of parameters, with the Rayleigh number 104≤Ra≤106 and the Reynolds number Re≤150. The convection flow is radially separated into two layers by the density-maximum isosurface, with the inner and outer layers dominated by a positive and negative radial density gradient, respectively. Three axisymmetric and four non-axisymmetric basic flow regimes with different generation mechanisms are defined to describe the double-layer convection flow. Eleven flow regimes are identified, each of which consists of one, two or three basic flow regimes. The spatio-temporal features of the flow regimes depend on Ra and Re. The mixed convection parameter is demonstrated to be capable of qualitatively measuring the relative strength of the flow between the inner and outer layers in certain ranges of Ra. The mechanisms of both the flow regime transitions and the mode transitions between steady and oscillatory states are explored in terms of the combined action of density inversion effect, centrifugal effect and buoyancy effect. The heat transfer behaviors depend on the spatio-temporal features of flow regimes. The mechanism of heat transfer enhancement and that for the non-monotonic variation of heat transfer ability are revealed. The limitation of the axisymmetric assumption is quantified.