In this work, a virtual body-fitted grid-based immersed boundary method (IBM) combined with the thermal lattice Boltzmann flux solver (TLBFS) is proposed to efficiently simulate thermal flows with Dirichlet and Neumann boundary conditions. The method involves generating a virtual body-fitted grid along the immersed boundary and implementing the boundary conditions using a fractional step technique. In the prediction step, the finite-volume TLBFS is used to obtain the predicted flow field on the Eulerian mesh without considering the immersed boundary, while in the correction step, the correction process of the flow field is conducted on the virtual grid. Thus, the effect of the boundary is transferred to the fluid domain through the virtual grid indirectly. For Dirichlet boundary conditions, such as the no-slip condition and the specified temperature condition, the velocity and temperature corrections are considered unknowns and solved through a matrix system formed from enforcing the boundary conditions. As for the Neumann boundary condition, such as the specified heat flux condition, the temperatures at virtual layers inside the immersed body are regarded as unknowns and computed from those at virtual layers outside the wall according to the quadratic distribution. Several benchmark cases including natural, forced, and mixed convection problems are well simulated to validate the method. The numerical results are in good agreement with reference data, demonstrating the capacity of the present method to simulate thermal flows with Dirichlet and Neumann boundary conditions while requiring fewer Lagrangian points and achieving higher computational efficiency.