We investigate the numerical problem of conjugate heat exchange between liquid magnesium, a steel retort, and the air-cooling zone of the titanium sponge reduction reactor furnace at the initial stages of production. The uneven distribution of heat fluxes on the side surface of the retort significantly affects the formation of convective turbulent flow inside the apparatus. The accurate consideration of thermal boundary conditions will make it possible to adequately solve the problem of heat and mass transfer in the reactor without using model representations of heat flows on the boundaries of the retort. The problem was solved in the axisymmetric nonstationary formulation using the k-ω SST turbulence model. In the numerical simulation, two conceptual solutions of the problem were implemented. The first one involves the end-to-end calculation in the framework of the coupled heat and mass transfer problem, with the calculations performed in all parts of the titanium reduction apparatus and thermal boundary conditions set on its outer boundaries. The second model relies on an effective thermal boundary condition that is applied to the outer boundary of the retort and excludes calculations in the cooling channel when investigating magnesium convection. The first model, due to its greater completeness, is more accurate, but requires considerable computational resources and time. The model with an effective thermal boundary condition, provided that the coefficients in it are selected appropriately, gives results qualitatively and quantitatively close to those obtained through end-to-end calculations. At the same time, such a model is more economical. The paper evaluates the possibility of applying an effective boundary condition. Convective flows of molten magnesium in the retort under different modes of blowing and heating have been studied. The influence of the radiation part of the heat flow on the formation of convective currents is shown.