We have recently proposed a solid state heat pump based on photon mediated heat transfer between two large-area light emitting diodes coupled by the electromagnetic field and enclosed in a semiconductor structure with a nearly homogeneous refractive index. Ideally the thermophotonic heat pump (THP) allows heat transfer at Carnot efficiency but in reality there are several factors that limit the efficiency. The efficient operation of the THP is based on the following construction factors and operational characteristics: (1) broad area semiconductor diodes to enable operation at optimal carrier density and high efficiency, (2) recycling of the energy of the emitted photons, (3) elimination of photon extraction losses by integrating the emitting and the absorbing diodes within a single semiconductor structure, and (4) eliminating the reverse thermal conduction by a nanometer scale vacuum layer between the diodes. In this paper we develop a theoretical model for the THP and study the fundamental physical limitations and potential of the concept. The results show that even when the most important losses of the THPs are accounted for, the THP has potential to outperform the thermoelectric coolers especially for heat transfer across large temperature differences and possibly even to compete with conventional small scale compressor based heat pumps.