We study heat transfer in plane Couette flow laden with rigid spherical particles by means of direct numerical simulations. In the simulations we use a direct-forcing immersed boundary method to account for the dispersed phase together with a volume-of-fluid approach to solve the temperature field inside and outside the particles. We focus on the variation of the heat transfer with the particle Reynolds number, total volume fraction (number of particles) and the ratio between the particle and fluid thermal diffusivity, quantified in terms of an effective suspension diffusivity. We show that, when inertia at the particle scale is negligible, the heat transfer increases with respect to the unladen case following an empirical correlation recently proposed in the literature. In addition, an average composite diffusivity can be used to approximate the effective diffusivity of the suspension in the inertialess regime when varying the molecular diffusion in the two phases. At finite particle inertia, however, the heat transfer increase is significantly larger, smoothly saturating at higher volume fractions. By phase-ensemble-averaging we identify the different mechanisms contributing to the total heat transfer and show that the increase of the effective conductivity observed at finite inertia is due to the increase of the transport associated with fluid and particle velocity. We also show that the contribution of the heat conduction in the solid phase to the total wall-normal heat flux reduces when increasing the particle Reynolds number, so that particles of low thermal diffusivity weakly alter the total heat flux in the suspension at finite particle Reynolds numbers. On the other hand, a higher particle thermal diffusivity significantly increases the total heat transfer.