Context. Detailed realistic 3D simulations of the photosphere of the Sun are now available, but 1D models of the average quiet-Sun photosphere are still widely used, in particular for spectro-polarimetric inversions. Aims. Here we present an empirical determination of the average radiation temperature variations as a function of the geometrical height above the continuum formation level in the solar photosphere.Methods. We used high resolution spectroscopic scans in the 630 nm Fe i line pair at varying heliocentric angles along the north-south polar axis of the Sun, made with SOT onboard Hinode. Implementing a new method for image reconstruction, we obtained images of the photospheric granulation at constant continuum opacity levels, from the upper photosphere seen at line centers to the low photosphere. The Fourier cross-spectra of images at different opacity levels were computed, and we derived the formation depths of images without referring to any atmospheric model, by measuring the slope of the cross-spectrum phase. Results. A modified Milne-Eddington model for the line formation was tested by comparing it with the average line-intensity profiles observed at solar disk center. It yields consistent results for the FeI 630.2 nm line, whereas the FeI line at 630.1 nm is not well reproduced by the model. We ascribe this discrepancy to non-LTE effects in the line formation processes. The average image intensities at the different FeI 630.2 nm levels were used to determine the depth-variation of the temperature for an average 1D model of the quiet photosphere. We compared our empirical temperature model with the widely used FALC model. Both models agree well for the temperature variations with the continuum optical depth. But in the low photosphere, the temperature gradient we measure with respect to the geometrical height is significantly softer than in Model C. We argue that some of the assumptions used to solve the pseudohydrostatic equilibrium in semi-empirical models are probably at fault. We also derived empirical values for the 630 nm continuum absorption coefficient as a function of the geometrical height in the low photosphere. Finally, we were able to measure the altitude of the base of the granulation contrast inversion layer, which is found at about 130 km above the base of the photosphere, in agreement with 3D MHD simulations.