Aims. The black hole binary SWIFT J1753.5-0127 is providing a unique data set to study accretion flows. Various investigations of this system and of other black holes have not, however, led to an agreement on the accretion flow geometry or on the seed photon source for Comptonization during different stages of X-ray outbursts. We place constraints on these accretion flow properties by studying long-term spectral variations of this source. Methods. We performed phenomenological and self-consistent broad band spectral modeling of Swift J1753.5-0127 using quasi-simultaneous archived data from INTEGRAL/ISGRI, Swift/UVOT/XRT/BAT, RXTE/PCA/HEXTE and MAXI/GSC instruments. Results. We identify a critical flux limit, F \sim 1.5 \times 10^{-8} erg/cm^2/s, and show that the spectral properties of SWIFT J1753.5-0127 are markedly different above and below this value. Above the limit, during the outburst peak, the hot medium seems to intercept roughly 50 percent of the disk emission. Below it, in the outburst tail, the contribution of the disk photons reduces significantly and the entire spectrum from the optical to X-rays can be produced by a synchrotron-self-Compton mechanism. The long-term variations in the hard X-ray spectra are caused by erratic changes of the electron temperatures in the hot medium. Thermal Comptonization models indicate unreasonably low hot medium optical depths during the short incursions into the soft state after 2010, suggesting that non-thermal electrons produce the Comptonized tail in this state. The soft X-ray excess, likely produced by the accretion disk, shows peculiarly stable temperatures for over an order of magnitude changes in flux. Conclusions. The long-term spectral trends of SWIFT J1753.5-0127 are likely set by variations of the truncation radius and a formation of a hot, quasi-spherical inner flow in the vicinity of the black hole. (abridged)