In this paper, we study computationally feasible bounds for relative free energies between two many-particle systems. Specifically, we consider systems out of equilibrium that admit a nonequilibrium steady state that is reached asymptotically in the long-time limit. The bounds that we suggest are based on the well-known Bogoliubov inequality and variants of Gibbs' and Donsker–Varadhan variational principles. As a general paradigm, we consider systems of oscillators coupled to heat baths at different temperatures. For such systems, we define the free energy of the system relative to any given reference system in terms of the Kullback–Leibler divergence between steady states. By employing a two-sided Bogoliubov inequality and a mean-variance approximation of the free energy (or cumulant generating function), we can efficiently estimate the free energy cost needed in passing from the reference system to the system out of equilibrium (characterised by a temperature gradient). A specific test case to validate our bounds are harmonic oscillator chains with ends that are coupled to Langevin thermostats at different temperatures; such a system is simple enough to allow for analytic calculations and general enough to be used as a prototype to estimate, e.g. heat fluxes or interface effects in a larger class of nonequilibrium particle systems.