We introduce the notion of relative entropy in the framework of thermodynamics of compressible, viscous and heat conducting fluids. The relative entropy is constructed on the basis of a thermodynamic potential called ballistic free energy and provides stability of solutions to the associated Navier-Stokes-Fourier system with respect to perturbations. The theory is illustrated by applications to problems related to the long time behavior of solutions and the problem of weak-strong uniqueness.