We show that a class of L-loop conformal ladder graphs are intimately related to twisted partition functions of free massive complex scalars in d=2L+1 dimensions. The graphs arise as four-point functions in certain two- and four-dimensional conformal fishnet models. The twisted thermal two-point function of the scalars becomes a generator of conformal ladder graphs for all loops. We argue that this correspondence is seeded by a system of two decoupled harmonic oscillators twisted by an imaginary chemical potential. We find a number of algebraic and differential relations among the conformal graphs that mirror the underlying free dynamics.