Linear response theory relates the decay of equilibrium magnetisation fluctuations in a ferrofluid to the frequency-dependent response of the magnetisation to a weak ac external magnetic field. The characteristic relaxation times are strongly affected by interactions between the constituent particles. Similarly, the relaxation of an initially magnetised system towards equilibrium in zero field occurs on a range of timescales depending on the structure of the initial state, and the interactions between the particles. In this work, ferrofluids are modelled as colloidal suspensions of spherical particles carrying point dipole moments, and undergoing Brownian motion. Recent theoretical and simulation work on the relaxation and linear response of these model ferrofluids is reviewed, and the effects of interactions, structure formation, and polydispersity on the characteristic time scales are outlined. It is shown that: (i) in monodisperse ferrofluids, the timescale characterising the collective response to weak fields increases with increasing interaction strength and/or concentration; (ii) in monodisperse ferrofluids, the initial, short-time decay is independent of interaction strength, but the asymptotic relaxation time is the same as that characterising the collective response to weak fields; (iii) in the strong-interaction regime, the formation of self-assembled chains and rings introduces additional timescales that vary by orders of magnitude; and (iv) in polydisperse ferrofluids, the instantaneous magnetic relaxation time of each fraction varies in a complex way due to the role of interactions.