Relating in situ measurements of relativistic solar particles to their parent activity in the corona requires understanding the magnetic structures that guide them from their acceleration site to the Earth. Relativistic particle events are observed at times of high solar activity, when transient magnetic structures such as Interplanetary Coronal Mass Ejections (ICMEs) often shape the interplanetary magnetic field (IMF). They may introduce interplanetary paths that are longer than nominal, and magnetic connections rooted far from the nominal Parker spiral. We present a detailed study of the IMF configurations during ten relativistic solar particle events of the 23rd activity cycle to elucidate the actual IMF configuration guiding the particles to Earth, where they are measured by neutron monitors. We use magnetic field (MAG) and plasma parameter measurements (SWEPAM) from ACE, and determine interplanetary path lengths of energetic particles through a modified version of the velocity dispersion analysis based on energetic particle measurements with SoHO/ERNE. We find that the majority (7/10) of the events is detected in the vicinity of an ICME. Their interplanetary path lengths are found to be longer (1.5-2.6 AU) than those of the two events propagating in the slow solar wind (1.3 AU). The largest apparent path length is found in an event within the fast solar wind, probably due to enhanced pitch angle scattering. The derived path lengths imply that the first energetic and relativistic protons are released at the Sun at the same time as electron beams emitting type III radio bursts. The timing of the first high-energy particle arrival at Earth is dominantly determined by the type of IMF in which the particles propagate. Initial arrival times are as expected from Parker's model in the slow solar wind, and significantly larger in or near transient structures such as ICMEs.