Context. Fast X-ray transients (FXTs) are X-ray flares that last from minutes to hours. Multi-wavelength counterparts to these FXTs have proven hard to find. As a result, distance measurements are made through indirect methods such as a host galaxy identification. Of the three main models proposed for FXTs, that is, supernova shock breakout emission (SN SBO), binary neutron star (BNS) mergers, and tidal dirsuption events (TDEs) of an intermediate-mass black hole (IMBH) disrupting a white dwarf (WD), the SN SBO predicts a much lower maximum peak X-ray luminosity (LX, peak). If the distance to FXTs were to be obtained, it would be a powerful probe for investigating the nature of these FXTs. Aims. We aim to obtain distance measurements to four FXTs by identifying candidate host galaxies. Through a redshift measurement of the candidate host galaxies, we derive LX, peak and the projected offset between the candidate host galaxy and the FXT. Methods. We obtained Very Large Telescope (VLT)/Multi Unit Spectroscopic Explorer (MUSE) observations of a sample of FXTs. We report the redshift of between 13 and 22 galaxies per FXT. We used these redshifts to calculate the distance, LX, peak and the projected offsets between the FXT position and the position of the sources. Additionally, we computed the chance alignment probabilities for these sources with the FXT postitions. Results. We find LX, peak > 1044 erg s−1 when we assume that any of the sources with a redshift measurement is the true host galaxy of the corresponding FXT. For XRT 100831, we find a very faint galaxy (mR, AB = 26.5 ± 0.3, z ∼ 1.22, LX, peak ∼ 8 × 1045 erg s−1 if the FXT is at this distance) within the 1σ uncertainty region with a chance alignment probability of 0.04. For XRT 060207, we find a candidate host galaxy at z = 0.939 with a low chance alignment probability within the 1σ uncertainty region. However, we also report the detection of a late-type star within the 3σ uncertainty region with a similar chance alignment probability. For the remaining FXTs (XRT 030511 and XRT 070618), we find no sources within their 3σ uncertainty regions. The projected offsets between the galaxies and the FXT positions is > 33 kpc at 1σ uncertainty. Therefore, if one of these candidate host galaxies turns out to be the true host galaxy, it would imply that the FXT progenitor originated from a system that received a significant kick velocity at formation. Conclusions. We rule out an SN SBO nature for all FXTs based on LX, peak and the projected offsets between the FXT position and the sources, assuming any of the candidate host galaxies with a redshift determination is the true host galaxy to the FXT. For XRT 100831, we conclude that the detected galaxy within the 1σ uncertainty position is likely to be the host galaxy of this FXT based on the chance alignment probability. From the available information, we are not able to determine whether XRT 060207 originated from the galaxy found within 1σ of the FXT position or was due to a flare from the late-type star detected within the 3σ uncertainty region. Based on LX, peak and the offsets within our sample, we are not able to distinguish between the BNS merger and the IMBD-WD TDE progenitor model. However, for the candidate host galaxies with an offset ≳30 kpc, we can conclude that the IMBH-WD TDE is unlikely due to the large offset.