With the integration of treatments with MRI-linacs to the clinical workflow, the understanding and characterization of detector response in reference dosimetry in magnetic fields are required. The external magnetic field perturbs the electron fluence. The degree of perturbation depends on the irradiation conditions and on the detector type. The purpose of this study is to evaluate the magnetic field impact on the electron fluence spectra in several detectors to provide a deeper understanding of detector response in these conditions. Monte Carlo calculations of the electron fluence are performed in six detectors (solid-state: PTW60012 and PTW60019, ionization chambers: PTW30013, PTW31010, PTW31021, and PTW31022) in water and irradiated by a 7 MV FFF photon beam with a small and a reference field, at 0 and 1.5 T. Three chamber axis orientations are investigated: parallel or perpendicular (either the Lorentz force pointing towards the stem or the tip) to the magnetic field and always perpendicular to the photon beam. One orientation for the solid-state detector is studied: parallel to the photon beam and perpendicular to the magnetic field. Additionally, electron fluence spectra are calculated in modified detector geometries to identify the underlying physical mechanisms behind the fluence perturbations. The total electron fluence in the Farmer chamber varies up to 1.24% and 5.12% at 1.5 T, in the parallel and perpendicular orientation, respectively. The interplay between the gyration radius and the Farmer chamber cavity length significantly affects the electron fluence in the perpendicular orientation. For the small-cavity chambers, the maximal variation in total electron fluence is 0.19% in the parallel orientation for the reference field. Significant small-field effects occur in these chambers; the magnetic field reduces the total electron fluence (with respect to the no field case) between 9.86% and 14.50%, depending on the orientation. The magnetic field strongly impacted the solid-state detectors in both field sizes, probably due to the high-Z components and cavity density. The maximal reductions of total electron fluence are 15.06 ± 0.09% (silicon) and 16.00 ± 0.07% (microDiamond). This work provides insights into detector response in magnetic fields by illustrating the interplay between several factors causing dosimetric perturbation effects: (1) chamber and magnetic field orientation, (2) cavity size and shape, (3) extracameral components, (4) air gaps and their asymmetry, (5) electron energy. Low-energy electron trajectories are more susceptible to change in magnetic fields, and are associated with detector response perturbation. Detectors with higher density and high-Z extracameral components exhibit more significant perturbations in the presence of a magnetic field, regardless of field size.