We determined the direction and strength of the photospheric and lower chromospheric magnetic field in the umbra and penumbra of a sunspot from inversions of spectropolarimetric observations of photospheric lines at 617\,nm and 1565\,nm, and the chromospheric \ion{Ca}{ii} IR line at 854\,nm, respectively. We compare the magnetic field vector with the direction of 75 flow channels that harbor the chromospheric inverse Evershed effect (IEF) near their downflow points (DFPs) in the sunspot's penumbra. The azimuth and inclination of the IEF channels to the line of sight (LOS) were derived from spatial maps of the LOS velocity and line-core intensity of the \ion{Ca}{ii} IR line and a thermal inversion of the \ion{Ca}{ii} IR spectra to obtain temperature cubes. We find that the flow direction of the IEF near the DFPs is aligned with the photospheric magnetic field to within about $\pm$\,15\,deg. The IEF flow fibrils make an angle of 30--90\,deg to the local vertical with an average value of about 65\,deg. The average field strength at the DFPs is about 1.3\,kG. Our findings suggest that the IEF in the lower chromosphere is a field-aligned siphon flow, where the larger field strength at the inner footpoints together with the lower temperature in the penumbra causes the necessary gas pressure difference relative to the outer footpoints in the hotter quiet Sun with lower magnetic field strength. The IEF connects to magnetic field lines that are not horizontal like for the regular photospheric Evershed flow, but which continue upwards into the chromosphere indicating an "uncombed" penumbral structure.