The electric dipole approximation is commonly adopted in the theoretical investigation of light-atom/molecule interaction, wherein the magnetic component of the driving electromagnetic field is neglected. Our study highlights the significant role of the magnetic field effect in the recollision dynamics of nonsequential double ionization (NSDI) driven by a mid-infrared laser. Due to the magnetic component of the laser field, in the multiple-returning events, the tunneling electron with a large initial momentum along the laser magnetic field direction at some specific tunneling time is inefficient for NSDI. The corresponding footprint is revealed in the correlated electron momentum distribution along the magnetic field direction. Moreover, we show that this effect becomes more obvious with increasing laser wavelength, leading to a notable reduction in the NSDI yield. Our findings provide an alternative perspective for studying the recollision dynamics involving the magnetic field effect.