Context. High resolution spectroscopy at high signal-to-noise ratios (S/Ns) is one the key techniques of the quantitative study of the atmospheres of extrasolar planets. Observations at near-infrared wavelengths with fiber-fed spectrographs coupled to extremely large telescopes are particularly important to tackle the ultimate goal of detecting biosignatures in rocky planets. Aims. To achieve high S/Ns in fiber-fed spectrogrpahs, the systematic noise effects introduced by the fibers must be properly understood and mitigated. In this paper we concentrate on the effects of modal noise in multimode fibers. Methods. Starting from our puzzling on-sky experience with the GIANO-TNG spectrometer we set up an infrared high resolution spectrometer in our laboratory and used this instrument to characterize the modal noise generated in fibers of different types (circular and octagonal) and sizes. Our experiment includes two conventional scrambling systems for fibers: a mechanical agitator and an optical double scrambler. Results. We find that the strength of the modal noise primarily depends on how the fiber is illuminated. It dramatically increases when the fiber is under-illuminated, either in the near field or in the far field. The modal noise is similar in circular and octagonal fibers. The Fourier spectrum of the noise decreases exponentially with frequency; i.e., the modal noise is not white but favors broad spectral features. Using the optical double scrambler has no effect on modal noise. The mechanical agitator has effects that vary between different types of fibers and input illuminations. In some cases this agitator has virtually no effect. In other cases, it mitigates the modal noise, but flattens the noise spectrum in Fourier space; i.e., the mechanical agitator preferentially filters the broad spectral features. Conclusions. Our results show that modal noise is frustratingly insensitive to the use of octagonal fibers and optical double scramblers; i.e., the conventional systems used to improve the performances of spectrographs fed via unevenly illuminated fibers. Fiber agitation may help in some cases, but its effect has to be verified on a case-by-case basis. More generally, our results indicate that the design of the fiber link feeding a spectrograph should be coupled with laboratory measurements that reproduce, as closely as possible, the conditions expected at the telescope.