The future Ricochet experiment aims at searching for new physics in the electroweak sector by providing a high precision measurement of the Coherent Elastic Neutrino-Nucleus Scattering (CENNS) process down to the sub-100 eV nuclear recoil energy range. The experiment will deploy a kg-scale low-energy-threshold detector array combining Ge and Zn target crystals 8.8 m away from the 58 MW research nuclear reactor core of the Institut Laue Langevin (ILL) in Grenoble, France. Currently, the Ricochet Collaboration is characterizing the backgrounds at its future experimental site in order to optimize the experiment’s shielding design. The most threatening background component, which cannot be actively rejected by particle identification, consists of keV-scale neutron-induced nuclear recoils. These initial fast neutrons are generated by the reactor core and surrounding experiments (reactogenics), and by the cosmic rays producing primary neutrons and muon-induced neutrons in the surrounding materials. In this paper, we present the Ricochet neutron background characterization using ^3He proportional counters which exhibit a high sensitivity to thermal, epithermal and fast neutrons. We compare these measurements to the Ricochet Geant4 simulations to validate our reactogenic and cosmogenic neutron background estimations. Eventually, we present our estimated neutron background for the future Ricochet experiment and the resulting CENNS detection significance. Our results show that depending on the effectiveness of the muon veto, we expect a total nuclear recoil background rate between 44 ± 3 and 9 ± 2 events/day/kg in the CENNS region of interest, i.e. between 50 eV and 1 keV. We therefore found that the Ricochet experiment should reach a statistical significance of 4.6 to 13.6 sigma for the detection of CENNS after one reactor cycle, when only the limiting neutron background is considered.