Nonlinear nanophotonics has many potential applications, such as in mode locking, frequency-comb generation, and all-optical switching. The development of materials with large nonlinear susceptibility is key to realizing nonlinear nanophotonics. Nanostructured graphene systems, such as graphene nanoribbons and nanoislands, have been predicted to have a strong plasmon-enhanced nonlinear optical behavior in the nonretarded regime. Plasmons concentrate the light field down to subwavelength scales and can enhance the nonlinear optical effects; however, plasmon resonances are narrowband and sensitive to the nanostructure geometry. Here we show that graphene nanoribbons, particularly armchair graphene nanoribbons, have a remarkably strong nonlinear optical response in the long-wavelength regime and over a broad frequency range, from terahertz to the near infrared. We use a quantum-mechanical master equation with a detailed treatment of scattering and show that, in the retarded regime, electron scattering has a critical effect on the optical nonlinearity of graphene nanoribbons, which cannot be captured via the commonly used relaxation-time approximation. At terahertz frequencies, where intraband optical transitions dominate, the strong nonlinearity (in particular, third-order Kerr nonlinearity) stems from the jagged shape of the electron energy distribution, caused by the interband electron scattering mechanisms along with the intraband inelastic scattering mechanisms. We show that the relaxation-time approximation fails to capture this quantum-mechanical phenomenon and results in a significant underestimation of the intraband nonlinearity. At the midinfrared to near infrared frequencies, where interband optical transitions dominate, the Kerr nonlinearity is significantly overestimated within the relaxation-time approximation. These findings unveil the critical effect of electron scattering on the optical nonlinearity of nanostructured graphene, and also underscore the capability of this class of materials for nonlinear nanophotonic applications.