Nitrogen-doped graphene quantum dots (N-GQDs) are a new class of quantum dot with unique electronic properties, whose electrons transport and biochemical properties have been extensively studied. However, the optical properties of N-GQDs, especially their nonlinear optical response and applications in ultrafast photonics and photoelectronic, need to be further explored. The nonlinear optical response and ultrafast carrier dynamics of N-GQDs in an aqueous solution are systematically investigated in this work. N-GQDs have an ultrafast carrier cooling delay of about 143 fs in the visible range. In the study of the nonlinear optical response of N-GQDs, the saturable absorption is transformed into reverse saturable absorption as the incident pulse intensity increases, which makes N-GQDs an effective saturable absorber for ultrashort pulse generation in mode-locked Er-doped fiber lasers. Our results may lead to further investigations into the optical properties of N-GQDs and demonstrate potential opportunities for ultrafast photonic applications.