Permeable pavement systems are important facilities for alleviating runoff pollution. However, few studies have focused on outflow nitrogen processes under varying rainfall conditions. In this study, we proposed a permeable pavement system model that can simulate the nitrogen transformation processes. The model was verified with data on outflow rates and outflow nitrogen concentrations (ammonia nitrogen (NH4-N), nitrate nitrogen (NO3-N), and total nitrogen (TN)) processes for six consecutive rainfall events of a permeable pavement system in Shenzhen, China. Based on the validated model, scenarios were designed to simulate the influences of rainfall conditions on nitrogen removal effects at the single rainfall event and annual scale, respectively. The results indicate that: (i) the model can explicitly describe hydrological and nitrogen processes of the system, and the Nash Sutcliffe Efficiency and percent bias of outflow rates and outflow nitrogen concentrations range from 0.5 to 0.9 and from −22.4% to 24.9%, respectively; (ii) the sensitivity analysis reveals that the empirical coefficient (C) related to evaporation in the gravel layer is sensitive at the annual scale but not at rainfall events scale. Parameters related to mineralization and microbial assimilation (i.e., mineralization rate constant in the gravel layer (k3N,mine), microbial assimilation rate constant of NH4-N, NO3-N, and ON (k1N,bio, k2N,bio, k3N,bio)) are sensitive annually but not at rainfall events scale; (iii) in the single rainfall event simulations, nitrogen removal efficiencies decrease with increasing rainfall return periods but is not significantly affected by rainfall peak coefficients. Nitrogen removal efficiencies improve notably with antecedent dry period extension for single rainfall event with antecedent dry period less than 2 days. In addition, we found that the outflow process significantly affects the outflow nitrogen concentrations process. In annual-scale simulations, extending the antecedent dry period only significantly boosts runoff reduction efficiency and nitrogen removal efficiencies for rainfall events with less than 10 mm.