Nitrogen (N) and phosphorus (P) play a critical role in agricultural production and cause many environmental disturbances. By combing life cycle assessment (LCA) method with the mass balance principle of substance flow analysis (SFA), this study establishes a nutrient-derived environmental impact assessment (NEIA) model to analyze the environmental impacts caused by nutrient-containing substances of agricultural production in China during 1978–2015. The agricultural production system is composed of crop farming and livestock breeding, and the environmental impacts include energy consumption, global warming, acidification, and eutrophication. The results show all these environmental impacts had increased to 8.22*109 GJ, 5.01*108 t CO2-eq, 2.41*107 t SO2-eq, and 7.18*107 t PO43--eq, respectively. It is noted the energy consumption and the climate change caused by the crop farming were always higher than those from livestock breeding, which were average 60 and two times, respectively. While the acidification and the eutrophication were opposite after 1995 and 2000, even they were similar. This was mainly due to the high N application including synthetic N fertilizer (from 1.33*109 GJ to 2.08*109 GJ), applied manure (from 4.94*108 GJ to 5.65*108 GJ) and applied crop residue (from 2.94*108 GJ to 5.30*109 GJ), while the synthetic N fertilizer was controlled and the livestock expanded rapidly after 1995. Among the sub-categories, the three staple crops (rice, wheat, and maize) contributed greater environmental impacts, which were about two to 10 times as other crops and livestock, due to their high fertilizer uses, sown areas and harvests. While the oil crops and fruit consumed the least energies because of their much lower fertilizer-use intensities. Pig and poultry especially pig also caused obvious effects on environment (even 20 times as other livestock) because of their large quantities and excretions, which emitted much higher N2O and P loss resulting in much higher climate change, acidification and eutrophication than other livestock. Then the study proposes the nutrient management in agricultural production by considering crop production, livestock breeding and dietary adjustment, so that some valuable experiences can be shared by the stakeholders in other Chinese regions.