The acid water-soluble fraction of the 95% EtOH extract of Leonurus japonicus exhibited significant anti-inflammatory activity by suppressing the p-ERK/ERK ratio and iNOS expression. A further phytochemical investigation of this acid water-soluble fraction led to the isolation of three previously undescribed sesquineolignan glycosides, leolignosides A–C (1–3), and their planar structures were elucidated using HR-ESI-MS and 1D and 2D NMR. To determine the complicated absolute configuration of sesquineolignan glycosides, coupling constants, NOESY data, empirical rules, enzymatic hydrolysis, and ECD data were comprehensively used. These isolated compounds were also screened for anti-inflammatory activity. Encouragingly, leolignosides A–C all suppressed LPS-induced NO overproduction in RAW 264.7 macrophages. The aglycone (3a) of leolignoside C was chosen for further investigation for its activity and the available amount. Similar to the acid water-soluble fraction, compound 3a inhibited NO overproduction and decreased the p-ERK/ERK and p-NF-κB/NF-κB ratios, indicating that 3a may exert anti-inflammatory effects through the ERK/NF-κB signaling pathway.