Adoption of conservation agriculture (CA) practices can lead to the dominance of perennial weeds including purple nutsedge (Cyperus rotundus L.) in different agroecologies over time. Without effective management of this weed, successful adoption of CA in smallholder farming systems is rather unlikely. Therefore, we aimed to develop an integrated strategy for managing C. rotundus in CA-based crop rotations. This study encompassed a two-year (during 2020–21 and 2021–22) field experimentation at Kanpur, India with a split-plot design with three replications. The treatments included purple nutsedge management practices in main plot and crop rotations in sub-plot under CA platform (no-tilled with added crop residues). Six management options were adopted in the summer season (during April to June) as main plots that were super-imposed with two crop rotations such as pigeonpea (Cajanus cajan (L.) Huth) – wheat (Triticum aestivum L.) and pearlmillet (Pennisetum glaucum (L.) R.Br) – chickpea (Cicer arietinum L.) as sub-plot. Growing of cover crop Sesbania (Sesbania aculeata (Willd.) Pers.) during the summer season for 45 days followed by knockdown with 2,4-D 500 g a.i. Ha−1 (cultural management; Sesbania+2,4-D) and/or summer irrigation followed by application of halosulfuron-methyl 70 g a.i. Ha−1 at 20 days after irrigation (chemical management; halosulfuron-methyl) reduced the C. rotundus density by 37–42% and 23–64% over years, respectively, than conventional practice (ploughing during summer followed by irrigation and glyphosate 1.5 kg a.e. Ha−1 application). Besides, these practices could decrease 25–27% total weed density (p < 0.05) than conventional practice (mean of years) in summer season. Importantly, these management practices had a carry-over effect on reduction in density of C. rotundus and total weeds of rainy and winter season crops. For instance, Sesbania +2,4-D and halosulfuron-methyl decreased 35% and 15% density of C. rotundus, respectively, during rainy season than conventional practice. Irrespective of crop rotations, Sesbania +2,4-D and halosulfuron-methyl decreased 30–40% density of C. rotundus (mean of years) in winter season than conventional practice (p < 0.05). A significant reduction in dry weight of C. rotundus was recorded under Sesbania +2,4-D and halosulfuron-methyl in summer season by 13–23%, rainy season by 23–29%, and winter season by 55–72% than conventional practice. Pearlmillet-chickpea rotation had a 10–35% lower infestation of C. rotundus than pigeonpea-wheat (p < 0.05). Management practices such as Sesbania +2,4-D and halosulfuron-methyl had higher system productivity by 15–19% in 2020–21 (p < 0.05), 5–7% in 2021–22 (p > 0.05), and 10–12% in average (p > 0.05) than conventional practice. Summer mungbean cultivation increased 45.2% and 10.1% density of C. rotundus in rainy season over Sesbania +2,4-D and halosulfuron-methyl in CA system, respectively (p < 0.05). Furthermore, the density of this weed was significantly higher under pigeonpea crop during rainy season by 41.4% than under pearlmillet crop (mean of years). Therefore, system-based approaches such as growing of cover crop Sesbania followed by knockdown with 2,4-D (Sesbania +2,4-D) and/or herbicide-based management through halosulfuron-methyl during summer season and subsequently recommended weed management practices in cropping systems (pearlmillet – chickpea and pigeonpea – wheat) can reduce the infestation of C. rotundus and total weeds in CA than conventional practice.