No-till management and the establishment of plant cover have been implemented in olive crops in recent years in order to prevent soil erosion and increase soil organic carbon. However, the effect of these conservation practices on the net CO2 exchange at the ecosystem scale has not been explored so far. In this study, we analyze the influence of resident vegetation cover (hereafter weeds) on the net ecosystem CO2 exchange (NEE) in an irrigated olive orchard located in Jaén (SE Spain) by using the eddy covariance technique. NEE was measured in the olive orchard under two treatments, one with weed cover in the alleys from autumn to spring, and another where weed growth was avoided by the application of a glyphosate herbicide. Our study demonstrates that the presence of weeds in the alleys increased carbon assimilation in the weed-cover treatment during the weed growing period (from December to April). However, the net ecosystem CO2 uptake decreased in the weed-cover treatment during late spring (May and June), after weeds were cut and left on the soil, compared to the weed-free treatment, probably due to an increase in soil respiration. On an annual basis, weed removal decreased net carbon uptake by 50% compared to the weed-cover treatment. The annual NEE was −140gCm−2y−1 in the weed-cover treatment and −70gCm−2y−1 in the weed-free treatment. In summary, our study demonstrates that, during the first year of differential treatment, maintenance of weed cover in olive groves has a positive effect on CO2 uptake and enhances the capacity of the agro-system to act as a net CO2 sink.