Winter cover crops are used in organic olive production to increase N-supply and yields, and to reduce weed competition. However, there is limited information on the effect of different cover crops on weed suppression, soil fertility and productivity of organic olive orchards. Here, we compared the relative effect of four contrasting cover crops established from (i) untreated vetch seed, (ii) vetch seed inoculated with a commercial Rhizobium seed inoculum, (iii) an untreated vetch/barley/pea seed mixture and (iv) untreated seed of Medicago polymorpha L. (a native legume species which establishes naturally in olive orchards in Crete) in a 35-year-old experimental table olive orchard. The use of a vetch/barley/pea mixture resulted in the greatest suppression of the dominant weed species Oxalis pes-caprae. Rhizobium inoculation of vetch seed resulted in significantly lower vetch establishment and significantly higher Oxalis suppression but had no significant effect on the root nodulation of vetch plants. There was no significant difference in fruit yield between cover crop treatments, but the fruit weight was significantly higher when cover crops were established from un-treated vetch seeds and the vetch/barley/pea seed mixture compared with the cover crops based on inoculated vetch or Medicago seed. However, although Medicago establishment was very low (<10 plants/m2), fruit yields were numerically 20% higher in the Medicago plots. These findings suggests that, overall, legume cover crops had no effect on fruit yields. This conclusion is supported by the results of the olive leaf analyses which detected no significant differences in nitrogen and other mineral macro- and micronutrient concentration between treatments, except for B (highest in olive leaves from Medicago and lowest in untreated vetch plots) and Mo (highest in olive leaves from Medicago and lowest in vetch/barley/pea mixture plots). Overall, our results suggest that the current recommendation to establish legume-based cover crops in organic olive orchards every year, may need to be revised and that establishing cover crops every 2–4 years may reduce costs without affecting olive fruit yields.