Phosphorus deficiency is one of the main constraints to olive production in alkaline soils. The high cost of phosphorus fertilizers may represent an insurmountable obstacle in many countries, leaving the exploitation of their own rock phosphate (RP) as the only low-cost and long-term alternative. The aim of this work was to study the long-term effect of agronomic application of RP (amounts applied: 150 kg ha−1) in a field of olive trees on specific microbial groups in the rhizosphere, nutrient uptake and oil accumulation in the fruit. The results revealed that agronomic application of RP increased the rhizosphere fatty acid methyl ester (FAME) biomarkers indicative of Gram-negative bacteria, fungi and actinobacteria. The level of FAME 16:1ω5 in rhizosphere decreased significantly after agronomic application of RP, indicating that the functioning of arbuscular mycorrhizas was altered. Decreased FAME 16:1ω5 due to RP amendment was not associated with a significant reduction of nutrient concentrations in the olive root; while a significant increase in root P and K contents was observed. However, the level of N did not appear to be significantly influenced. The experimental results also showed that RP-fertilized plants had significantly greater fruit weight and pulp/stone ratio. Fruits of RP-fertilized plants also had significantly higher oil content (+8.67% on average). Our results support the hypothesis that, following agronomic application of RP, the complex interactions of root and microbial community in the olive rhizosphere is a potential factor influencing oil accumulation in the olive fruit.