Olive oils have been shown to be more resistant to oxidation than other vegetable fats, mainly due to their fatty acid (FA) profile which is rich in oleic acid and to their high content of antioxidants, principally phenols and tocopherols. This has situated virgin olive oils (VOOs) among the fats of high nutritional quality. However, it is important to stress that the oil’s commercial category (olive oil, virgin olive oil, extra-virgin olive oil), the variety of the source plant, and the extraction-conservation systems all decisively influence the concentration of these antioxidants and the oil’s shelf-life. The present work studied the fatty acid (FA) and phenolic composition and the oxidative stability (OS) of eight olive varieties grown in Extremadura (Arbequina, Cornicabra, Manzanilla Cacereña, Manzanilla de Sevilla, Morisca, Pico Limón, Picual, and Verdial de Badajoz), with the olives being harvested at different locations and dates. The Cornicabra, Picual, and Manzanilla Cacereña VOOs were found to have high oleic acid contents (>77.0%), while the VOOs of Morisca and Verdial de Badajoz had high linoleic acid contents (>14.5%). Regarding the phenol content, high values were found in the Cornicabra (633 mg·kg−1) and Morisca (550 mg·kg−1) VOOs, and low values in Arbequina (200 mg·kg−1). The OS was found to depend upon both the variety and the date of harvesting. It was higher in the Cornicabra and Picual oils (>55 h), and lower in those of Verdial de Badajoz (26.3 h), Arbequina (29.8 h), and Morisca (31.5 h). In relating phenols and FAs with the OS, it was observed that, while the latter, particularly the linoleic content (R = −0.710, p < 0.001, n = 135), constitute the most influential factors, the phenolic compounds, especially o-diphenols, are equally influential when the oils’ linoleic content is ≥12.5% (R = 0.674, p < 0.001, n = 47). The results show that VOOs’ resistance to oxidation depends not only on the FA or phenolic profile, but also on the interaction of these compounds within the same matrix.