Given its huge economic, nutritional, and social value, extra virgin olive oil (EVOO) is an essential food. This flagship product of the countries bordering the Mediterranean basin is one of the most frauded products worldwide. Although traditional chemical analyses have demonstrated to be reliable tools for olive drupes and EVOO quality assessment, they present several drawbacks; the urgent need for fast and non-destructive techniques thus motivated this review. Given the lack of comprehensive reviews in the literature, our first aim was to summarize the current knowledge regarding applying spectroscopies, imaging analyses, and other non-destructive techniques to olives and EVOO. The second aim was to highlight the most innovative and futuristic applications and outline the future research prospects within this strategic production chain. With respect to olive drupes, the most interesting results were obtained using RGB imaging and NIR spectroscopy, particularly using portable NIR devices and specific digital cameras for in-field or in-mill monitoring. Nevertheless, it is important to highlight that RGB imaging and NIR spectroscopy need to be integrated with flesh hardness measurements, given the higher reliability of this parameter compared to olive skin color. Finally, with respect to EVOO, although several useful applications of visible imagining, UV–Visible, NIR, and Mid-Infrared spectroscopies have been found, the online monitoring of EVOO quality using NIR spectroscopy strikes us as being the most interesting technique for improving the EVOO production chain in the near future.