The harvesting time of olive (Olea europaea L.) fruit, which significantly affects the characteristics of virgin olive oil, is mainly determined empirically based on the fruit’s skin color. Developing objective methods such as electrical impedance spectroscopy (EIS) for assessing ripeness is essential. This study aimed to explore the potential of EIS as a rapid and objective technique for detecting the harvesting time of olives. Olive fruits from two varieties, ‘Picholine’ and ‘Buža momjanska’, were harvested in two periods and sorted into four color groups. EIS was applied to each color group to establish a relationship between fruit color and electrochemical properties. The distance of the coordinate at the top of the circular arc of the Cole–Cole plot from the origin (LTO) indicated tissue degradation. The LTO values varied depending on the olive variety, fruit color, and harvest date. The LTO values decreased from green to black fruits in both varieties, indicating textural changes in the olive fruit tissue. This study contributes to the knowledge and understanding of the electrical properties of olive fruit tissue during ripening. EIS shows potential as an innovative tool for determining the harvesting time of olives and for ‘in-field’ olive ripeness assessment.