The role of polyphenols in affecting the structural and rheological properties of oleogels was investigated. Polyphenols were selectively removed from extra virgin olive oil (EVOO), and the resulting oils at three different polyphenol levels were gelled by using 10% (w/w) of monoglycerides (MG), rice wax (RW), sunflower wax (SW), and a mixture of β-sitosterol/γ-oryzanol (PS). The structural characteristics of oleogels were assessed by visual appearance, rheology, polarized light microscopy, calorimetry, XRD, and FTIR. Polyphenol content differently affected oleogel characteristics depending on network features. While EVOO-polyphenols did not influence PS- and SW-based oleogels, they reinforced MG- and RW-based oleogel network. As polyphenol content increased, the critical stress and melting temperature also increased, concomitantly with changes in crystal morphology. This was attributed to the capacity of polyphenols to form additional junction points in the crystalline network.