Extra virgin olive oil (EVOO) has proven to yield a better health outcome than other saturated fats widely used in the Western diet, including a distinct dysbiosis-preventive modulation of gut microbiota. Besides its high content in unsaturated fatty acids, EVOO also has an unsaponifiable polyphenol-enriched fraction that is lost when undergoing a depurative process that gives place to refined olive oil (ROO). Comparing the effects of both oils on the intestinal microbiota of mice can help us determine which benefits of EVOO are due to the unsaturated fatty acids, which remain the same in both, and which benefits are a consequence of its minority compounds, mainly polyphenols. In this work, we study these variations after only six weeks of diet, when physiological changes are not appreciated yet but intestinal microbial alterations can already be detected. Some of these bacterial deviations correlate in multiple regression models with ulterior physiological values, at twelve weeks of diet, including systolic blood pressure. Comparison between the EVOO and ROO diets reveals that some of these correlations can be explained by the type of fat that is present in the diet, while in other cases, such as the genus Desulfovibrio, can be better understood if the antimicrobial role of the virgin olive oil polyphenols is considered.