Vascular dementia (VD) is one of the most common forms of dementia worldwide, characterized by problems with reasoning, planning, judgment, and memory. This study investigated the effect of a histone methyltransferase inhibitor on cognition and mitochondrial function in a rat model of VD, as well as its impact on H2O2-induced neurotoxicity in hippocampal neuronal cultures. In the in vivo experiments, VD was induced by bilateral occlusion of the common carotid artery (CCA) for one month. The histone methyltransferase inhibitor, BIX01294, was administered intracerebroventricularly for one month (22.5 µg.kg−1 three times/week). On day 30, behavioral tests, including the novel object recognition test and elevated plus maze test, were conducted. Mitochondrial enzyme activities, including aconitase, α-ketoglutarate dehydrogenase (α-KG), complex I, and complex IV, were evaluated in the hippocampus of rats following CCA ligation. In the in vitro experiments, the effect of BIX01294 (50–600 μM) on H2O2 (400 µM)-induced cytotoxicity in hippocampal neuronal cells was assessed using the MTT assay. Flow cytometry was performed to evaluate apoptosis. Our findings revealed that BIX01294 effectively improved memory function, Krebs cycle enzyme activity, and mitochondrial function in the rat model of VD. Moreover, in vitro results showed that BIX01294 at a concentration of 100 µM significantly reversed the cytotoxicity and apoptosis induced by H2O2 in neuronal cells. These findings suggest that BIX01294 may have the potential to improve VD complications by reducing oxidative stress and inhibiting histone methylation.