Oxymatrine (OMT) is the major quinolizidine alkaloid extracted from the root of Sophora flavescens Ait and has been shown to exhibit a diverse range of pharmacological properties. The aim of the present study was to investigate the role of OMT in diabetic brain injury in vivo and in vitro. Diabetic rats were induced by intraperitoneal injection of a single dose of 65 mg/kg streptozotocin (STZ) and fed a high-fat and high-cholesterol diet. Memory function was assessed using a Morris water maze test. A SH-SY5Y cell injury model was induced by incubation with glucose (30 mM/l) to simulate damage in vitro. The serum fasting blood glucose, insulin, serum S100B, malondialdehyde (MDA), and superoxide dismutase (SOD) levels were analyzed using commercial kits. Morphological changes were observed using Nissl staining and electron microscopy. Cell apoptosis was assessed using Hoechst staining and TUNEL staining. NADPH oxidase (NOX) and caspase-3 activities were determined. The effects of NOX2 and NOX4 knockdown were assessed using small interfering RNA. The expression levels of NOX1, NOX2, and NOX4 were detected using reverse transcription-quantitative PCR and western blotting, and the levels of caspase-3 were detected using western blotting. The diabetic rats exhibited significantly increased plasma glucose, insulin, reactive oxygen species (ROS), S-100B, and MDA levels and decreased SOD levels. Memory function was determined by assessing the percentage of time spent in the target quadrant, the number of times the platform was crossed, escape latency, and mean path length and was found to be significantly reduced in the diabetic rats. Hyperglycemia resulted in notable brain injury, including histological changes and apoptosis in the cortex and hippocampus. The expression levels of NOX2 and NOX4 were significantly upregulated at the protein and mRNA levels, and NOX1 expression was not altered in the diabetic rats. NOX and caspase-3 activities were increased, and caspase-3 expression was upregulated in the brain tissue of diabetic rats. OMT treatment dose-dependently reversed behavioral, biochemical, and molecular changes in the diabetic rats. In vitro, high glucose resulted in increases in reactive oxygen species (ROS), MDA levels, apoptosis, and the expressions of NOX2, NOX4, and caspase-3. siRNA-mediated knockdown of NOX2 and NOX4 decreased NOX2 and NOX4 expression levels, respectively, and reduced ROS levels and apoptosis. The results of the present study suggest that OMT alleviates diabetes-associated cognitive decline, oxidative stress, and apoptosis via NOX2 and NOX4 inhibition.