Biodiesel is rapidly becoming an efficient substitute for fuel and a potentially significant future renewable energy source. In recent years, used cooking oil has been used as a feedstock for biofuel to reduce production costs. Due to its high catalytic activity, low cost, and eco-friendliness, Nano magnesium oxide (MgO) has attracted attention as a catalyst for biodiesel production. Our work presents the preparation of nanomagnesium oxide (MgO) by the sol–gel method, and its characterization. Optimum conditions and the productive combination of waste cooking oil, methanol, and the synthesized nanocatalyst were predicted using response surface methodology. The optimum conditions were methanol to oil ratio of 7:1, temperature of 50 °C and time of 60 min. The expected values for the yield of biodiesel production responses are quite like the actual values, demonstrating the consistency of the models used for establishing a relationship between the independent process variables and the responses. The predicted model's F-value was 9.09 indicating that the model is significant. The model's pure error had a poor correlation, as the "Lack of Fit F-values” 4.16. The quadratic model fits the data well because the R-squared value for the model equation 92%. The expected values for the yield of biodiesel production responses are quite like the actual values, demonstrating the consistency of the models used for establishing a relationship between the independent process variables and the responses. Biodiesel was characterized using gas chromatography–mass spectrometry and Fourier transform infrared spectroscopy.