In this study, for the first time, pharmaceutical waste was used as a heterogeneous catalyst to produce biofuel from waste cooking oil (WCO). An efficient and low-cost heterogeneous catalyst was prepared from waste tablets of calcium carbonate (CaCO3) and magnesium oxide (MgO). A certain amount of alumina nanoparticles (Al2O3) was added to the pharmaceutical waste to participate in the transesterification of the WCO, which positively affected mass yield. The mentioned catalysts were identified by X-ray diffraction (XRD), Fourier transform infrared (FT-IR), field-emission scanning electron microscopy (FESEM), and Brunauer–Emmett–Teller (BET) analyses. According to the BET analysis, the specific surface of the catalysts increased. Also, the effects of various reaction parameters such as temperature, time, catalyst loading, and the oil: methanol ratio were investigated and optimized by response surface methodology (RSM). Furthermore, an adaptive neuro-fuzzy inference system (ANFIS) was coupled with a firefly optimization algorithm to predict biofuel yield. Under optimum conditions (Al2O3 0.952 wt%, catalyst 4.978wt%, oil:methanol ratio 0.5 vol:vol, reaction time 120 min, and reaction temperature 69.6 °C), the mass yield of MgO and CaO catalysts was 95.6 and 90.4 wt%, respectively. The composition of biofuel was identified using gas chromatography–mass spectrometry (GC-MS).