The depletion of fossil fuels necessitates the development of sustainable and energy-efficient techniques for biodiesel production. In recent years, cavitation reactors have emerged as a viable alternative to conventional biodiesel synthesis methods due to their superior conversion rates and shorter processing times. These reactors possess a high surface-to-volume ratio and facilitate efficient heat and mass transfer. This study aims to optimize the production of biodiesel from linseed oil using a novel ultrasonic cavitation reactor through a hybrid approach. In order to achieve this, an L50 orthogonal array with five factors and three levels was developed using a Box-Behnken design based on response surface methodology (RSM). These factors included the molar ratio (4:1, 6:1, and 8:1), ultrasonic power (100, 125, and 150 W), temperature (25, 35, and 45 °C), time (3, 6, and 9 min), and ultrasonic frequency (25, 30, and 35 kHz). The parameters were optimized using RSM-based desirability, genetic algorithm (GA), and particle swarm optimization (PSO) approaches. The results indicated that the RSM-based optimization approach outperformed the other methods. The optimal combination of parameters obtained through RSM consisted of molar ratio of 6.58:1, ultrasonic power of 133.65 W, temperature of 37.44 °C, time of 7.71 min, and pulse frequency of 26.29 kHz. This combination resulted in a biodiesel yield of 95.25%. Furthermore, this study explored the impact of different linseed oil methyl ester, octanol, and diesel blends (B10, B20, B30, B10 (O-10), and B20 (O-10)) on engine performance and emission characteristics. The B20 (O-10) blend exhibited significant potential for simultaneously reducing emissions and enhancing engine performance. When used as engine fuel, the B20 (O-10) blend increased brake thermal efficiency (BTE) by 0.848%, decreased brake specific fuel consumption (BSFC) by 0.607%, and decreased CO, HC, and NOx emissions by 18.75%, 6.55%, and 0.72%, respectively, compared to pure diesel at rated power.