Waste-derived biogas and third-generation algal biodiesel are attractive alternative fuels to substitute fossil diesel in a diesel engine. However, using biodiesel as a pilot liquid fuel and biogas as the main fuel in a diesel engine is a complicated and highly non-linear process. The current study seeks to predict and optimize the combustion and exhaust emission characteristics of a variable compression dual-fuel combustion engine. Data from experiments were obtained at a variety of engine loads, compression ratios, pilot fuel injection pressures, and timings. A multi-layer perceptron network was employed to develop an Artificial Neural Network (ANN) based prognostic model using the experimental data. The developed prognostic model was used to estimate brake thermal efficiency, biogas flow rates, peak in-cylinder pressure, carbon dioxide, unburned hydrocarbons, oxides of nitrogen, and carbon monoxide. The predictive model's robustness is demonstrated by statistical metrics such as R (0.9723–0.988) and R2 (0.9453–0.9761), Nash-Sutcliffe model efficiency (94–97%), and mean absolute percentage error (0.013–0.128%), Kling-Gupta efficiency (0.9548–0.9836), and Theil's U2 model uncertainty (0.162–0.368). To optimize the parameters of dual-fuel combustion, the Multi-Output Response Surface Methodology (RSM) was employed. The trade-off assessment between emission and efficiency using the desirability approach revealed that 84% engine load, 244 bar of fuel injection pressure, 28 °BTDC of injection timing, and 17.5 compression ratio are the best-operating conditions for the test engine. An experimental investigation was used to corroborate the RSM research findings, and errors were less than 9%. It was revealed that ANN-linked RSM is a good hybrid technique for modeling, prediction, and optimization of the performance of a dual-fuel engine.