The use of inorganic-organic composite membranes or mixed matrix membranes in membrane technology has gained worldwide recognition as a reliable method for producing drinkable water and effectively treating and purifying wastewater. This study aimed to create a model using response surface methodology (RSM) and artificial neural network (ANN) that can accurately predict and optimize the flux behavior of a membrane made of a polymeric-ceramic composite material for oil-water separation. In addition, as mentioned earlier, the optimization of the factor has been achieved through the use of RSM to establish the most optimal mode for the oil-water separation process. The findings of the RSM analysis show that the projected R2 value of 0.9764 closely matches the Adjusted R2 value of 0.9789, indicating a strong agreement between the two. Furthermore, the ANN modeling exhibited significant agreement with the actual data. The ANN model achieved a high level of performance, with an R-squared value of 0.9998 and a mean squared error (MSE) of 2.57E-03. The results were obtained by employing a model structure comprising two concealed layers, with the initial concealed layer comprising 3 neurons and the subsequent concealed layer comprising 10 neurons. The RSM was tweaked to find that the best flow mode happened when the oil emulsion level was 107.117 mg/L, the transmembrane pressure (TP) was 134.219 kPa, and the time period was 12.836 s.