Capillary/residual CO2 trapping is one of the main mechanisms of CO2 storage in underground formations. Therefore, it is required to estimate the brine/CO2 interfacial tension under different conditions. Although many methods have been proposed so far, the error of estimation is still high. This paper proposes a novel deep learning method to estimate the brine/CO2 interfacial tension at various temperatures, pressures, and salinities. The proposed method is a neural network with the Group Method of Data Handling (GMDH) learning method. The GMDH has the advantage of handling the structural and parametric optimization of the network automatically. The proposed method is tested on an experimental dataset of brine/CO2 interfacial tension with CalCl2 and MgCl2 salts. The results of the proposed method were compared with four of the best performing methods in the literature. The Average Absolute Percentage Error (AAPE) of the method on the training, testing and all data was 1.3 %, 2.95 %, 1.73 %, respectively, while the best method from the literature could reach an AAPE of 8.16 % on all data. Therefore, the proposed method performs far better than the existing methods. Also, a sensitivity analysis was done to determine the most influential inputs to estimate the output. The contribution of this work is to show the applicability of the GMDH method to construct more optimal data-driven models to estimate the brine/CO2 interfacial tension. Also, the utilized dataset is collected under a wide range of pressure, temperature and salinity conditions that increases the generality of the model.