Abstract The problem that the exact nonlinear connection between fluids flow and heat transfer in direct contact heat evaporator was studied. The direct contact heat evaporator structure can be optimized and direct contact heat transfer process can be improved with the help of the optimum matching relationship between the complex flow and heat transfer. The methods used are the signal processing technology (i.e., empirical mode decomposition) and the machine learning algorithm (i.e., least squares support vector machine) for calculating experimental heat transfer coefficient here. The important specific and quantitative results are that the volumetric heat transfer coefficient prediction accuracy can be improved by the proposed the hybrid model. When empirical mode decomposition is combined with least squares support vector machine, the accuracy of the hybrid model can be improved by 20% without and 62% with the influencing factors. The conclusions that can be drawn from the results are two-fold. The optimal matching relationship between internal flow characteristics and heat transfer performance can be understood in direct contact heat transfer system for waste heat utilization. The hybrid model can improve the prediction accuracy and efficiency, reduce the number of experiments and costs, save raw materials and shorten the design cycle. Hence, the novelty of the work is that a new hybrid model is proposed for predicting VHTC. The gap it filled in the literature is that the prediction of volumetric heat transfer performance during direct contact heat transfer process is considered. In addition, it is carried out in a highly accurate and highly efficient way using this proposed model.