Physisorption relying on crystalline porous materials offers prospective avenues for sustainable separation processes, greenhouse gas capture, and energy storage. However, the lack of end-to-end deep learning model for adsorption prediction confines the rapid and precise screen of crystalline porous materials. Here, we present DeepSorption, a spatial atom interaction learning network that realizes accurate, fast, and direct structure-adsorption prediction with only information of atomic coordinate and chemical element types. The breakthrough in prediction is attributed to the awareness of global structure and local spatial atom interactions endowed by the developed Matformer, which provides the intuitive visualization of atomic-level thinking and executing trajectory in crystalline porous materials prediction. Complete adsorption curves prediction could be performed using DeepSorption with a higher accuracy than Grand canonical Monte Carlo simulation and other machine learning models, a 20-35% decline in the mean absolute error compared to graph neural network CGCNN and machine learning models based on descriptors. Since the established direct associations between raw structure and target functions are based on the understanding of the fundamental chemistry of interatomic interactions, the deep learning network is rationally universal in predicting the different physicochemical properties of various crystalline materials.