Jet substructure observable basis is a systematic and powerful tool for analyzing the internal energy distribution of constituent particles within a jet. In this work, we propose a novel method to insert neural networks into jet substructure basis as a simple yet efficient interpretable IRC-safe deep learning framework to discover discriminative jet observables. The Energy Flow Polynomial (EFP) could be computed with a certain summation order, resulting in a reorganized form which exhibits hierarchical IRC-safety. Thus inserting non-linear functions after the separate summation could significantly extend the scope of IRC-safe jet substructure observables, where neural networks can come into play as an important role. Based on the structure of the simplest class of EFPs which corresponds to path graphs, we propose the Hierarchical Energy Flow Networks and the Local Hierarchical Energy Flow Networks. These two architectures exhibit remarkable discrimination performance on the top tagging dataset and quark-gluon dataset compared to other benchmark algorithms even only utilizing the kinematic information of constituent particles.