Designable 3D hierarchical architecture with multi-component heterostructure has been built to form a connected network by fully utilizing the advantages of one-dimensional carbon nanofibers. Fe3O4 decorated carbon nanofibers, which exhibited core-shell microstructures, were successfully prepared by solvothermal and electrospinning technology. Their electromagnetic wave absorption (EWA) performances were explored. The morphology which determined the electromagnetic (EM) parameters can be controlled by adjusting the mass ratio of the raw material. The Fe3O4 decorated carbon nanofibers composite showed an excellent absorbing capability. At a thickness of 2.5 mm, the minimum reflection loss (RLmin) value was −28.28 dB. In particular, the widest effective absorption bandwidth (EAB) can reach up to 2.96 GHz at 1.5 mm. It was more surprising that the effective absorption (RL ≤ −10 dB) occurred when the thickness was only 1.0 mm. The absorption mechanism study revealed that the connected structure facilitated the formation of conductive networks, leading to conductive loss, multiple reflection and scatterings, and interfacial polarization loss. The defects that created during carbonization process enhanced the dipole polarization. The generated magnetic loss which mainly consisted of natural resonance and exchanged resonance loss enhanced the attenuation of EW. Moreover, good impedance matching resulting from the synergistic effect of composition and unique porous network also played an important role. The little filler loading and the thin thickness make the composite have a high EWA performance, which is promising for EWA applications.