At present, efficient and stable low-cost electromagnetic (EM) wave absorbing materials have been widely explored, but further improvement is still necessary. In this research, three types of hierarchical Co3O4/N-doped carbon/short carbon fiber (SCF) composites with different assembly structures were produced by annealing the ZIF-67/SCF and Co-LDHs/SCF precursors at 700 °C. The obtained Co3O4/N-doped carbon particles were uniformly attached on SCF in the form of nanocages or thin layer to compose a unique hierarchical structure. Notably, all three composites displayed high-performance EM wave absorption with a low filling ratio of only 20 wt% in paraffin matrix. Among them, cage-like Co-LDHs/SCF derived hierarchical carbon composite demonstrates the best performance, with a broad absorption bandwidth (RL ≤ −10 dB) of 6.08 GHz at 2.0 mm. Such excellent properties are attributed to the formed 3D conductive network, abundant Debye dipolar relaxation centers and strong interfacial polarization. These novel lightweight 2-methylimidazole-mediated Co3O4/N-doped carbon/SCF composites are expected to show great potential in EM wave absorption fields.