Carbon-based materials like carbon nanotubes, graphene nanoplatelets, graphite, and carbon fibers (CF) are highly promising fillers for enhancing the thermal conductivity of thermal interface materials (TIMs) due to their high thermal conductivity and low thermal expansion. Aligning these fillers can further improve thermal conductivity, but current alignment methods using high magnetic fields are impractical for industrial use. In this study, we investigated the enhancement of the thermal conductivity of CF–polymer composites by aligning CF fillers with a low magnetic field. We observed up to 17 times enhancement of the thermal conductivity (from 3.1 to 52.8 W/mK) in a CF–graphene–polymer composite film by vertically aligning the CF fillers with a magnetic field of 0.75 T. The analysis of the structural properties of the films using X-ray diffraction and field-emission scanning electron microscopy imaging confirmed that the crystalline c-axis of the graphite plates in the CF was oriented perpendicular to the magnetic field direction. The large anisotropy in the diamagnetic susceptibility of the laminated graphene structure of the CF flakes is at the origin of the filler alignment. Furthermore, the thermal conductivity of the composites showed a strong correlation with the degree of filler alignment, which was dependent on the CF–graphene hybridization and the type of polymer matrix. This study provides a scalable and cost-effective method for enhancing the physical properties of carbon composites by controlling their microarchitecture using a low magnetic field.