In order to investigate the relationship between the molecular structure of fibers and the differences in physicochemical interactions between fibers and asphalt on the performance of fiber-modified asphalt, this paper chose two types of fibers with different chemical structures: straw fiber and polyester fiber. First, the differences in molecular interactions between the two fibers and asphalt were explored using molecular dynamics, then the differences in the adsorption capacity of the two fibers on asphalt components were tested by attenuated total reflection infrared spectroscopy experiments, and finally, the differences in the rheological properties of the two fiber-modified asphalts were tested by dynamic shear rheology and low-temperature creep experiments. The molecular dynamics simulation findings reveal that polyester fibers may intersperse into asphalt molecules and interact with them via structures such as aromatic rings, whereas straw fibers are merely adsorbed on the asphalt’s surface. Straw fibers and asphalt exhibit hydrogen bonding, whereas polyester fibers and asphalt display van der Waals interactions. The results of attenuated total reflectance infrared spectroscopy indicated that polyester fiber absorbed asphalt components better than straw fiber. The rheological tests revealed that the polyester fiber had the highest complex shear modulus in the temperature range of 46–82 °C, and at 64 °C, the phase angle was 4.289° lower than that of the straw fiber-treated bitumen. Polyester fiber-modified asphalt had a 32.48%, 15.72%, and 6.09% lower creep modulus than straw fiber-modified asphalt at three low-temperature conditions: −6 °C, −12 °C, and −18 °C. It is clear that fibers with aromatic rings as a chemical structure outperform lignin-based fibers in terms of improving asphalt characteristics. The research findings can serve as a theoretical foundation for the selection of fibers to produce fiber-modified asphalt.